@dbx-tools/appkit-web-search

Add web_search and web_fetch to an AppKit or Mastra backend. web_search runs on the Databricks Model Serving native web-search tool: the model searches the web server-side and returns a synthesized answer plus the sources it used without a third-party search API key. web_fetch reads a single page through got-scraping.

Quick start

Register the plugin and give its tools to an agent:

import { createApp, server } from "@databricks/appkit";
import { webSearch } from "@dbx-tools/appkit-web-search";
import { agents, mastra } from "@dbx-tools/appkit-mastra";

const researcher = agents.createAgent({
  instructions: "Search the web, then read the most relevant sources.",
  async tools(plugins) {
    return { ...(await plugins["web-search"].toolkit()) };
  },
});

await createApp({
  plugins: [server(), webSearch(), mastra({ agents: { researcher } })],
});

Registration provides:

• AppKit plugin registration that resolves and logs the effective policy at boot.
• Two Mastra tools: web_search (answer + citations) and web_fetch (page contents as readable text or raw HTML), plus the same pair as AppKit agent tools through the plugin's ToolProvider.
• web_search resolves a dedicated web-search-capable model - defaulting to Gemini, then GPT - independently of the calling agent's chat model (which may not support web search). Loose names ("gemini", "gpt") fuzzy-match the live catalogue through @dbx-tools/model.
• A built-in provider -> request-fragment map (OpenAI Responses API {"tools":[{"type":"web_search"}]}, Gemini Chat Completions {"google_search":{}}), overridable per provider with the WEB_SEARCH_TOOLS setting.
• A named URL policy (allowlist or unrestricted) over a glob allowlist built on @dbx-tools/path's match matcher: web_search silently filters disallowed citations, web_fetch refuses a disallowed URL, and the mode in force is logged at boot.
• Per-tool approval gating: gate every call, or (for web_fetch) only calls whose URL matches a pattern, mapped onto Mastra's requireApproval.
• Every outbound call runs through AppKit's execute() chain - per-user cache, retry with jittered backoff, timeout, telemetry - and unwinds on an AbortSignal. Each plugin instance keeps its policy and executor separate, so multiple apps in one process remain isolated.
• Page and fallback-result HTML use parser-backed text conversion and selectors, including complete HTML entity decoding and malformed-markup recovery.

Choose a web provider

AppKit has no first-party web-search or page-fetch surface. Use this package when an agent needs to look things up on the open web or read a URL the user pasted, with a policy layer (allowlist + optional approval) around it so a deployment controls which sites are reachable and which calls pause for a human. The web-search tool resolves its own web-search-capable model, so an agent running on any chat model (including one without web search) can still search. It provides a Mastra tool pair and an AppKit plugin, like @dbx-tools/email.

Use native Databricks web search when a supported GPT or Gemini endpoint is deployed. Use the DuckDuckGo scrape fallback only when no capable endpoint is available and unsynthesized search results are acceptable. Use web_fetch for a specific public HTTPS page. For indexed application data, use @dbx-tools/search instead.

Configure the AppKit plugin

import { createApp, lakebase, server } from "@databricks/appkit";
import { webSearch } from "@dbx-tools/appkit-web-search";
import { agents, mastra } from "@dbx-tools/appkit-mastra";

const researcher = agents.createAgent({
  instructions: "Research questions using web_search, then read sources with web_fetch.",
  async tools(plugins) {
    return { ...(await plugins["web-search"].toolkit()) };
  },
});

await createApp({
  plugins: [
    server(),
    lakebase(),
    webSearch({
      model: "gemini", // defaults to Gemini, then GPT, when omitted
      allowedUrls: ["*.databricks.com", "docs.example.com"],
    }),
    mastra({ agents: researcher, storage: true }),
  ],
});

plugin.webSearch() resolves config (over env), compiles the URL policy, and provides the executor used by its tools. App-integrated Mastra agents can consume the native plugin toolkit as shown above, which preserves that plugin's policy, OBO scope, and AppKit execution chain. Approval, when enabled, requires Mastra storage, so register lakebase() or configure storage in the Mastra plugin.

The plugin is also an AppKit ToolProvider, so a native code agent can take the same two tools without Mastra. Put the definition at server/agents/researcher/agent.ts so AppKit discovers it:

import { createAgent } from "@databricks/appkit/beta";

export default createAgent({
  name: "Researcher",
  default: true,
  model: "databricks-gpt-6-1-sol",
  instructions: "Research questions with web_search, then read sources with web_fetch.",
  tools: (plugins) => ({ ...plugins["web-search"].toolkit() }),
  skills: ["research-policy"],
});

Place the curated skill at server/agents/skills/research-policy/SKILL.md, register native agents() with no deprecated agent map, and build a compiled native-agent server with:

import { appkitServerConfig } from "@databricks/appkit/tsdown";

export default appkitServerConfig();

Both tools are annotated effect: "read" and require user context, and neither is auto-inheritable: an agent has to ask for them, because web_fetch reaches whatever URL the model produces.

Native volume skills run as the service principal. The 0.81 skillCredentialMode: "obo" option is not wired. Keep request-scoped writable skills, remote sources, asynchronous Graphiti discovery, Sandbox/Monty, Mastra memory, and durable approvals on @dbx-tools/appkit-mastra.

Choose the web-search model

The native web-search tool only runs on certain models, and it is provider- specific. This package resolves deployed endpoints through @dbx-tools/model and applies this order:

1. Highest-ranked web-search-capable Gemini.
2. Without Gemini, a capable GPT matching the calling/requested model.
3. Highest-ranked capable GPT.
4. Any additional modelFallbacks searches in order.

When Databricks reports that Gemini web search is unavailable because cross-region processing is disabled, the plugin suppresses Gemini for 24 hours in that runtime and transparently retries the same tool call with the next GPT candidate.

DATABRICKS_SERVING_ENDPOINT_NAME is AppKit's standard name for a Model Serving binding, and it is the field the plugin declares in its manifest, so app.yaml / bundle wiring reaches this plugin the way it reaches any other. The request's model, plugin model, and environment model values are used to match the calling GPT after Gemini availability is checked. When the tool runs through @dbx-tools/appkit-mastra, the host injects the resolved chat model, overriding any different model argument generated by the LLM.

Model resolution runs against the live workspace catalogue through @dbx-tools/model, so it only picks an endpoint that is actually deployed - a configured id that doesn't exist is never called.

Scrape fallback

The native tool is preferred. When the workspace has no deployed GPT/Gemini endpoint (so native web search can't run at all), the tool falls back to a DuckDuckGo scrape using got-scraping so it still returns results instead of erroring. Fallback results set model to "scrape:duckduckgo" and carry the substance in citations (there is no model synthesizing an answer, so answer is a lead-in over the top results). It is enabled by default. Set scrapeFallback: false (or WEB_SEARCH_SCRAPE_FALLBACK=0) to require a native model and error when none is deployed.

The plugin selects the provider request fragment from the resolved model: OpenAI GPT uses the Responses API {"tools":[{"type":"web_search"}]}. Gemini uses the top-level Chat Completions field {"google_search":{}}. Override or extend that map per provider with the webSearchTools setting (env WEB_SEARCH_TOOLS, JSON) as the platform evolves:

plugin.webSearch({
  model: "databricks-gemini-3-pro",
  webSearchTools: { gemini: { request: { google_search: {} } } },
});

Search and fetch without an agent

import { search, fetch, runtime, tool } from "@dbx-tools/appkit-web-search";
import { getExecutionContext } from "@databricks/appkit";

const webRuntime = runtime.createWebSearchRuntime({
  model: "gemini",
  allowedUrls: ["*.databricks.com"],
});

// web_search needs the OBO client + host from the active execution context:
const ctx = getExecutionContext();
const host = (await ctx.client.config.getHost()).toString();

const result = await search.runWebSearch(
  { query: "unity catalog lineage best practices" },
  webRuntime,
  { client: ctx.client, host },
);
// result.answer, result.citations, result.model

const page = await fetch.runWebFetch({ url: result.citations[0]!.url, format: "text" }, webRuntime);

Use direct calls for operational lookups, tests, or admin flows where a model is not involved. Standalone Mastra tool factories likewise accept runtime or config. They never discover or attach to a process-global plugin:

const webRuntime = runtime.createWebSearchRuntime({
  allowedUrls: ["*.databricks.com"],
});

const tools = {
  web_search: tool.webSearchTool({ runtime: webRuntime }),
  web_fetch: tool.webFetchTool({ runtime: webRuntime }),
};

Pass the runtime explicitly or provide the executor to createWebSearchRuntime.

Restrict reachable URLs

import { allowlist } from "@dbx-tools/appkit-web-search";

const list = allowlist.toUrlAllowList(
  allowlist.parseAllowedUrls(["*.databricks.com", "docs.example.com/api/**"]),
);

list.allows("https://docs.databricks.com/aws/en/index.html"); // true
list.allows("https://evil.example.com/"); // false

Each entry is a glob compiled by @dbx-tools/path's match matcher. A host entry (no path, for example databricks.com or *.databricks.com) matches the URL's hostname, and a bare host also matches its subdomains. A path entry (docs.example.com/api/**) matches host + pathname. Enforcement is asymmetric by design: web_search citations are silently filtered to the permitted set (the model never surfaces a source it then can't fetch), while an explicit web_fetch of a disallowed URL is refused with an error (a visible, correctable mistake). web_fetch accepts only HTTPS targets that resolve to public addresses, and reapplies both checks before every redirect request.

Which of the two modes is in force is named by urlPolicy, and the effective mode is in the boot log:

| urlPolicy | Effect | | ---------------- | ---------------------------------------- | | "allowlist" | Only allowedUrls entries are reachable | | "unrestricted" | Any public HTTPS URL is reachable |

Leave urlPolicy unset and it follows allowedUrls: "allowlist" when there are entries, "unrestricted" when there are none. Set it to say which one you meant. The two contradictions fail at startup rather than quietly picking a side: "allowlist" with no entries, and "unrestricted" with entries.

// An open deployment, said out loud:
plugin.webSearch({ urlPolicy: "unrestricted" });

// A closed one:
plugin.webSearch({ urlPolicy: "allowlist", allowedUrls: ["*.databricks.com"] });

Gate calls for approval

Both tools run without approval by default. Pass approval to a tool (or set it plugin-wide) to require approval. true gates every call. A URL pattern (or list) gates only calls whose URL matches - a web_fetch is evaluated precisely against its target URL, while a web_search (whose result URLs are unknown before the call) treats a pattern gate as "gate every call".

import { tool } from "@dbx-tools/appkit-web-search";

// Every fetch requires approval:
tool.webFetchTool({ approval: true });

// Only fetches of an internal domain require approval:
tool.webFetchTool({ approval: "*.internal.example.com" });

// The same thing, with the mode named:
tool.webFetchTool({ approval: { mode: "urls", patterns: ["*.internal.example.com"] } });

// Plugin toolkit default:
plugin.webSearch({ approval: ["*.internal.example.com", "*.corp.example.com"] });

// Standalone tool config:
tool.webFetchTool({ config: { approval: ["*.internal.example.com"] } });

Every spelling normalizes to one of three modes - { mode: "none" }, { mode: "always" }, { mode: "urls", patterns } - which is what the resolved config carries and the boot log reports. Approval uses the same glob syntax as the allowlist, so the two policies read identically.

Configuration

Resolution order is explicit config first, then the environment variable, then a built-in default.

| Option | Type | Default | Description | | --------------------- | ------------------------------------------------- | ----------------------- | --------------------------------------------------------------------------- | | model | string | resolved from fallbacks | Web-search endpoint: an endpoint name, a loose name, or a capability class. | | modelFallbacks | string \| string[] | Gemini, then GPT | Ordered candidates tried when model is unset. | | webSearchTools | Record<string, unknown> | built-in provider map | Provider -> request-fragment overrides, merged over the defaults. | | modelFuzzyMatch | boolean | true | Fuzzy-match loose model names against the live catalogue. | | modelFuzzyThreshold | number | 0.4 | Fuse.js score below which a fuzzy match is accepted. | | maxCitations | number | 10 | Hard cap on citations returned from one search. | | fetchMaxLength | number | 50000 | Hard cap on web_fetch content length, in characters. | | timeoutMs | number | 60000 | Per-request network timeout. | | scrapeFallback | boolean | true | Scrape DuckDuckGo when no web-search-capable model is deployed. | | urlPolicy | "unrestricted" \| "allowlist" | follows allowedUrls | Which URLs the tools may reach. | | allowedUrls | string \| string[] | none | Allow-list globs or bare hosts. | | approval | boolean \| string \| string[] \| ApprovalPolicy | { mode: "none" } | Default approval gate for both tools. |

| Environment variable | Sets | | ---------------------------------- | ----------------------------------------------------------------- | | WEB_SEARCH_MODEL | model (dedicated override, wins over the binding below) | | DATABRICKS_SERVING_ENDPOINT_NAME | model (AppKit's Model Serving binding, treated as a preference) | | WEB_SEARCH_MODEL_FALLBACKS | modelFallbacks (comma/space-separated) | | WEB_SEARCH_TOOLS | webSearchTools (JSON object. Any other value fails at boot.) | | WEB_SEARCH_FUZZY | modelFuzzyMatch | | WEB_SEARCH_FUZZY_THRESHOLD | modelFuzzyThreshold | | WEB_SEARCH_MAX_CITATIONS | maxCitations | | WEB_SEARCH_FETCH_MAX_LENGTH | fetchMaxLength | | WEB_SEARCH_TIMEOUT_MS | timeoutMs | | WEB_SEARCH_SCRAPE_FALLBACK | scrapeFallback | | WEB_SEARCH_URL_POLICY | urlPolicy | | WEB_SEARCH_ALLOWED_URLS | allowedUrls (comma/space-separated) |

The plugin's maxCitations / fetchMaxLength are hard caps: a per-call request may narrow them but never exceed them.

Every outbound call - the serving POST, a page fetch, the scrape fallback - runs through the plugin's execute() chain, so all three are cached, retried with jittered backoff, timed out, and traced. The cache is namespaced per user, so an on-behalf-of result never crosses identities. See defaults.ts for the TTLs and retry budgets and why each was chosen.

OpenAI Responses web searches stream native search, page-read, and completion status through the Mastra tool writer. The common tool-session pill updates while the search runs. The completed Responses payload remains the tool result.

Requirements: the native web-search tool runs on pay-per-token GPT / Gemini serving endpoints with cross-region processing enabled. It is unavailable on provisioned throughput, for external models, or under HIPAA/BAA compliance. See the Databricks docs.

Modules

• plugin - WebSearchPlugin, webSearch() AppKit plugin factory, and the AppKit ToolProvider implementation.
• tool - webSearchTool() / webFetchTool() Mastra tools.
• defaults - the cache / retry / timeout settings every outbound call runs with, and the reasoning behind each.
• search - runWebSearch() over the Databricks native web-search tool.
• provider - provider detection + the provider -> tool-spec map.
• scrape - runScrapeSearch() DuckDuckGo fallback plus parser-backed parseDdgHtml() result extraction for workspaces with no native web-search model.
• fetch - runWebFetch() over got-scraping.
• html-text - htmlToText() / htmlFragmentToText() / decodeHtmlEntities(), shared by fetch and the DuckDuckGo scrape fallback.
• runtime - isolated runtime construction and execution: createWebSearchRuntime(), createResolvedWebSearchRuntime(), and executeRead().
• config - config types, JSON schema, resolveWebSearchConfig(), URL policy and approval helpers.
• allowlist - URL allowlist parsing/compiling on top of @dbx-tools/path.
• schema - zod tool contracts and inferred types.

Pair this package with @dbx-tools/appkit-mastra to add the tools to an agent and resolve its model through @dbx-tools/model. Use @dbx-tools/email for the sibling approval-gated tool pattern.