@swissspidy/webmcp-audit

Crawl a site and audit its WebMCP tools from the outside: lint every page, optionally call the tools with inputs derived from their schemas, detect tools whose description or schema drift between pages, and compare the tool surface against a previous run. Built on @swissspidy/playwright-webmcp. No test suite or code access needed; point it at a URL.

Lighthouse or this?

Use both. Lighthouse's Agentic Browsing category checks a single page: it lists the registered tools, flags <form>s without WebMCP annotations, and reports the problems Chrome itself raises for declarative tools. That is the right first look, and it is where a page's agent-readiness score belongs.

This audit covers what a single-page report does not:

• Calling the tools. --smoke executes read-only tools with valid, boundary and invalid inputs from their schemas and judges the results: errors on valid input, nulls, oversized or slow results, invalid input accepted silently, instruction-like text.
• The whole site. It crawls same-origin pages and reports a tool whose description or schema differs from page to page (cross-page-drift).
• Change over time. --baseline turns an earlier report.json into a contract and reports tools that changed or pages that disappeared, so CI can fail (--fail-on warning) when the tool surface moves.
• Every frame and the full rule set. Tools in iframes are audited too, with all @swissspidy/webmcp-lint rules: descriptions, naming, schema shape, injection, sensitive parameters, and the page-level rules.

npm install --save-dev @swissspidy/webmcp-audit @playwright/test
npx playwright install chromium
npx @swissspidy/webmcp-audit https://shop.example --max-pages 20 --smoke --out .webmcp-audit

The browser has to implement WebMCP. Playwright's Chromium does behind a flag, which the audit passes on its own (--enable-features=WebMCP); --executable points it at another build, such as Chrome Beta or Canary.

Usage: @swissspidy/webmcp-audit <url> [options]

  --max-pages <n>       Maximum pages to visit (default 10)
  --no-crawl            Audit only <url>; do not follow links
  --smoke               Call tools with inputs derived from each tool's inputSchema
                        (required parameters only, all parameters, boundary values,
                        invalid values) and judge what comes back. Only tools annotated
                        read-only (readOnlyHint) are called unless --all-tools is given;
                        the report lists every input that ran.
  --all-tools           With --smoke: also call tools that are not annotated read-only.
                        These may have side effects (adding to a cart, sending mail).
  --ignore-request <glob>
                        With --smoke: a URL glob for requests the page sends on its own,
                        such as analytics, so result-writes does not blame them on a
                        read-only tool (** matches anything, * anything but /); repeatable
  --first-party-origin <origin>
                        With --smoke: an origin besides the audited one that is the
                        site's own backend, e.g. https://api.shop.example; a read-only
                        tool's write there is an error rather than a warning; repeatable
  --fail-on <severity>  Exit 1 when a finding of this severity or worse exists: error
                        (default), warning, info, or never. Pages that fail to load
                        always exit 1.
  --baseline <file>     A previous report.json; report tools whose description, schema
                        or annotations changed on a page since then (contract-changed)
  --header <name: value>
                        HTTP header sent with every request to the audited origin
                        (not to third-party frames or resources), e.g. an
                        Authorization header for a protected staging site; repeatable
  --settle <ms>         Wait this long after load for tools to register, then for the
                        tool list to hold still for as long again (default 500)
  --out <dir>           Output directory (default .webmcp-audit)
  --format <format>     What to print: md (default, the Markdown report), json (the
                        report), or github (one workflow-command annotation per
                        finding; also appends the Markdown to $GITHUB_STEP_SUMMARY)
  --executable <path>   Chrome/Chromium binary (default: Playwright's Chromium, or $PW_CHROMIUM).
                        The browser has to implement WebMCP; --enable-features=WebMCP is
                        always passed, which turns it on in Chromium builds that carry it
  --arg <flag>          Extra browser argument; repeatable
  --quiet               Do not print the report to stdout

Writes report.json and report.md. Exit code 2 on usage errors. Further browser flags go through --arg, for example --arg --enable-experimental-web-platform-features or --arg=--enable-experimental-web-platform-features.

What --smoke runs

Nothing is invented: every input comes from the tool's own inputSchema. For each tool the audit generates

• the required parameters only, with a sample value per type (enum first value, format aware strings, minimum for numbers),
• all parameters,
• boundary values: each number at its minimum and maximum, strings at maxLength and empty, each further enum value, empty arrays,
• invalid inputs: each required parameter missing, each parameter with the wrong type, values outside enum, below minimum, above maximum,

and executes them through the page's modelContext, then judges the results (errors on valid input, null in results, unserializable or oversized results, slow tools, invalid input accepted silently, instruction-like text in results). The Markdown report has one table row per input with its arguments and outcome, so you can see exactly what was called.

Because these are real executions, only tools that declare annotations: { readOnlyHint: true } are called by default. A page with no read-only tools reports zero runs and says so; --all-tools calls everything, which on a shop means adding to carts and submitting forms, so use it against staging. The audit takes the hint at its word, and it can only report a wrong one after the call: a read-only tool that sent a POST, PUT, PATCH or DELETE while it ran is reported as result-writes (error) when the request went to the audited origin, and as result-writes-cross-origin (warning) when it went elsewhere, but either way the request has already reached the site. Add a backend on another host with --first-party-origin https://api.shop.example so writes to it are errors too. Requests the page sends on its own while a tool runs, such as first-party analytics beacons, are blamed on the tool as well; --ignore-request "https://shop.example/api/track" leaves them out. Both flags are repeatable. Run toPassSmoke() from @swissspidy/playwright-webmcp in your test suite to catch a mislabeled tool before the audit ever calls it.

On CI

- run: npx @swissspidy/webmcp-audit https://staging.shop.example --smoke --format github --fail-on warning --header "Authorization: Bearer ${{ secrets.STAGING_TOKEN }}" --baseline audit-baseline/report.json

Findings become annotations on the run, the Markdown report lands in the job summary, and --baseline turns a stored report.json from an earlier run into a contract: a tool whose description, schema or annotations changed on a page reports contract-changed, a page that disappeared reports baseline-page-missing. Logins, cookies and storage state are out of scope here; use the @swissspidy/playwright-webmcp fixture for those.

To audit with another Chrome build, for example the current beta: --executable /opt/google/chrome-beta/chrome.

As a library

import { audit, renderMarkdown } from "@swissspidy/webmcp-audit";

const report = await audit({
  url: "https://shop.example",
  maxPages: 20,
  smoke: true,
  smokeOptions: { tools: ["search_products", "list_reviews"] }, // or { all: true }
});
console.log(renderMarkdown(report));
for (const page of report.pages)
  console.log(page.url, page.lint?.findings.length, page.smoke?.runs.length);

smokeOptions accepts everything webmcp.smoke() does: tools (names or a predicate), all, kinds, ignoreRequests, firstPartyOrigins, maxBoundary, maxInvalid, maxResultChars, maxDurationMs. See the repository README for the rules.