@scality/webmcp-eval
v0.1.2
Published
Evaluate WebMCP tool descriptions against a real model: does a user's phrasing reach the right tool?
Keywords
Readme
Scality WebMCP Eval
Checks that a product's WebMCP tool descriptions do what they are supposed to: given a user's phrasing, does a real model reach for the right tool?
Unit tests call execute() and check the returned object. They never read a tool's name,
description or inputSchema — and those three fields are the entire basis on which a model
decides what to call. A description can contradict itself and every unit test still passes, because
no code is wrong.
This package is the machinery. Each product supplies the things only it can know: its tool catalogue, a fake backend for its tools to run against, the cases, and the agent to drive.
Installation
npm install --save-dev @scality/webmcp-evaljest is a peer dependency (>=29). It must be your jest: judge() asserts with jest
matchers, and a nested second copy produces matchers from a different realm, which fails in ways
that are very hard to read.
Node 20 or later. This is a Node package, not a browser one. If your repo defaults jest to jsdom, give each eval file its own environment:
/** @jest-environment node */What you supply
The package is the machinery. Four things are yours, because only you can know them:
| | What it is |
| --- | --- |
| Catalogue | your tools as the model sees them — name, description, inputSchema. Supplied rather than imported, because the catalogue worth evaluating is often more than one product's: the shell merges every mounted micro-frontend into one menu, and that is the choice a suite should test. Concatenate a second product's to test the merged menu. |
| Agent | which agent to drive, the source it is invoked under, where to reach it, and the credential. agent and source are required — a default would be one deployment's names in everyone's package, and it would fail quietly, since an agent that does not advertise MCP support accepts the request and ignores the catalogue. |
| Cases | a transcript and an expectation about which tool the model reaches for. |
| Platform | a fake backend your real tools run against, for scenarios. An interface rather than something this package provides: a fake cluster is a statement about one product's backend, and nothing general can be said about them. |
import { createAgent } from '@scality/webmcp-eval';
const agent = createAgent({
catalogue, // [{ name, description, inputSchema }, …]
agent: process.env.MY_AGENT_NAME, // required
source: process.env.MY_AGENT_SOURCE, // required
token: process.env.MY_AGENT_TOKEN, // CI injects it from a secret
tokenFile: join(homedir(), '.my-agent', 'token'),
});→ EXAMPLES.md has the rest, worked through in the order you would build it: a case, a suite, prose assertions, the fake platform, a scenario, and the jest file that runs them.
Running it
Keep the eval out of the normal suite — it is non-deterministic and costs tokens:
testPathIgnorePatterns: ['/node_modules/', '\\.eval\\.test\\.ts$'],and give it a script of its own:
"eval:mcp": "jest --testPathIgnorePatterns=/node_modules/ --testMatch=\"**/*.eval.test.ts\" --forceExit"Environment this package reads
| Variable | Default | Effect |
| --- | --- | --- |
| EVAL_TOOLS | all | Comma-separated tool names; narrows cases and scenarios |
| EVAL_RUNS | 1 | Votes per scenario. Model output is not reproducible, so it votes rather than asserting once; the bar is 0.8, overridable with runSuite's passRatio. Use 3+ for a verdict you trust |
| EVAL_AGENT_URL | http://localhost:8081 | Fallback when url is not passed to createAgent |
| EVAL_AGENT_TOKEN | — | Fallback credential, below token and above tokenFile |
| EVAL_TOOLS_DIR | — | Required by webmcp-eval-tools-for: where this repo's tools live, relative to its root |
| EVAL_CONTRACT_URL | — | Base URL for webmcp-eval-contract, if not passed as an argument |
| EVAL_CONTRACT_TOKEN | — | Credential for that fetch, if --token is not passed |
Conventions worth keeping but not read here — your own eval entrypoint decides what they mean:
EVAL_SCENARIOS=1 to enable the expensive end-to-end half, EVAL_DRY_RUN=1 to enumerate without
calling a model.
CI
The job belongs in your repository, not here. It carries your gateway URL, your identity provider, your secret names and your paths — none of which another consumer can use, and some of which should not be visible to everyone who can read this package.
What this package contributes to CI is one script:
EVAL_TOOLS_DIR=src/mcpTools/tools npx webmcp-eval-tools-for src/mcpTools/tools/myTool.tsGiven the files a PR changed, it prints the tool names they affect, including through shared
modules. Feed that into EVAL_TOOLS and a one-line description edit stops paying for the whole
suite.
and one more:
npx webmcp-eval-contract https://your-gateway --token "$TOKEN"Whether the chat API still accepts the request this package builds. When it does not, every
scenario fails at once with a 422, in a job named after tool descriptions — so this exists to say
which of the two it is before anyone starts rewriting descriptions. Run it non-blocking; a red
there is fixed by publishing this package, not by changing the pull request that was open at the
time. The checker's own header in src/v1/contract.ts says what it checks and what it does not.
The rest — detecting changed files, minting a token, choosing an environment — stays with the product, where the values live.
Contributing
See CONTRIBUTING.md. Working on this with a coding agent: AGENTS.md.
