npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2026 – Pkg Stats / Ryan Hefner

@scality/webmcp-eval

v0.1.2

Published

Evaluate WebMCP tool descriptions against a real model: does a user's phrasing reach the right tool?

Readme

Scality WebMCP Eval

Checks that a product's WebMCP tool descriptions do what they are supposed to: given a user's phrasing, does a real model reach for the right tool?

Unit tests call execute() and check the returned object. They never read a tool's name, description or inputSchema — and those three fields are the entire basis on which a model decides what to call. A description can contradict itself and every unit test still passes, because no code is wrong.

This package is the machinery. Each product supplies the things only it can know: its tool catalogue, a fake backend for its tools to run against, the cases, and the agent to drive.

Installation

npm install --save-dev @scality/webmcp-eval

jest is a peer dependency (>=29). It must be your jest: judge() asserts with jest matchers, and a nested second copy produces matchers from a different realm, which fails in ways that are very hard to read.

Node 20 or later. This is a Node package, not a browser one. If your repo defaults jest to jsdom, give each eval file its own environment:

/** @jest-environment node */

What you supply

The package is the machinery. Four things are yours, because only you can know them:

| | What it is | | --- | --- | | Catalogue | your tools as the model sees them — name, description, inputSchema. Supplied rather than imported, because the catalogue worth evaluating is often more than one product's: the shell merges every mounted micro-frontend into one menu, and that is the choice a suite should test. Concatenate a second product's to test the merged menu. | | Agent | which agent to drive, the source it is invoked under, where to reach it, and the credential. agent and source are required — a default would be one deployment's names in everyone's package, and it would fail quietly, since an agent that does not advertise MCP support accepts the request and ignores the catalogue. | | Cases | a transcript and an expectation about which tool the model reaches for. | | Platform | a fake backend your real tools run against, for scenarios. An interface rather than something this package provides: a fake cluster is a statement about one product's backend, and nothing general can be said about them. |

import { createAgent } from '@scality/webmcp-eval';

const agent = createAgent({
  catalogue,                             // [{ name, description, inputSchema }, …]
  agent: process.env.MY_AGENT_NAME,      // required
  source: process.env.MY_AGENT_SOURCE,   // required
  token: process.env.MY_AGENT_TOKEN,     // CI injects it from a secret
  tokenFile: join(homedir(), '.my-agent', 'token'),
});

→ EXAMPLES.md has the rest, worked through in the order you would build it: a case, a suite, prose assertions, the fake platform, a scenario, and the jest file that runs them.

Running it

Keep the eval out of the normal suite — it is non-deterministic and costs tokens:

testPathIgnorePatterns: ['/node_modules/', '\\.eval\\.test\\.ts$'],

and give it a script of its own:

"eval:mcp": "jest --testPathIgnorePatterns=/node_modules/ --testMatch=\"**/*.eval.test.ts\" --forceExit"

Environment this package reads

| Variable | Default | Effect | | --- | --- | --- | | EVAL_TOOLS | all | Comma-separated tool names; narrows cases and scenarios | | EVAL_RUNS | 1 | Votes per scenario. Model output is not reproducible, so it votes rather than asserting once; the bar is 0.8, overridable with runSuite's passRatio. Use 3+ for a verdict you trust | | EVAL_AGENT_URL | http://localhost:8081 | Fallback when url is not passed to createAgent | | EVAL_AGENT_TOKEN | — | Fallback credential, below token and above tokenFile | | EVAL_TOOLS_DIR | — | Required by webmcp-eval-tools-for: where this repo's tools live, relative to its root | | EVAL_CONTRACT_URL | — | Base URL for webmcp-eval-contract, if not passed as an argument | | EVAL_CONTRACT_TOKEN | — | Credential for that fetch, if --token is not passed |

Conventions worth keeping but not read here — your own eval entrypoint decides what they mean: EVAL_SCENARIOS=1 to enable the expensive end-to-end half, EVAL_DRY_RUN=1 to enumerate without calling a model.

CI

The job belongs in your repository, not here. It carries your gateway URL, your identity provider, your secret names and your paths — none of which another consumer can use, and some of which should not be visible to everyone who can read this package.

What this package contributes to CI is one script:

EVAL_TOOLS_DIR=src/mcpTools/tools npx webmcp-eval-tools-for src/mcpTools/tools/myTool.ts

Given the files a PR changed, it prints the tool names they affect, including through shared modules. Feed that into EVAL_TOOLS and a one-line description edit stops paying for the whole suite.

and one more:

npx webmcp-eval-contract https://your-gateway --token "$TOKEN"

Whether the chat API still accepts the request this package builds. When it does not, every scenario fails at once with a 422, in a job named after tool descriptions — so this exists to say which of the two it is before anyone starts rewriting descriptions. Run it non-blocking; a red there is fixed by publishing this package, not by changing the pull request that was open at the time. The checker's own header in src/v1/contract.ts says what it checks and what it does not.

The rest — detecting changed files, minting a token, choosing an environment — stays with the product, where the values live.

Contributing

See CONTRIBUTING.md. Working on this with a coding agent: AGENTS.md.