mcp-agent-reliability

1. Value Proposition

Problem: AI agents frequently call the wrong tools. Vague tool descriptions, overloaded context windows, and zero visibility into selection quality cause wasted tokens, failed tasks, and frustrated users.

Solution: mcp-agent-reliability is a lightweight MCP server that acts as a reliability coach for your agents. It helps you:

• Score how clear and LLM-friendly your tool descriptions are (0–100)
• Estimate how many tokens your tools will consume
• Simulate which tool an agent is most likely to pick for a given prompt
• Generate simple test prompts to verify correct tool selection
• Produce a full reliability report with actionable recommendations

All features are pure computation — no paid API keys, no external calls, zero ongoing cost.

Built for entrepreneurs, founders, and teams who are tired of agents calling the wrong tools and burning money.

2. Why This Project Exists

Imagine you give a 10-year-old child a big list of 30 toys and say “go play with the right one”.
If the labels are confusing, the child will pick the wrong toy.

AI agents are the same.

When you connect many MCP servers, the agent sees a long menu of tools.
If the descriptions are vague, it picks the wrong tool → wasted tokens → failed tasks.

This server is the label checker and practice teacher for that menu.

In 2026, as agents become more autonomous and tool counts grow, reliability is no longer optional — it is the difference between a demo and a production system.

3. Features

| Feature | Description | |---------|-------------| | Tool Description Scoring | Heuristic 0–100 score with reasons and concrete suggestions | | Token Cost Estimation | Rough token count for a list of tools + advice on progressive loading | | Tool Choice Simulation | Keyword-heuristic prediction of which tool an agent would select | | Test Prompt Generation | 3 ready-to-use prompts to verify an agent picks the correct tool | | Reliability Report | Combined score + token summary with overall status and recommendation | | Zero External Cost | Pure local computation, no API keys required | | Stateless-friendly | Compatible with modern MCP updates | | TypeScript + Official SDK | Built on @modelcontextprotocol/sdk |

4. Architecture

flowchart TD
    A[MCP Client<br/>Cursor / Claude / etc.] -->|stdio| B[mcp-agent-reliability Server]
    B --> C[ListTools Handler]
    B --> D[CallTool Handler]
    D --> E1[score_tool_description]
    D --> E2[estimate_token_cost]
    D --> E3[simulate_tool_choice]
    D --> E4[generate_agent_tests]
    D --> E5[reliability_report]
    E1 & E2 & E3 & E4 & E5 --> F[Pure Heuristic Utils<br/>scoring.ts]
    F --> G[JSON Response back to Client]

• Transport: stdio (standard for local MCP servers)
• Runtime: Node.js ≥ 18
• Core logic: Pure functions in src/utils/scoring.ts (no network, no side effects)
• Tools: Five focused tools registered via the official MCP SDK

5. Installation

Option A — Local (recommended while developing)

git clone https://github.com/princeruhulofficial/mcp-agent-reliability.git
cd mcp-agent-reliability
npm install
npm run build
npm start

Option B — After publishing to npm

npx -y mcp-agent-reliability

Option C — From source with tsx (dev)

npm run dev

6. MCP Client Configuration

Cursor / Claude Desktop / most MCP clients

Add this to your MCP config (~/.cursor/mcp.json or claude_desktop_config.json):

Local path version:

{
  "mcpServers": {
    "agent-reliability": {
      "command": "node",
      "args": ["/ABSOLUTE/PATH/TO/mcp-agent-reliability/dist/index.js"]
    }
  }
}

After npm publish (recommended for others):

{
  "mcpServers": {
    "agent-reliability": {
      "command": "npx",
      "args": ["-y", "mcp-agent-reliability"]
    }
  }
}

Replace /ABSOLUTE/PATH/TO/... with the real full path on your machine.

Restart the client after saving the config.

7. All 5 Tools

7.1 score_tool_description

Purpose: Score how clear, specific, and LLM-friendly a tool description is (0–100). Use this before adding a new tool to an agent to reduce wrong tool calls.

Parameters:

| Name | Type | Required | Description | |------|------|----------|-------------| | description | string | Yes | The full tool description text to score | | name | string | No | Optional name of the tool (e.g. create_invoice) |

Return schema (example):

{
  "score": 85,
  "reasons": [
    "Good length for an LLM to read",
    "Language looks specific",
    "Mentions inputs or outputs — helpful for the model",
    "Overall: strong description — agent should select it reliably"
  ],
  "suggestions": [],
  "interpretation": "Excellent — agent should pick this tool reliably"
}

Example call:

Tool: score_tool_description
name: create_invoice
description: Create a new invoice for a customer. Requires customer_id and amount. Returns invoice_id.

7.2 estimate_token_cost

Purpose: Roughly estimate how many tokens a list of tool definitions will consume in the agent context window. Helps decide whether to enable progressive loading.

Parameters:

| Name | Type | Required | Description | |------|------|----------|-------------| | tools | array | Yes | List of objects with name and description |

Return schema (example):

{
  "total_estimated_tokens": 1240,
  "tool_count": 5,
  "average_per_tool": 248,
  "breakdown": [
    { "name": "create_invoice", "tokens": 210 },
    { "name": "send_email", "tokens": 185 }
  ],
  "advice": "Low — should be fine for most agents"
}

Advice thresholds:

• > 15000 → High — consider progressive disclosure or fewer tools
• > 8000 → Moderate — monitor context usage
• otherwise → Low — should be fine

7.3 simulate_tool_choice

Purpose: Given a user prompt and a list of available tools, predict which tool an agent is most likely to pick. Useful for testing tool selection before production.

Parameters:

| Name | Type | Required | Description | |------|------|----------|-------------| | prompt | string | Yes | The user message or task the agent will see | | tools | array | Yes | List of tools (name + description) |

Return schema (example):

{
  "predicted_tool": "create_invoice",
  "confidence": 78,
  "all_scores": [
    { "name": "create_invoice", "score": 6 },
    { "name": "send_email", "score": 2 }
  ],
  "note": "This is a keyword-heuristic simulation, not a real LLM. Use it for quick checks."
}

7.4 generate_agent_tests

Purpose: Generate 3 simple test prompts that you can feed to an agent to verify it correctly selects and uses a given tool.

Parameters:

| Name | Type | Required | Description | |------|------|----------|-------------| | tool_name | string | Yes | Name of the tool to test | | description | string | Yes | Description of the tool |

Return schema (example):

{
  "tool": "create_invoice",
  "test_prompts": [
    "Please use the create_invoice tool to Create a new invoice for a customer...",
    "I need to Create a new invoice for a customer. Can you call the right tool?",
    "Call create_invoice with a safe example input and show me the result."
  ],
  "how_to_use": "Copy each prompt into your agent chat (with only this tool enabled) and check if it calls the correct tool."
}

7.5 reliability_report

Purpose: Create a short reliability report for a set of tools. Combines description scores and token estimates into one actionable summary.

Parameters:

| Name | Type | Required | Description | |------|------|----------|-------------| | tools | array | Yes | List of tools (name + description) |

Return schema (example):

{
  "overall_status": "Good",
  "average_description_score": 78,
  "total_estimated_tokens": 1240,
  "tool_count": 5,
  "tools": [
    {
      "name": "create_invoice",
      "score": 85,
      "estimated_tokens": 210,
      "top_suggestion": "Looks good"
    }
  ],
  "recommendation": "You are in a healthy range. Keep monitoring as you add more tools."
}

Overall status logic:

• Needs attention if average score < 55 or total tokens > 20 000
• Acceptable with room to improve if average score < 70
• otherwise Good

8. Scoring Methodology

The scoring engine is a pure heuristic (no LLM calls). It starts at a neutral 50 and adjusts based on observed MCP failure patterns:

| Check | Effect | |-------|--------| | Description length < 20 chars | −25 | | Length between 40–300 chars | +15 | | Length ≥ 300 chars | −10 | | Contains vague words (stuff, things, handle, process…) | −15 | | Language looks specific | +10 | | Mentions inputs / outputs / returns | +10 | | Destructive action without safety note | −10 | | Tool name follows snake_case | +5 | | Final score clamped to 0–100 | — |

Interpretation bands:

• ≥ 80 → Excellent — agent should pick this tool reliably
• 60–79 → OK — improve with the suggestions
• < 60 → Weak — high chance of wrong or missed tool calls

Token estimation uses ≈ 3.5 characters per token (slightly denser than plain text because of schema overhead) plus a fixed 40-token schema boilerplate per tool.

9. Examples

Score a strong description

score_tool_description
name: create_invoice
description: Create a new invoice for a customer. Requires customer_id and amount. Returns invoice_id.

→ Score around 85, interpretation “Excellent”.

Score a weak description

score_tool_description
name: handle_stuff
description: Does things with data.

→ Low score, suggestions to be more specific and mention inputs/outputs.

Full reliability report

Pass a list of your real tools to reliability_report and get an overall status + per-tool breakdown in one call.

10. Use Cases

| Who | How they use it | |-----|-----------------| | Founders / Entrepreneurs | Quickly check if their agent’s tool set is production-ready before shipping | | Agent builders | Score every new tool description before adding it to the system | | Teams with many MCP servers | Estimate total token overhead and decide on progressive disclosure | | QA / Testing | Generate test prompts and simulate tool choice before real LLM runs | | Cost-conscious operators | Catch token-heavy tool lists early |

11. Design Principles

1. Pure computation — no external API, no secrets, no side effects
2. Fast & free — runs entirely locally
3. Actionable — every score comes with reasons and concrete suggestions
4. Focused — only five tools, each solving one clear problem
5. Honest — the simulator is a heuristic, not a real LLM (clearly stated)
6. Entrepreneur-friendly — simple language, clear value, zero ongoing cost

12. Performance

• All tools are synchronous pure functions
• Typical response time: < 5 ms on modern hardware
• Memory footprint: negligible (no large models or caches)
• Scales linearly with number of tools (usually tens, not thousands)

13. Security & Privacy

• No network calls
• No data leaves your machine
• No API keys required or stored
• No logging of tool descriptions or prompts beyond the current request
• MIT licensed — audit the full source in minutes

14. FAQ

Q: Does this replace a real LLM evaluation?
A: No. It is a fast, free, local heuristic for early feedback. Use it before expensive LLM-based evals.

Q: Why not use an LLM to score descriptions?
A: That would require API keys and cost money. This version is deliberately zero-cost. An optional LLM-backed mode is on the roadmap.

Q: Can I use it with remote / hosted MCP?
A: Current version is stdio-only. A hosted version is planned.

Q: Is the token estimate accurate?
A: It is a rough approximation (±20–30% typical). Good enough for “is this too heavy?” decisions.

Q: Will you publish to npm?
A: Yes — once the package is published, the npx one-liner will work for everyone.

15. Roadmap

• [ ] Optional LLM-backed scoring (higher accuracy when you want it)
• [ ] Hosted version with dashboard
• [ ] Integration with progressive disclosure patterns
• [ ] npm package publication for one-command install
• [ ] More sophisticated simulation (optional embedding similarity)
• [ ] Export reports as Markdown / HTML

16. Contributing

Contributions are welcome!

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-improvement)
3. Make your changes and add tests if relevant
4. Open a Pull Request with a clear description

Please keep the core philosophy: pure, free, fast, and entrepreneur-friendly.

17. License

MIT License — see LICENSE for details.