specspector

Diagnose any OpenAI-compatible endpoint. Claiming "OpenAI-compatible" says nothing about whether tool calling actually works, whether usage is reported on streams, or what happens to your TTFT when 16 coding agents hit it at once. Paste a URL, get the truth in about five minutes.

npx specspector --url http://localhost:30000/v1 --model Qwen/Qwen3-32B

  specspector v0.1.0 — endpoint diagnostic
  Endpoint  http://localhost:30000/v1
  Model     Qwen/Qwen3-32B

  OPENAI COMPATIBILITY          94/100
  AGENTIC CODING SUITABILITY    91/100
  MAX USEFUL CONCURRENCY        16 agents
  ...

What it measures

Compatibility — the OpenAI contract that agentic clients depend on: /models, basic and multi-turn chat, streaming chunk shape and the data: [DONE] terminator, stream_options.include_usage, tool calling (single, parallel, tool_choice: none/required, tool-result round-trip), structured output (json_object and strict json_schema), max_completion_tokens, stop sequences, n>1, logprobs, and error shapes (401 for bad keys, 404 for unknown models, 4xx + {error:{…}} for malformed requests). Optional: vision input, reasoning-token accounting.

Performance under agentic load — single-query benchmarks don't represent coding agents, so this suite measures the things that matter to them:

• TTFT p50/p95 and per-user + aggregate throughput at 1/4/8/16 concurrent requests (--levels), against multi-thousand-token prompts.
• Context ceiling: binary search for the largest accepted prompt and whether an early needle is still recalled — "accepted" and "actually usable" are different numbers.
• Prefix cache effectiveness: TTFT on a repeated long prefix vs. fresh prefixes of the same size. Agents re-send huge system prompts every turn; a missing prefix cache silently taxes every single call.

Agentic workload — concurrent simulated coding-agent sessions (run tests → read failing file → report root cause) with multi-turn tool calls, measuring turn-latency degradation, malformed tool-call arguments, and the highest concurrency level that stays healthy: max useful concurrency.

Scores

• OpenAI compatibility /100: weighted pass ratio — tool calling 25, streaming 20, core chat 20, structured output 15, parameters 10, error shapes 10. warn (degraded/partial) counts half; skipped checks are excluded rather than penalized.
• Agentic coding suitability /100: tool protocol 30, throughput retention under load 20, concurrency ceiling 20, latency health at peak level 15, prefix-cache effectiveness 10, usable context 5.

Both scores are computed from the check results in the same report, so you can always see why a score is what it is.

Cost control

You're spending your own tokens, so everything is budgeted:

• --dry-run prints every check and its estimated token cost before sending anything.
• --max-tokens 3000000 is a hard budget; when it's hit, remaining checks are skipped with a note, never half-run.
• --suites compat runs the cheap protocol suite alone (a few thousand tokens); the perf and agentic suites are the expensive parts.
• --context-max 32000 caps the context-ceiling probe; raise it to verify a claimed 128K/200K window (this is the expensive part of the probe).

Everything else

--key sk-…            # or $SPECSPECTOR_API_KEY / $OPENAI_API_KEY; omit for keyless local servers
--json result.json    # machine-readable results
--report              # write specspector-report.md (shareable)
--suites compat,perf  # pick suites
--levels 1,4,8,16,32  # probe higher concurrency
--max-ttft 8000       # p95 turn-latency bar for "useful" concurrency
--vision --reasoning  # opt-in checks
--verbose             # per-check progress

Exit codes: 0 all good · 1 at least one check failed · 2 bad invocation or crash.

Library use

The engine is importable — the planned web app and CI wrappers are thin shells over it:

import { runDoctor, renderMarkdown } from "specspector";

const report = await runDoctor({
  baseUrl: "https://api.example.com/v1",
  apiKey: process.env.OPENAI_API_KEY,
  model: "Qwen/Qwen3-32B",
  levels: [1, 4, 8, 16],
  /* … */
});
console.log(report.scores.compat, report.scores.agent);

Development

npm install
npm test        # offline: the whole pipeline runs against local mock servers,
                # including deliberately-broken ones for every failure mode

No runtime dependencies. The HTTP/SSE layer is hand-rolled on purpose: the product is wire-protocol inspection, and an SDK would hide exactly the behavior being tested.

Roadmap

• v0.2 — hosted reports, badges, and continuous monitoring (daily re-runs with history and regression alerts: "your upgrade improved throughput 23% but broke tool_choice: required"), plus a specspector-cta landing site following the suite's commercial-site convention.
• Config templates for popular clients based on what the endpoint actually supports.

License

MIT — see LICENSE. Part of the Joltify platform.