llm-interceptor

Captures how developers actually interact with coding agents — what they asked, what they got, whether it worked — without collecting transcripts.

Workflow diagram + package install + test steps: see WORKFLOW.md.
Architecture deep dive: see SETUP.md.

Sits between any OpenAI- or Anthropic-compatible agent and its provider, so it works whether the traffic is billed to your gateway key or the user's own provider key. Analysis produces one compact record per completed task, not per message.

How it works

agent ──▶ proxy (streaming tee) ──▶ upstream provider / your gateway
             │
             └─▶ worker thread ──▶ normalize ──▶ segment ──▶ score ──▶ egress
                                                     ▲
                                        MCP tools ───┘  (agent self-reports intent + outcome)

Three design rules everything else follows from:

1. The hot path is pure I/O. Response chunks are forwarded to the client as they arrive and copied in passing. All parsing and scoring happens on a worker thread after the client already has the full response, so the event loop is never blocked mid-stream.
2. Only the delta is processed. Chat APIs are stateless and resend the whole conversation every turn, so prior turns are hashed and skipped. The next user message is also the evidence about the previous assistant message, which is what makes satisfaction scoring possible without a database.
3. Fail open. If analysis or egress breaks, requests still proxy untouched.

Quick start

npm install
npm run build          # or use the tsx scripts below for development

Run the proxy against a fake provider and prove it adds no latency:

npm run mock           # terminal 1 — fake streaming provider on :9911
npm run proxy          # terminal 2 — interceptor on :8788
npm run bench          # terminal 3 — direct vs proxied time-to-first-byte

npm run baseline starts a no-op forwarding proxy on :8789. Benchmark against that (BENCH_PROXY_URL=http://127.0.0.1:8789 npm run bench) to separate this tool's cost from the cost of the extra network hop — on a Windows dev box the hop dominates at ~16 ms while the capture layer itself adds well under 1 ms.

Exercise the scoring with scripted happy and unhappy conversations:

INTERCEPTOR_IDLE_TASK_CLOSE_MS=3000 npm run proxy
npm run scenario
# wait for the idle window, then read events.jsonl

Pointing real agents at it

Claude Code

$env:ANTHROPIC_BASE_URL = "http://localhost:8788"
claude

Codex CLI — add a provider to ~/.codex/config.toml with base_url = "http://localhost:8788/v1" and select it.

Cursor / OpenAI-compatible clients — override the OpenAI base URL in settings.

GitHub Copilot is not supported. It authenticates to GitHub's own endpoints with no base-URL override, so covering it requires a VS Code extension.

Verify the exact config keys against the agent version you are running; these are internal formats that shift between releases.

MCP server

Register the same binary in MCP mode and the agent gets three tools. It also starts the proxy if it is not already listening, so there is no OS service to install.

{
  "mcpServers": {
    "llm-interceptor": {
      "command": "npx",
      "args": ["-y", "llm-interceptor", "mcp"]
    }
  }
}

• log_task_summary — the agent reports intent and outcome when it finishes a task. Because the model already has the conversation in context, this costs nothing extra and no raw transcript ever leaves the machine. Records land with source: "mcp" and roughly 2.5x the confidence of a purely inferred label.
• submit_feedback — explicit thumbs up/down, the ground truth used to calibrate the heuristics.
• get_my_stats — lets the developer see their own captured data, which matters for adoption.

Add a line to CLAUDE.md or AGENTS.md telling the agent to call log_task_summary when it finishes a task. INTERCEPTOR_INJECT_SYSTEM_PROMPT=1 does this at the proxy instead, covering every platform — disclose it to users if you turn it on.

npm run mcp-smoke drives the server over stdio the way a host would and checks all three tools.

Satisfaction scoring

There is no satisfaction field in any provider API, so it is inferred from behaviour. Current signals, strongest first: switching to a more capable model mid-task, correction language ("no, that's still failing"), rephrasing the original ask, tool error loops, abandonment, and high turn count — offset by positive markers ("perfect, thanks"). Explicit feedback and agent self-reports override the inference and carry much higher confidence.

Two segmentation rules earn their keep: similarity is measured against the task's opening ask rather than the previous turn, because a rephrase restates the original intent; and a low-similarity turn only starts a new task if it is substantive and is not a correction or an acknowledgement.

The token-overlap similarity is a deliberate stand-in for embeddings. Swap in a MiniLM-class ONNX model when the heuristics stop being good enough — the interface is similarity(a, b) in src/score/heuristics.ts.

Validate before trusting any of this. Hand-label a few hundred real conversations and compare. Inferred labels currently ship with confidence around 0.3–0.5 for a reason.

Configuration

All via environment variables.

| Variable | Default | Purpose | | --- | --- | --- | | INTERCEPTOR_PORT | 8788 | Proxy listen port | | INTERCEPTOR_GATEWAY_URL | — | Sends both providers to your gateway | | INTERCEPTOR_UPSTREAM_ANTHROPIC | https://api.anthropic.com | Per-provider upstream override | | INTERCEPTOR_UPSTREAM_OPENAI | https://api.openai.com | Per-provider upstream override | | INTERCEPTOR_GATEWAY_KEY_PREFIX | — | Credentials with this prefix are tagged keyOwner: gateway | | INTERCEPTOR_CAPTURE_MODE | redacted | none, redacted, or full | | INTERCEPTOR_IDLE_TASK_CLOSE_MS | 600000 | Idle window before a task closes | | INTERCEPTOR_EGRESS_URL | — | Collector endpoint; falls back to a local JSONL file | | INTERCEPTOR_EGRESS_FILE | events.jsonl | Local sink, so it runs with no backend | | INTERCEPTOR_EGRESS_RATE_PER_SEC | 5 | Outbound token bucket |

captureMode: redacted scrubs API keys, tokens, JWTs, private keys, emails, IPs, home paths, and high-entropy strings before anything is stored. Only the first and last user message of a task are ever retained, never the full conversation.

Privacy

You are monitoring developers, which carries real legal weight in some jurisdictions and will get you blocked by security teams if it looks like exfiltration. Publish the field list, keep the get_my_stats tool so people can see their own data, make installation reversible, and disclose system-prompt injection if you enable it.

Status

Working: streaming pass-through with measured near-zero overhead, Anthropic and OpenAI request and SSE parsing, session identity, delta extraction, task segmentation, heuristic scoring, bounded fail-open egress, and all three MCP tools.

Not built yet: gateway-side middleware reusing this core, the collector and ClickHouse schema, embedding-based similarity, LLM escalation for ambiguous tasks, and Copilot coverage.