English | 한국어

ctxprobe

Measure the effective context window (MECW) of any OpenAI-compatible LLM endpoint.

Your 1M-token window is probably not 1M. Models reliably lose the ability to retrieve information well before their advertised context limit, and where that collapse happens depends on your system prompt, your document style, and your provider — not just the base model's spec sheet.

ctxprobe runs a needle-in-a-haystack probe across a grid of context lengths and needle depths against any OpenAI-compatible endpoint, scores the retrieval, and reports a single number — the Maximum Effective Context Window (MECW) — plus a degradation curve and a length × depth heatmap so you can see exactly where things fall apart.

Why

Vendors publish a context limit; that number describes what the model can accept, not what it can reliably use. Retrieval accuracy degrades — often sharply — as prompts get longer, and the point of collapse moves depending on where in the document the answer sits, what the provider does to your prompt under the hood, and how the endpoint is actually configured (a local Ollama server capped at a few thousand tokens will silently truncate a "128k" model's input, for instance). ctxprobe measures that collapse directly against your actual endpoint instead of trusting the spec sheet.

Install

No install needed — run it with npx:

export OPENAI_API_KEY=sk-...
npx ctxprobe scan --endpoint https://api.groq.com/openai/v1 --model llama-3.1-8b-instant

Requires Node >= 18.

Quick start

ctxprobe always shows a token/$ estimate and asks for confirmation before spending a single API call. Pass -y/--yes to skip the prompt (e.g. in CI):

npx ctxprobe scan --endpoint https://api.groq.com/openai/v1 --model llama-3.1-8b-instant

Check the cost first, without touching the network at all:

npx ctxprobe estimate --endpoint https://api.groq.com/openai/v1 --model llama-3.1-8b-instant \
  --input-price 0.05 --output-price 0.08

Re-render report.html/heatmap.svg from a previously saved results.json at any time, with zero API spend:

npx ctxprobe render ./ctxprobe-report/results.json

Methodology

The probe. A single needle type: a uuid stated once inside the haystack ("The secret registration code is <uuid>. Remember this code."), asked for at the end of the prompt. This is deliberately narrow — string-match scoring that can't be gamed by partial credit or fuzzy matching, and uncontaminated by anything else that might already be in the model's training data.

The grid. Lengths expand geometrically (log2) from --min-length to --max-length — e.g. 8k, 16k, 32k, 64k, 128k — because context collapse tends to happen on a log scale, not a linear one. At each length, the needle is planted at --depth evenly-spaced positions from 0% (near the top of the document) to 100% (near the end), each repeated --trials times.

Scoring. A response counts as correct if it contains the exact uuid (dashes/case-insensitive). A response that refuses or contains no plausible answer at all counts as abstain. Anything else — including a different uuid, which is recorded as a confabulation — counts as wrong.

MECW. Following the RULER convention: the largest length whose mean score across all its depths/trials is ≥ threshold (0.85 by default). A dip below threshold at some length shorter than the reported MECW is flagged as an "unstable region" rather than silently ignored — it means retrieval is non-monotonic, which is itself worth knowing.

Calibration. The first response's usage.prompt_tokens is compared against ctxprobe's own local token count to get a provider/local ratio, which retargets later lengths so the actual prompt sent lands where you asked. In a live 128k scan of meta-llama/llama-3.1-8b-instruct via OpenRouter, that ratio landed within 1.28% of ctxprobe's local count — well inside the ±2% band that keeps it pinned to exactly 1.0. If the ratio is suspiciously low (< 0.6), ctxprobe assumes the endpoint is silently truncating the prompt (the classic case: a local Ollama server with a num_ctx far below what you're sending) and aborts with an explicit message rather than reporting a misleadingly low MECW — confirmed against a real Ollama server (num_ctx 32k) sent a 64k prompt, which produced a calibration ratio of 0.50 and an immediate abort.

Sanity gate. Before spending the full grid, ctxprobe runs every depth/trial at the smallest length first. If that doesn't already score above 0.95, the probe or harness itself is broken — not the model — and the scan aborts immediately with that explanation, without ever reaching for the larger (more expensive) lengths.

The filler caveat — read this before trusting a MECW number

The "haystack" filling the space around the needle is plain, topic-neutral noise — short, unrelated sentences about nothing in particular (a deliberately original, copyright-free sentence bank, not a real essay or document corpus). This is a real methodological trade-off, and it cuts one way: noise filler is easy to see past. A model doesn't have to work hard to notice one out-of-place sentence sitting inside pages of generic prose, so retrieval against pure noise is close to a best case.

That means MECW as reported here is a ceiling, not a floor. It tells you the most context a model/endpoint can be expected to use reliably. Your actual documents — code, logs, contracts, chat transcripts — are full of things that look enough like the answer to distract the model (near-duplicate IDs, similar-sounding facts, restated-but-wrong versions of the same claim), which is exactly the harder "distractor" and "semantic" needle tiers that RULER-style benchmarks add on top of pure noise. Those tiers are out of scope for this v0 — plain-uuid-in-noise is the only probe type shipped — and are called out here as a v1 candidate (see Roadmap), not silently glossed over.

Output artifacts

Every scan run produces four artifacts:

• the terminal report — printed on completion: the MECW banner, a length × depth summary table, and the paths of everything written to disk.
• results.json — the full machine-readable result (config minus the API key, calibration, every trial, per-length aggregates, and the MECW verdict). Written even if the scan aborts partway through, so a rate limit or a wifi drop never throws away work already paid for.
• report.html — a single self-contained HTML file (no external requests, works from a plain file:// URL): the MECW score, the degradation curve, and a hover-able length × depth heatmap.
• heatmap.svg — the same score block + curve + heatmap as a single standalone SVG, for dropping straight into a README, a Slack message, or a Show HN post without opening the HTML report.

All three files are written into --output (default ./ctxprobe-report/).

Example terminal output

A real 8k–128k scan of meta-llama/llama-3.1-8b-instruct via OpenRouter — every length up to 64k retrieves perfectly, 128k is a hard context limit:

  MECW ≈ 64k  (threshold 85% correct)
  hard context limit at 128k

len     d0 d25 d50 d75 d100  mean
---------------------------------
8k     100 100 100 100 100  100%
16k    100 100 100 100 100  100%
32k    100 100 100 100 100  100%
64k    100 100 100 100 100  100%
128k     ×   ×   ×   ×   ×    0%

  artifacts:
    ./ctxprobe-report/results.json
    ./ctxprobe-report/report.html
    ./ctxprobe-report/heatmap.svg

A truncation-detecting abort looks like this instead — a real run against a local Ollama server (num_ctx 32k) sent a 64k prompt:

  MECW ≈ 64k  (threshold 85% correct)
  calibration ratio 0.50 (provider/local tokens)

len    d100  mean
-----------------
64k    100  100%

  artifacts:
    ./ctxprobe-report/results.json
    ./ctxprobe-report/report.html
    ./ctxprobe-report/heatmap.svg

  scan aborted (truncation): endpoint reported far fewer tokens than sent
  (ratio < 0.6) — the prompt is being silently truncated. Check the server
  context limit (e.g. Ollama num_ctx).

If the sanity gate itself fails (Phase 0 mean score below 0.95 at the smallest length), ctxprobe aborts before ever reaching for the larger, more expensive lengths with scan aborted (sanity): Phase 0 mean score <score> < 0.95 at <length> tokens — the probe or harness is broken, not the model.

CLI reference

ctxprobe scan

Runs the full probe grid and writes a report. This is the default command (ctxprobe --endpoint ... --model ... also works).

| Flag | Default | Meaning | | --- | --- | --- | | --endpoint <url> | (required) | OpenAI-compatible base URL | | --model <name> | (required) | model name to probe | | --api-key <key> | $OPENAI_API_KEY | API key | | --min-length <tokens> | 8k | smallest context length sampled | | --max-length <tokens> | 200k | largest context length sampled | | --depth <n> | 5 | number of needle-depth samples per length (evenly spaced 0–100%) | | --trials <n> | 2 | trials per (length, depth) cell | | --concurrency <n> | 4 | max concurrent requests | | --threshold <ratio> | 0.85 | mean-score threshold a length must clear to count toward MECW | | --output <dir> | ./ctxprobe-report | where results.json / report.html / heatmap.svg are written | | --input-price <usd-per-1m> | — | input token price (for the pre-run cost estimate) | | --output-price <usd-per-1m> | — | output token price (for the pre-run cost estimate) | | -y, --yes | false | skip the pre-run cost estimate confirmation |

Token shorthand: anywhere a length is accepted, "8k" means 8 * 1024 = 8192 tokens — k is always 1024, never 1000.

ctxprobe estimate

Computes the same token/$ estimate scan shows you before running, entirely locally — no API key, no network call. Useful for sizing a run before you commit to it.

| Flag | Default | Meaning | | --- | --- | --- | | --endpoint <url> | (required) | label only — not called | | --model <name> | (required) | label only — not called | | --min-length <tokens> | 8k | same meaning as scan | | --max-length <tokens> | 200k | same meaning as scan | | --depth <n> | 5 | same meaning as scan | | --trials <n> | 2 | same meaning as scan | | --input-price <usd-per-1m> | — | input token price | | --output-price <usd-per-1m> | — | output token price |

ctxprobe render <resultsPath>

Reload a saved results.json and re-render report.html/heatmap.svg, with zero API spend. Accepts --output <dir> (default: alongside the json). Intentionally left out of --help — it's a re-render utility, not a primary workflow — but fully supported.

The estimate is a cache-unaware upper bound: it does not model the provider prompt-cache savings ctxprobe's own cache-friendly cell ordering is designed to earn, so a real scan billed with caching will typically cost less than the number estimate prints.

Architecture

config → grid → phase0 (sanity gate + calibration) → phase1 (cache-aware grid) → aggregate → render (terminal / html / svg / json)

probe.ts is the one designed extension point: a needle type is exactly the 3-function interface makeNeedle(rng) / renderPrompt(parts) / scoreResponse(needle, text). The v1 distractor/semantic tiers (see Roadmap) are meant to land as sibling files next to it, not a rewrite.

Full file-by-file breakdown: AGENTS.md.

Testing

pnpm test    # vitest, 175 tests across 14 files, deterministic, no network

Unit tests are colocated (*.test.ts next to the module) and cover filler determinism, haystack token targeting, grid expansion, probe scoring, MECW aggregation, tokenizer calibration math, and SVG output. The endpoint and runner are tested against AI SDK's MockLanguageModelV2 — no real network calls, no API key needed to run the suite.

Roadmap

v0 (shipped): uuid-in-noise probe, OpenAI-compatible endpoints only, log2 grid, pre-run cost estimate, terminal/HTML/SVG/JSON output.

v1 candidates:

• Distractor and semantic needle tiers (see the filler caveat above)
• Adaptive binary search across the length grid, instead of a fixed log2 sweep
• Per-cell error outcome surfaced distinctly from context_exceeded
• Native Anthropic endpoint support (alongside OpenAI-compatible)

Contributing

This repo's conventions — build/test/lint commands, colocated test style, the probe extension-point interface, known gaps — are documented in AGENTS.md. Read it before sending a PR.

License

MIT