mcp-queue-doctor

An MCP server that diagnoses Postgres job queues — pg-boss and graphile-worker. Retry storms, stuck workers, missed schedules, expiry overruns: what is broken, why, and the safest way to recover.

❌ "3 jobs in enrichment/corpus-fill are in state 'failed'."

✅ "enrichment/corpus-fill failed 140 times over 3m, peaking at 50 failures in a
   single minute. 91% share one error, which looks like an upstream rate limit.
   This is one fault reproduced many times, not many separate faults — so the fix
   belongs at the source, and retrying the jobs individually will reproduce it.

   Recovery, safest first:
     1. Stop enqueuing to this queue — every new job feeds the same failure.
     2. Confirm when the upstream quota resets; treat that as the time to resume.
     3. Add a cooldown gate after N consecutive 429s.
     ⚠ Do NOT bulk-retry yet — the upstream is still limited.

   Evidence: 140 failures, 91% 'HTTP 429 Too Many Requests (daily quota
   exceeded)', peak 50/min, busiest minutes [...], 12 other errors [...]"

The second answer is the product. Every finding carries the evidence it was drawn from, so you — or an agent — can check the reasoning instead of trusting it.

Where the heuristics come from

The rules are extracted from a morning health check that has run daily in production since April 2026 against a pg-boss instance driving ~30 cron queues. Every threshold was tuned by a real false positive or a real missed failure, and each rule below names the incident that motivated it. That provenance is the point: these are not heuristics invented for a README.

Install

npm install -g mcp-queue-doctor

{
  "mcpServers": {
    "queue-doctor": {
      "command": "mcp-queue-doctor",
      "env": {
        "QUEUE_DOCTOR_DATABASE_URL": "postgres://readonly:pw@localhost:5432/app"
      }
    }
  }
}

Then ask: "Is anything wrong with my job queue?"

Want to see it work first? examples/demo spins up a throwaway Postgres and manufactures seven failures in about a minute. It also plants a graphile-worker instance in the same database, where four of the seven rules go deliberately silent — the clearest way to see what capability declaration actually buys you.

Connecting it

The server speaks stdio, so every MCP client starts it as a subprocess. The only thing that varies is where the config lives — and whether that process can reach your database.

Claude Desktop — ~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %APPDATA%\Claude\claude_desktop_config.json on Windows. Use the mcpServers block above, then restart the app.

Desktop launches its subprocesses from the app bundle, not a login shell, so PATH is minimal and a bare mcp-queue-doctor or npx often fails to resolve. Give it an absolute path — which mcp-queue-doctor after a global install, or the absolute path to npx with ["-y", "mcp-queue-doctor"] as its args.

Claude Code — one command, no file editing:

claude mcp add queue-doctor -e QUEUE_DOCTOR_DATABASE_URL=postgres://... -- npx -y mcp-queue-doctor

Add -s project to write a checked-in .mcp.json at the repo root instead of your personal config, so everyone working in that repo gets the tool.

Cloud / remote sessions (Claude Code on the web, and any other headless runner) — a checked-in .mcp.json is the only mechanism that works, because nobody is there to answer an approval prompt. Reference the connection string rather than committing it; Claude Code expands ${VAR} and ${VAR:-default} in .mcp.json:

{
  "mcpServers": {
    "queue-doctor": {
      "command": "npx",
      "args": ["-y", "mcp-queue-doctor"],
      "env": { "QUEUE_DOCTOR_DATABASE_URL": "${QUEUE_DOCTOR_DATABASE_URL}" }
    }
  }
}

Project-scoped servers still need to be trusted before they start. In a headless session that means setting enableAllProjectMcpServers: true in the repo's .claude/settings.json, since the interactive approval never arrives.

Reachability is the real constraint, not the config. The server runs wherever the client runs, and it connects to Postgres directly — there is no hosted component in between. A cloud sandbox can therefore only diagnose a database inside that sandbox: the demo stack, or a dev stack the session brought up itself. A production queue bound to loopback on your own host is not reachable from a sandbox at all, and exposing it to make it reachable is the wrong trade.

Diagnose production from a client on a machine that already has a route to it — your laptop, over an SSH tunnel:

ssh -N -L 5432:127.0.0.1:5432 prod-host

and point QUEUE_DOCTOR_DATABASE_URL at 127.0.0.1:5432. The tunnel is the access grant, it lasts exactly as long as the terminal stays open, and the credentials never leave your machine.

Tools

| Tool | Answers | |:---|:---| | diagnose | Start here. Runs the whole rule catalog, returns ranked findings with evidence and recovery steps | | queue_overview | Per-queue counts by state, stuck jobs, and each queue's expiry/retention/retry config | | failed_jobs | Failures in a window with error messages, plus a per-queue error-frequency breakdown | | stuck_jobs | Jobs active past a threshold, with age, expiry, and heartbeat staleness | | missed_schedules | Cron queues whose latest firing is older than their expression implies | | schedule_status | Every registered schedule with cron, timezone, last firing, and next expected | | job_detail | One job's full record: state, timings, retries, payload, output | | server_info | Connectivity, detected schema, matched dialect, and reduced capabilities |

Schedule expectations are derived from pg-boss's own schedule table by parsing each cron expression, so the common case needs no configuration. The health check this was extracted from carried a hand-maintained list of expected jobs that silently stopped covering whatever nobody remembered to add.

The rule catalog

| Rule | Fires when | Motivating incident | |:---|:---|:---| | retry-storm | Many failures, densely packed, dominated by one error | A daily API quota tipped over and 875 corpus-fill jobs failed in one night. The count suggested 875 problems; the shape showed one | | expiry-overrun | Failure durations cluster at the queue's expiry | A full-corpus sweep couldn't finish inside a 30-minute expiry once upstream throttling slowed it. It reported as a job failure nightly; the fix was an internal wall-clock budget | | stuck-jobs | Jobs active far too long, or heartbeats stopped | A worker killed without graceful shutdown leaves rows active until maintenance reclaims them | | missed-schedule | Latest firing predates the last expected tick | Distinguishes "never fired" (scheduler never booted) from "stopped firing" | | duplicate-registration | A cron queue enqueued twice for one tick | An instrumentation hook invoked job registration twice per process, so every cron ran double for weeks | | retention-window | Failed-row count disagrees with the windowed count | A health email stayed yellow for days after the bug was fixed, counting rows that failed days earlier | | dead-queue | Registered long ago, unscheduled, holds nothing | A producer that stopped, or a registration dropped in a refactor |

Failures are classified (rate_limit, transient_transport, auth, not_found) because the class changes the advice: the right response to a storm of 429s is close to the opposite of the right response to connection resets.

Backends

| Backend | Support | Verified against | |:---|:---|:---| | pg-boss v11+ | Full | 11.1.2 (schema 26), 12.27.0 (schema 37) | | pg-boss v10 | Recognised, refused — see below | 10.4.2 (schema 24) | | pg-boss v9 and earlier | Recognised, refused | — | | graphile-worker 0.17 | Partial, capability-declared | 0.17.3 |

Select with QUEUE_DOCTOR_BACKEND=pgboss (default) or graphile; the schema default follows the backend.

Capabilities, not zeros

Backends don't just name columns differently — they model work differently. graphile-worker deletes a job when it succeeds, has no per-job expiry, no worker heartbeats, and keeps cron expressions in a file rather than the database. So "how many completed in the last day" has no answer there at any price.

Every backend therefore declares what it can answer, and rules that depend on missing data stay silent rather than reporting a zero — a zero reads like a measurement.

| Rule | pg-boss v11+ | graphile-worker | |:---|:---:|:---:| | retry-storm | ✅ | ✅ | | stuck-jobs | ✅ (with heartbeats) | ✅ (age only) | | expiry-overrun | ✅ | — no expiry exists | | missed-schedule | ✅ | — cron lives in a file | | duplicate-registration | ✅ | — no firing history | | retention-window | ✅ | — nothing is retained | | dead-queue | ✅ | — no queue registry |

server_info reports the capability set and spells out each limitation.

Versioned against pg-boss

pg-boss's tables are not a stable API. Across versions it has renamed every timestamp column (createdon → created_on), dropped a whole table (archive, removed in v11), changed a duration from an interval to an integer (expire_in → expire_seconds), partitioned the job table, and added columns (heartbeat_on) that materially change what can be diagnosed.

A tool that hard-codes one shape breaks on the next upgrade — silently, if it is unlucky. That is exactly how the health check this is extracted from spent weeks emitting a confident, wrong "missed schedules" warning that was really SQLSTATE 42P01 after pgboss.archive disappeared.

So schema knowledge lives in one file, src/pgboss/dialect.ts, as data:

• Every relation and column name is declared in a dialect. Query builders emit identifiers from it, so supporting a new pg-boss layout is an edit to that file — no SQL elsewhere mentions a pg-boss table by name.
• Dialects are matched on observed shape, not on a version number. pg-boss's release→schema-version mapping is not published as a contract, and a guessed mapping would reintroduce the very failure this guards against. The version integer is read, reported, and used to say "this is newer than anything we have verified" — but it never decides which SQL runs.
• Optional columns are feature-detected. No heartbeat_on? Stuck-job detection degrades to age-based and says so, instead of failing.
• Unknown layouts are refused, by name. A pre-v10 schema is recognised specifically and rejected with the reason, because diagnosing it against modern queries would silently miss every archived job. A wrong diagnosis is worse than a refusal.

server_info reports the matched dialect, the schema version, whether that version has been verified against real pg-boss, and any reduced capabilities.

This is not a theoretical concern — it has already caught a real bug. The dialect originally claimed a v10 floor, on the belief that v10 removed the archive table. Booting pg-boss 10.4.2 showed the archive table still present and expiry still an expire_in interval, so the dialect was rejecting v10 outright and matching nothing at all for it. The real floor is v11, and v10 now has its own dialect: recognised, and refused by name, because reading the job table alone on v10 silently misses everything already archived.

CI keeps this honest. The integration suite boots pg-boss 10, 11 and 12 into separate schemas and asserts that the observed schema version appears in the dialect's verified list — so a future pg-boss that changes the schema fails loudly rather than running unverified SQL.

Read-only, by construction

Every query runs inside a BEGIN READ ONLY transaction with a statement_timeout and a row cap, and is always rolled back. Recovery actions are recommended, with exact commands — never executed. A confused agent cannot purge your queue, because the database itself refuses the write.

Three independent guarantees, because the failure being guarded against is writing to someone's production queue:

1. BEGIN READ ONLY on every transaction
2. default_transaction_read_only=on at connection level
3. The docs tell you to connect as a least-privilege role — the only guarantee that does not depend on this code being correct

Timeouts bind as parameters via set_config(..., is_local => true) rather than being interpolated into SQL. The schema name — the one identifier that cannot be a bind parameter — is validated against an identifier grammar and quoted.

Log correlation (optional)

Queue state says that a job failed; application logs usually say why. Point the server at a log backend and findings quote the lines behind a failure.

QUEUE_DOCTOR_AXIOM_TOKEN=xapt-...      # read-capable PAT
QUEUE_DOCTOR_AXIOM_DATASET=app-prod
QUEUE_DOCTOR_AXIOM_ORG_ID=your-org
QUEUE_DOCTOR_AXIOM_QUEUE_FIELD=job     # field carrying the queue name

Deliberately optional, and deliberately unable to break anything: a dead log backend never turns a working diagnosis into a failed one, and "we did not look" stays distinguishable from "we looked and found nothing" — otherwise an absent log line reads as evidence of absence. Half-configured settings are a startup error rather than a silent downgrade.

Reaching a database you cannot connect to

Production queues are often the ones you most want diagnosed and least able to reach: Postgres bound to loopback, no port forwarding, only the application in front of it exposed. Opening the database to the network so a diagnostic can connect is a poor trade — the grant is permanent and far wider than the need.

So the server can run its SQL over HTTPS against a read-only SQL endpoint instead:

QUEUE_DOCTOR_HTTP_SQL_URL=https://your-app.example/api/admin/sql
QUEUE_DOCTOR_HTTP_SQL_TOKEN=...

Set these and no connection string is needed; set both and the HTTP transport wins, so an ambient DATABASE_URL cannot quietly become the target. The endpoint must accept {"query": "...", "params": [...]} and answer with {"rows": [...], "truncated": bool}. Reference implementation: showbook's /api/admin/sql.

The safety properties move to the far end, which is an improvement rather than a compromise. The endpoint opens its own read-only transaction, enforces its own timeout and row cap, can rate-limit, can log every query, and can be backed by a role with narrower grants than the application's own — none of which depend on this client being correct. What changes for you: the endpoint's statement_timeout and row cap win over QUEUE_DOCTOR_STATEMENT_TIMEOUT_MS and QUEUE_DOCTOR_MAX_ROWS, a truncating endpoint is reported as truncated rather than silently short, and one diagnose costs roughly a dozen requests against whatever rate limit is in force.

Bind parameters are required, not optional: a client forced to inline its own literals to reach a read-only endpoint would be building an injection sink to get there.

Configuration

| Variable | Default | Purpose | |:---|:---|:---| | QUEUE_DOCTOR_DATABASE_URL / DATABASE_URL | — | Required, unless the HTTP transport is used. Postgres connection string | | QUEUE_DOCTOR_HTTP_SQL_URL | — | Read-only SQL endpoint to query through instead of connecting | | QUEUE_DOCTOR_HTTP_SQL_TOKEN | — | Bearer token for that endpoint | | QUEUE_DOCTOR_BACKEND | pgboss | pgboss or graphile | | QUEUE_DOCTOR_SCHEMA | per backend | Schema the queue was installed into | | QUEUE_DOCTOR_STATEMENT_TIMEOUT_MS | 5000 | Per-query timeout (100–120000) | | QUEUE_DOCTOR_MAX_ROWS | 500 | Row cap per query (1–10000) | | QUEUE_DOCTOR_LOG_LEVEL | info | debug/info/warn/error/silent (stderr) | | QUEUE_DOCTOR_THRESHOLDS | — | JSON object overriding rule thresholds (see below) |

See .env.example. Requires Node.js ≥ 20.11.

Tuning the rules

The thresholds are tuned to the queue these rules were extracted from. That is a defensible starting point and a poor universal answer: a queue that legitimately fails fifty times an hour against a flaky upstream does not have a retry storm, and being told it does every time teaches you to stop reading.

Override any of them with a JSON object — only the keys you set change:

QUEUE_DOCTOR_THRESHOLDS='{"stormMinFailures":50,"idleQueueSeconds":2592000}'

| Key | Default | Governs | |:---|---:|:---| | stormMinFailures | 20 | Failures before a burst counts as a storm | | stormDominantShare | 0.5 | Share one error must hold to be called dominant | | stormPeakPerMinute | 5 | Failures in a minute that mark a burst, not a trickle | | stormCriticalFailures | 100 | Above this a storm is critical, not a warning | | expiryProximity | 0.95 | Fraction of expiry that looks killed rather than failed | | expiryMinJobs | 3 | Jobs at expiry before it is a pattern | | heartbeatMissedMultiplier | 3 | Missed heartbeats before a worker counts as gone | | missedScheduleCriticalSeconds | 86400 | Lateness beyond which a miss is critical | | retentionMismatchMin | 5 | Extra stale failed rows before flagging retention | | duplicateTickMin | 2 | Ticks with duplicate firings before suspecting double registration | | idleQueueSeconds | 604800 | Age at which an empty queue is worth mentioning | | correlatedLogSample | 5 | Log lines attached to a finding as evidence |

An unknown key is a startup error, not a warning — a typo that silently leaves the default in place is the failure this prevents. server_info reports the effective values and which ones you set, so you can confirm an override took.

Publishing to the MCP registry

server.json is the registry manifest. Its version and the npm version it points at are both synced by npm version (see scripts/sync-version.mjs), and a test fails if they drift — a registry entry naming a version that is not on npm sends clients to a 404, which is worse than a stale entry.

Ownership is proved by the mcpName field in the published package.json, so npm must be published first:

npm version patch          # syncs src/version.ts and server.json
npm publish                # the registry reads mcpName off this
mcp-publisher login github # device auth as the io.github.<user> namespace owner
mcp-publisher publish

What this is not

• Not a queue browser. To page through jobs, psql is better.
• Not a dashboard. This is agent infrastructure; your MCP client is the UI.
• Not a Redis queue tool. Both supported backends are Postgres-native, which is what makes the read-only transaction guarantee possible at all. BullMQ and Celery would need a different safety story.
• Not a writer. It will not retry, cancel, or purge anything.

Roadmap

• [x] Read-only database layer, schema probe, CI
• [x] The read-only tool surface
• [x] The diagnosis engine
• [x] A docker compose up demo with a chaos worker
• [x] Integration tests against real pg-boss 10/11/12 in CI
• [x] Log correlation, so findings can cite application logs
• [x] A second adapter (graphile-worker)
• [ ] Opt-in write tools (retry_job, cancel_job) behind an explicit flag
• [ ] Configurable rule thresholds

Development

npm install
npm run verify     # lint + typecheck + test + build

The unit suite drives the database layer through a scripted fake client and the rules through fixtures reconstructing each motivating incident, so npm test runs with no Postgres, no containers, and no network.

The integration suite boots real pg-boss (v10, v11, v12) and real graphile-worker against a live Postgres:

docker run -d -p 55432:5432 -e POSTGRES_USER=qd -e POSTGRES_PASSWORD=qd \
  -e POSTGRES_DB=qd postgres:16-alpine
QUEUE_DOCTOR_TEST_DATABASE_URL=postgres://qd:[email protected]:55432/qd \
  npm run test:integration

It skips itself when that variable is unset, so a contributor without Postgres is never blocked. For a hands-on run, use examples/demo.

License

MIT