rpc-triage

Diagnose an EVM JSON-RPC endpoint and get evidence you can paste into a support ticket.

npx rpc-triage https://rpc.example.com

  ✗  Persistent block lag, 6 blocks behind references  FRESH-002

  Endpoint    https://eth-mainnet.g.alchemy.com/v2/<redacted>
  Chain       Ethereum Mainnet (1) · Geth/v1.14.11
  References  ethereum-rpc.publicnode.com, eth.drpc.org

  FINDINGS

  ✗  FRESH-002  Persistent block lag, 6 blocks behind references
       Target was behind the reference median in 5 of 5 usable samples, taken
       over 16.0s. Median gap 6 blocks, worst 6.
       runbooks/block-lag.md

  !  FRESH-004  Latest block is 73s old, the same lag as FRESH-002
       Latest block timestamp is 73s behind this machine's clock.
       runbooks/block-lag.md

  ✓  3 checks passed: connectivity, method support, and chain identity

  SAMPLES

  #   spread   eth-mainnet.g.al  ethereum-rpc.pub  eth.drpc.org      gap
  1   41ms     23,499,994        23,500,000        23,500,000        6
  2   38ms     23,499,995        23,500,001        23,500,001        6
  3   44ms     23,499,996        23,500,002        23,500,002        6

  NEXT STEPS

  1. Attach the sample table from this run to the ticket. The per-sample
     timestamps are what distinguish a real lag from propagation noise.

  1 critical, 1 warning, 3 ok · 27 requests · 17.2s · -v for full reasoning
  Reproduce: npx rpc-triage diagnose <your-endpoint> --samples 5

Contents

| | | |---|---| | Install · Why · What it checks | Getting started | | Four rules · False positives | How it decides what to say | | Client identification · History · compare | The hardest features | | Safety · Usage · Configuration · CI | Running it | | Reports · Playground · Layout | Everything else |

Install

Node 20 or newer. No API key, no account, no configuration file required.

npx rpc-triage <endpoint>          # no install
npm install -g rpc-triage          # or keep it around

Why

The expensive ticket in the RPC business is "is it your node or my code?". Most of the time it resolves to the customer's own code, their query pattern, or a documented plan limit. Those tickets are slow because at the point they are opened, neither side has evidence.

This gathers the evidence up front, so a developer can self-diagnose before opening a ticket, and the ticket they do open starts from facts.

It is not a provider audit tool. A free tier returning 429 is the product working correctly, and a tool that calls that a fault has misunderstood how RPC is sold.

What it checks

Every finding has a stable code, and every code maps to one runbook. 43 codes across 13 families:

| Family | Covers | |---|---| | CONN | Reachability, TLS, DNS, redirects | | AUTH | Missing or rejected credentials | | PROTO | JSON-RPC conformance, including responses that are not JSON-RPC at all | | NET | Chain identity, and whether it is the chain you expected | | FRESH | Block lag against references, head age, and whether a lag persists | | CLIENT | Which client is answering, and whether more than one is | | METH | Which methods are served, and which are gated by plan | | HIST | eth_getLogs caps, archive horizon, and plan gates | | RATE | Rate limits, and whether they match what the provider documents | | PERF | Latency against the expected range | | WS | WebSocket connect, subscribe, and time to first event | | REF | Problems with the reference endpoints rather than the target | | BUDGET | The run's own limits, when they stopped it finishing | | TREND | What changed since previous runs of the same endpoint |

rpc-triage codes                 # every code
rpc-triage explain FRESH-002     # what one means

Four rules

"Working as designed" is a verdict. Findings carry a severity, and info means the endpoint did the right thing. A documented plan limit is never reported as a fault. config/provider-errors.yaml holds 43 verbatim provider error strings, each with a source, 15 of them captured off a live endpoint rather than read out of documentation. That registry is what tells a documented cap apart from an unexplained one.

It says "I don't know." inconclusive is a success state. If the reference endpoints disagree with each other, the tool reports that instead of picking a winner. If the local clock is skewed, it says head-age numbers cannot be trusted in this run, and keeps the verdicts that do not depend on them.

Every claim is labelled. observed is a measurement, inferred follows from one, possible is consistent with the evidence, unknown is not established. A reader can tell at a glance which sentences are facts and which are the tool's opinion, because the second kind is where a confident wrong diagnosis costs someone a day.

Findings are countable. FRESH-002 on 14 tickets this month is a pattern; "customers keep complaining about lag" is not.

The false positive problem

Whoever runs this will point it at their own production endpoint. If it invents a block lag, it does net damage. So:

• Target and references are queried concurrently, never in sequence. Sequential reads mean the last endpoint is read later and legitimately shows a higher block.
• Samples whose response spread exceeds 250ms are discarded, not used.
• Reference height is the median, not the maximum. One reference that happens to be a block ahead should not set the bar.
• Staleness thresholds come from measured block gaps, not nominal block time. Scroll's nominal block time is 3 seconds, but it only seals a block when transactions arrive, so a healthy head reaches 93 seconds old. A nominal × 3 rule fires on a working endpoint roughly one run in fifty.
• eth_syncing is never used as proof of anything. It returns false on a node that has stopped following the chain as well as on one that is caught up.
• Calling a lag persistent requires the samples to span more than one block. Five samples two seconds apart cover eight seconds, which is less than one Ethereum block, so they are one observation repeated rather than five. Below that bar the tool reports the lag and withholds the claim about time.

The most important test in the repo asserts that the healthy fixture produces zero warnings.

Which client is answering

web3_clientVersion cannot be trusted. dRPC reports Geth/v10.0.0, a version Geth has never shipped. Cloudflare reports cloudflare-ethereum, naming no client. 1RPC returns an object where the spec requires a string. A gateway rewriting the field is normal operation, not deception, but it means the field cannot answer the question on its own.

So the client is identified from behaviour: the verbatim wording of its method-not-found error, which namespaces it serves, whether it rejects a quantity with a leading zero, and the shape of its eth_syncing reply.

  ✓  CLIENT-001  Behaves like go-ethereum
       observed:  An unknown method returns "the method rpctriage_probeUnknownMethod does not
                  exist/is...", it does not serve erigon_, ots_, health_, and reth_ and it
                  rejects a block tag with a leading zero.
       inferred:  Its wording for an unknown method is go-ethereum's, though 1 other client uses
                  the same words and it numbers the argument in a parameter error. That fits
                  go-ethereum better than Nethermind, the next closest.

Signals that two clients share are weight-divided, so a string Geth and Erigon both emit narrows the field to two rather than naming one. Below a score threshold and a margin over the runner-up, the tool declines to name anything.

It also detects pools

Repeated asks that name different nodes mean more than one machine is answering, which is the answer to a question support engineers get constantly: the same request worked an hour ago and fails now, with nothing changed at the caller.

  ·  CLIENT-005  Answers came from 3 nodes running the same build
       observed:  Asked 4 times what it was, the endpoint named 3 different nodes
                  (3dacdf230cd9, 76e8741e9f15, and e59626f2514c), all running bor/v2.10.0.

Nodes and builds are counted separately, because they are different facts. Geth emits Client/<nodename>/<version>/<os>/<toolchain> when the operator sets a node identity, and that name is per pod. Counting distinct strings and calling them builds reported one Polygon build on eight pods as "8 different builds", with a runbook that told the reader to quote container hostnames into a ticket as build identifiers. Several machines on one build does not make the software ambiguous, so it no longer suppresses the identification.

What was tried and thrown away

An earlier version used the order of the fields in a block, which looked ideal: nothing in the JSON-RPC spec constrains it, no client normalises it, and unlike an error string it cannot be copied by imitating another implementation.

It is wrong, because field order belongs to whatever last serialized the JSON, and on a commercial endpoint that is usually the edge. At one pinned height on a single Nethermind node:

8 single requests    7 alphabetical, 1 nethermind
8 batched requests   8 nethermind        ← batching bypasses the response cache

The variation was a cache reserializing its own hits, so one node looked like a pool of two. Across roughly fifty endpoints, alphabetical order was the most common answer and appeared on Reth, Nethermind, ethrex and OpenEthereum, none of which are written in Go. The signal was removed. The batch trick is documented in runbooks/client-identity.md, because it is still useful by hand when an edge is normalising away the answers.

"It worked yesterday"

A single run can tell you an endpoint is slow. It cannot tell you it got slower, and that second question is the one support actually gets asked. Nobody opens a ticket because an endpoint is slow in the abstract; they open one because something that worked stopped working.

--history records a summary of each run and compares against the previous ones.

rpc-triage <endpoint> --history           # compare, and record this run
rpc-triage <endpoint> --history-dir <dir> # keep it somewhere else
rpc-triage history                        # what is stored
rpc-triage history --clear                # remove it

Four codes, all of them shown here against one endpoint whose previous runs were faster, lagging, and on a different client:

  ·  TREND-001  CLIENT-001 and CLIENT-004 are new since the last run
  ·  TREND-002  FRESH-002 no longer appears
  ·  TREND-003  Round trip is 3.0x slower than this endpoint's own history
  ·  TREND-004  Now running Geth/v1.17.1, previously reth/v1.11.2

  https://ethereum-rpc.publicnode.com/
    5 runs, 2026-08-29 to 2026-08-31
    last: info · Geth/v1.17.1 · p50 363ms

Every trend finding is info, including the ones reporting something getting worse. The severity belongs to the underlying finding: if an endpoint started lagging, FRESH-002 is the critical one and TREND-001 only says when it started. Stacking a second warning on top would count one problem twice and make a run look worse the more history it had, which is the wrong incentive for something you want left switched on.

What a trend cannot know

Every run was measured from wherever it was run. A change has three possible causes: the endpoint, the network path, or the machine you ran it from. Only the first is the provider's problem, and nothing measurable from outside separates them, so TREND-003 names all three rather than implying the endpoint moved. Before escalating one, check the earlier runs came from the same machine on the same connection.

Latency needs three previous runs before it will speak, and the threshold is a doubling rather than a percentage, because round trip from a laptop varies by more than 50% between a wired connection and a coffee shop.

A finding has to be absent from every one of the last five runs to count as new. That is what stops a fault that comes and goes announcing itself as a regression on every other run. "New" here means "not seen before", not "did not exist before".

TREND-004 is usually a routine provider upgrade and needs no action. Check whether CLIENT-005 is also reported first: if the endpoint is a pool of mixed builds, a different member answering looks identical to a change over time and is not one.

Full detail in runbooks/trends.md.

What is stored

Records are summaries, not runs. No response bodies, so a history file cannot contain data returned by someone's private endpoint. Files live in $XDG_STATE_HOME/rpc-triage/history or ~/.local/state/rpc-triage/history, mode 0600, newest 50 per endpoint.

The filename is a hash of the URL and the URL itself is never stored, only its redacted form. Hashing the full URL rather than the redacted one is deliberate: two endpoints at the same provider differ only in their API key, so keying on the redacted display form would file both under one name and compare one against the other's history.

compare, and the strongest signal the tool can produce

diagnose has a target and references, so it asks whether one endpoint is behaving. compare has neither, so it asks where several of them differ.

                           127.0.0.1:62028  127.0.0.1:62029  127.0.0.1:62030
  ----------------------------------------------------------------------------
   chain                   1                1                1
   client                  Geth/v1.14.11    Geth/v1.14.11    Geth/v1.14.11
   latency p50             3ms              3ms              5ms
   height                  23,500,000       23,500,000       23,500,000
  !block hash @23,499,980  0xf6ceb4..d1e3   0xf6ceb4..d1e3   0xbc6af7..0bb6
   balance @23,499,980     0x1bc16d..0000   0x1bc16d..0000   0x1bc16d..0000
   eth_call @23,499,980    0x000000..0000   0x000000..0000   0x000000..0000

  ! rows where the endpoints do not agree

Three fixtures, one of them on a different fork.

The row that matters is the last one. Everything above it can differ for innocent reasons: different clients are normal, latency varies by geography, and a one-block height difference is propagation. A block hash that differs at a pinned height is none of those. A hash commits to the entire block including its state root, so two endpoints returning different hashes for the same height are not serving the same chain.

Pinning is what makes this detectable. Two endpoints reading latest a moment apart are reading different blocks, so any difference proves nothing. At a pinned height there is exactly one correct answer. How far below the shallowest endpoint the pin sits is derived from the chain's block time, targeting roughly two minutes and clamped between 20 and 100 blocks, so it is 20 on Ethereum and 100 on Arbitrum.

Two honest caveats about that pin. Two minutes is past reorg range on Ethereum in practice, where the deepest observed post-Merge reorg is 7 blocks, but it is not a finality guarantee: finality is two epochs, roughly 64 slots. On the other chains it is a heuristic and the tool says so rather than asserting the disagreement cannot be a reorg. And "shallow enough that a full node still holds the state" is true relative to the shallowest endpoint: one endpoint lagging badly drags the pin below everyone, and the state comparison then degrades to a hash check, which the run reports.

Worse than a fork, and rarer, is agreeing on the block hash and disagreeing on state read from it. That means an endpoint is asserting a state root and then contradicting it, so the disagreement is being introduced above the chain rather than by it. The tool separates the two cases and says which it found.

It does not name a winner. With three endpoints and a two-to-one split the majority is where to look first, not proof, and with two endpoints there is no majority at all. A worked example is in docs/examples/divergent-compare.txt.

Safety

The tool takes arbitrary URLs, so it must never be usable as a load tester.

Defaults are 100 requests, 5 concurrent, 30 seconds. The budget lives in the transport, so no probe can opt out. Running out of it produces an inconclusive finding rather than a crash. Redirects are refused, because following one would send a credentialed request to a host the user never supplied. The tool never signs or submits a transaction, and never contacts an endpoint it was not given.

Credentials. Provider API keys live in the URL path, and reports get pasted into public tickets, so the endpoint is not a string. It is a class with the raw URL in a private field and toString, toJSON and the util.inspect symbol all overridden to return the redacted form. Reading the real URL takes a call to unsafeRawUrl(), which greps as a red flag.

Endpoint    https://ethereum-mainnet.core.chainstack.com/<redacted>

Verified against a real 32-character provider key across every output path, including JSON, the invalid-key error path, the WebSocket path, and both generated reports.

Both layers were reviewed by someone whose brief was to break them. They found nine credential leaks and six safety bugs, all fixed, and their tests are kept in the suite.

Usage

rpc-triage <endpoint>                          # the default run
rpc-triage <endpoint> -v                       # every claim, with its label
rpc-triage <endpoint> --report                 # customer-facing Markdown
rpc-triage <endpoint> --escalation             # internal handoff note
rpc-triage <endpoint> --json                   # machine-readable, stable schema
rpc-triage <endpoint> --samples 15             # more samples, for a transient lag
rpc-triage <endpoint> -r <url> -r <url>        # your own references
rpc-triage <endpoint> --expect-chain 1         # fail if it is not this chain
rpc-triage <endpoint> --history                # compare against previous runs
rpc-triage compare <a> <b> ...                 # side by side, including state
rpc-triage explain FRESH-002                   # what a code means

Opt-in probes, none of which run by default:

--historical-probe    # eth_getLogs range cap and archive horizon
--rate-limit-probe    # ramp request rate until the endpoint refuses
--ws <url>            # WebSocket connect, subscribe, time to first event
--aggressive          # raise the ceilings. Asks first.

The default output is short on purpose: the verdict, what was measured, and where the runbook is. -v adds every claim with its epistemic label and the observation ids behind it. Passing checks collapse to one line, because four lines of "this was fine" above a critical finding is noise at the moment it matters least.

Configuration

The reference endpoints this ships with rot. Three that were reasonable defaults when the registry was written stopped working while it was being written: Ankr's keyless tier began rejecting, LlamaRPC went offline, and Blast's keyed endpoints shut down. Without somewhere to put your own, the only recourse is retyping -r url -r url every time, which nobody does, which means running without references and losing the comparison that is the point of the tool.

.rpc-triage.yaml in the working directory, or ~/.config/rpc-triage/config.yaml. The project file wins, so a repo can pin the nodes its team uses.

references:
  1:
    - https://my-node.internal/eth
    - https://backup-node.internal/eth
  8453:
    - https://my-base-node.internal
defaults:
  samples: 5
  maxRequests: 100
failOn: warning

Precedence is the flag, then this file, then the built-ins. Nothing here changes what a finding means: that belongs in code, not in a file the reader of the output cannot see.

CI

| Exit | Meaning | |---|---| | 0 | Nothing above info, including documented plan limits | | 1 | Warnings, or the tool could not determine something | | 2 | Faults | | 64 | Usage error: a bad argument, not a bad endpoint |

--fail-on <severity> replaces that mapping with a threshold you pick. No command waits for input when stdin is not a terminal.

- name: Check RPC endpoint
  run: npx rpc-triage "$RPC_URL" --expect-chain 1 --fail-on warning
  env:
    RPC_URL: ${{ secrets.RPC_URL }}

Two reports, because they have different readers

--report is written to be pasted into a ticket unedited. It leads with a summary that stands alone, keeps the epistemic labels, and carries the reproduction command and a scope note saying what the run could not have seen.

--escalation is the handoff to whoever fixes it. Same evidence, three things the customer report does not have: what is actually being asked, keyed by finding code (because "the endpoint is broken, please look" comes back with questions instead of an answer), what was already ruled out so the next engineer does not start from the beginning, and a blank for what the customer has already been told, because an engineer who contradicts the last reply turns one ticket into two.

Worked examples for both, generated from the fixtures, are in docs/examples.

The playground

npx tsx playground/demo.ts

Stands up RPC endpoints with deliberately injected faults and runs the tool against each. No network, no API key, no real outage, so the demo is reproducible by anyone who clones the repo. The same fixtures are the test suite.

Scenarios: healthy, lagging behind peers, a quiet chain whose head is legitimately old, references that disagree with each other, and a rejected API key.

Testing

npm run check         # typecheck, lint, and 542 tests
npm test              # tests alone

CI runs the same gate on Node 20 and 22.

Layout

src/core/        endpoint redaction, request budget, JSON-RPC transport
src/probes/      measure things, emit Observations, no judgment
src/classify/    pure functions: (Observation[], registries) -> Finding[]
src/registry/    chains, provider errors, client signatures (YAML)
src/report/      terminal, JSON, Markdown
config/          the registries. Editable without a rebuild.
runbooks/        one per finding code family
playground/      fault-injecting fixtures. Test suite and demo.

The split between probes and classifiers is the load-bearing design decision. Probes measure, classifiers judge, and because a classifier is a pure function over recorded data, the false-positive suite runs as fast unit tests instead of only as integration tests against live servers. It also means a real run can be captured with --json and replayed forever as a regression fixture.

ARCHITECTURE.md has the rest. PRD.md has the scope and what was deliberately cut.

Prior art

rpc-doctor (SwiftNodes, published 2026-08-25) covers similar ground in 318 lines, published by an RPC provider as marketing for their own service. The concept is not novel. The difference is in the places this spends its effort: it takes a single reference endpoint and a single sample, and ends on a composite Verdict: HEALTHY with no severity axis, so a documented free-tier limit and a broken node read the same.

Chainbench and Paradigm flood are load testers. Chainbench tells you how fast an endpoint is. This tells you what is wrong with it, and how sure it is.

Status

EVM only, by choice. Half-built adapters for other ecosystems read as abandoned scope rather than ambition, so the chain-adapter seam exists with one implementation.

Not built: the AI drafting layer (--explain).

License

MIT.