pinax-test-suite

Point-and-test RPC & Firehose endpoints for The Graph indexing — a Bun test suite + pinax-test CLI. Framework-free libraries lifted from the Pinax subgraphs dashboard so they can be run, tested and reused on their own; the only runtime dependency is commander (used by the CLI).

bun install
bun test          # run every suite (110 tests)
bun run typecheck # tsc --noEmit

The Firehose CLI/tests shell out to grpcurl — install it if you'll test Firehose endpoints (brew install grpcurl).

What's here

Every test falls into one of three buckets. Subgraphs testing answers "can a Graph indexer actually use this endpoint?"; the general-purpose buckets answer "is this RPC / Firehose reachable, healthy and capable?" regardless of subgraphs.

| Suite | Tests | Category | Validates | |---|---:|---|---| | tests/rpctests.test.ts | 25 | 🟢 General RPC + 🔷 Subgraphs | Capability grade of any EVM/Solana/Bitcoin/Beacon RPC, auth resolver, withQuery gateway flags, ?disallowCached=true cache-bypass — and the mapping from RPC capabilities → which subgraph manifest features can sync | | tests/firehose.test.ts | 38 | 🟠 General Firehose | Concatenated-JSON block parsing, extended-block feature detection + sampled block range, auth/plaintext + ingestability checks | | tests/poi.test.ts | 11 | 🔷 Subgraphs | Proof-of-Indexing cross-check across peers before closing an allocation, and cross-node POI-safety trace diffs | | tests/nodehealth.test.ts | 6 | 🔷 Subgraphs | Head-lag classification (ok / behind / degraded / down) of an indexer's RPC & Firehose nodes | | tests/stress.test.ts | 8 | 🟢 General RPC | Block-range walk-back math + latency percentiles for the rpc --stress throughput tester | | tests/gateway-stress.test.ts | 23 | 🟢 General RPC | Scenario/flag/schedule math for the gateway rpcx stress harness: smooth weighted round-robin, deterministic flag-rate sequencing, node-id normalization, per-chain/per-stage/per-method/error aggregation, latency histogram + over-time timeline | | tests/gateway-report.test.ts | 12 | 🟢 General RPC | The gateway --html report renderer: self-contained/no-CDN output, HTML escaping, chain/node/method/error surfacing, over-time + distribution charts, chain+method+backend filter dropdowns, routing buckets, every-scenario coverage, hover-tooltip marks, deterministic render | | tests/rpc-report.test.ts | 4 | 🟢 General RPC + 🔷 Subgraphs | The rpc --html capability-report renderer: self-contained output, HTML escaping, grade/capabilities/probes/features surfacing, status pills, deterministic render | | tests/registry.test.ts | 6 | 🟢 General | Registry parse/filter (Pinax vs public RPC) + chain resolution by id/alias + CAIP-2 family mapping | | tests/arc.test.ts | 16 | 🟢 General RPC | Arc (arc.io) EVM-difference evaluators: 8-byte base-fee extraData, empty withdrawals, parentBeaconBlockRoot==parent hash, omitted beacon-roots, non-decreasing timestamps, EIP-7708 Transfer logs |

🔷 Subgraphs testing

Indexer-operator oriented — "will subgraphs sync/serve safely here?"

• RPC → subgraph feature mapping (src/lib/rpc/tests.ts: deriveFeatures, deriveGrade) — turns raw RPC capabilities (full / archive / traces) into the subgraph manifest features they enable: eventHandlers, blockHandlers, callHandlers, eth_call from mappings. (in rpctests.test.ts)
• Firehose ingestability (src/lib/firehose.ts: isIngestable, detectExtendedFeatures) — whether a Firehose stream carries the block features a Substreams-powered subgraph needs.
• POI safety (src/lib/poi-peers.ts + src/lib/rpc/poi-trace.ts) — cross-check the Proof-of-Indexing across indexer peers, and diff call-traces between two nodes, so an allocation isn't closed at a divergent POI (which forfeits rewards).
• Node health (src/lib/node-health.ts) — classify a fleet of indexing nodes by how far each lags the best head.

🟢 General-purpose RPC testing

Point at any JSON-RPC endpoint (EVM, Solana, Bitcoin) or the Beacon REST API and get a capability grade — no subgraphs involved.

• src/lib/rpc/probe.ts — dependency-free JSON-RPC / REST client: single + batch calls, latency, three failure modes (transport / HTTP / RPC-error), auth (none / basic / header / configured), withQuery for gateway request flags, endpoint redaction.
• src/lib/rpc/tests.ts (EVM) & src/lib/rpc/chains.ts (Solana / Bitcoin / Beacon) — probe connectivity, full-node, archive, traces & performance → a grade. rpc --html renders that grade as a standalone report via src/lib/rpc/rpc-report.ts (renderRpcReport): the headline grade, capability cards, per-probe latency + outcome + X-Pinax-Node-Id served-by (node / gateway / cache / fallback), and which subgraph features could sync.
• src/lib/rpc/report-ui.ts — shared chrome for the HTML reports (gateway --html, rpc --html): one Pinax-branded stylesheet, stat tiles, generic inline-SVG charts, tables and a hover-tooltip layer, so the reports look and behave the same and don't drift.
• src/lib/rpc/benchmarks.ts — match & diff a run against curated reference node profiles (Geth, Erigon, Reth, …).
• src/lib/rpc/stress.ts — block-fetch throughput test: walk back from head through a bounded worker pool, reporting blocks/s, MB/s and latency percentiles (runBlockStress; pure computeRange / percentiles).
• src/lib/rpc/gateway-stress.ts — scenario-driven stress harness for a Pinax rpc-gateway (rpcx) deployment: open-loop load at a target RPS across many chains (EVM / Solana / Bitcoin method mixes), multi-stage plans (ramps, spikes, soaks) and deterministic per-request gateway flags (?forceFallback, ?upstreamPolicy, ?fallbackAll=false, ?disallowCached). Reads X-Pinax-Node-Id / X-Cache-Status off every response, so the summary shows exactly which nodes served the traffic, the fallback share and cache behaviour per chain and per stage (runGatewayStress, discoverGatewayChains; pure weightedPicker / flagSequencer / summarizeGatewayRun).
• src/lib/rpc/gateway-report.ts — pure renderer that turns a GatewayStressSummary into a standalone, self-contained, Pinax-branded HTML report (inline CSS + hand-rolled inline SVG charts + a tiny vanilla script, no CDN, no external font; black canvas, mint accent): stat tiles, latency + throughput over time, latency distribution histogram, per-chain latency plotted against the request-timeout limit (min–max band + p50/p95/p99 ticks), a routing-distribution composition (Pinax node vs paid fallback vs external cache vs gateway self-answer), per-stage table and errors grouped by chain/error. Chain, method and backend dropdowns (default "All") rescope the tiles + latency charts and a drill-down table — pick a chain for its per-method split, a method for its per-chain split, or a backend (pinax / fallback / cache / gateway) for its per-method and per-chain splits — and filter the chain-keyed detail tables; every mark shows a hover tooltip with its values. Driven by gateway --html (renderGatewayReport).
• src/lib/rpc/arc.ts — Arc-specific EVM-difference checks (8-byte base-fee extraData, empty withdrawals, parentBeaconBlockRoot==parent hash, omitted beacon-roots contract, non-decreasing timestamps, EIP-7708 native Transfer logs). Opt-in via rpc --arc.

🟠 General-purpose Firehose testing

Point at any Firehose endpoint.

• src/lib/firehose.ts — health check (checkFirehose), ingestability verdict (isIngestable), block streaming (streamBlocks), concatenated-JSON parsing (parseConcatenatedJson), extended-block feature detection, auth/plaintext helpers. Point at any endpoint with an X-Api-Key; no graph-node config needed.

CLI tools

pinax-test is a single subcommand CLI (commander) over the suite. Two test commands — rpc and firehose — each test one target: an endpoint URL/host, or a chain id/alias resolved from the registry. gateway stress-tests a whole rpcx deployment across its chains. chains is a discovery helper. Every command has --help. Install the bin once with bun link (or run in-repo with bun run pinax-test <command>):

bun link                                 # exposes the `pinax-test` bin globally
export PINAX_KEY=...                      # or pass --api-key; falls back to $PINAX_API_KEY

# rpc — JSON-RPC feature report (the ?disallowCached=true cache-bypass check is built in)
pinax-test rpc https://eth.rpc.pinax.network/v1/$PINAX_KEY/
pinax-test rpc eth                        # a chain id/alias → resolved from the registry
pinax-test rpc eth --stress               # block-fetch throughput instead (blocks/s, MB/s)
pinax-test rpc eth --reference eth.drpc.org   # compare two (dev vs prod) side-by-side
pinax-test rpc eth --html                  # also write + open a visual HTML report
pinax-test rpc https://arc-testnet.drpc.org --arc   # + Arc-specific EVM-difference checks

# firehose — health + graph-indexer ingestability (needs grpcurl on PATH)
pinax-test firehose eth --deep            # chain id → eth.firehose.pinax.network:443
pinax-test firehose eth.firehose.pinax.network:443 --reference <other-host>

# chains — discovery: list supported chains, or a chain's example endpoints
pinax-test chains
pinax-test chains --service firehose --json
pinax-test chains hyperevm                # copy-paste endpoints for a chain

# gateway — scenario-driven stress test of an rpc-gateway (rpcx) deployment
pinax-test gateway rpcx.<cluster>.pinax.io --discover        # which chains does it serve?
pinax-test gateway rpcx.<cluster>.pinax.io --scenario list   # available scenarios
pinax-test gateway rpcx.<cluster>.pinax.io                   # baseline: light load, every chain
pinax-test gateway rpcx.<cluster>.pinax.io --scenario single-chain --chains eth --rps 50
pinax-test gateway rpcx.<cluster>.pinax.io --scenario spike --chains eth,base,arbone
pinax-test gateway rpcx.<cluster>.pinax.io --scenario fallback-ramp --yes  # ⚠ paid provider
pinax-test gateway rpcx.<cluster>.pinax.io --scenario ramp --html report.html  # visual report

A <target> is a URL / host:port, or a chain id/alias resolved from The Graph Networks Registry (cached in data/registry.json, refreshed daily). --reference runs the same test on a second target and prints a side-by-side comparison (e.g. dev vs prod).

gateway — rpcx stress scenarios

gateway load-tests a whole rpc-gateway (rpcx) deployment rather than one endpoint: it spreads open-loop traffic (fixed target RPS, latency can't slow the sender) across chains with realistic per-family method mixes, and uses the gateway's own request flags to steer routing. Scenarios: baseline, single-chain, fallback (20% forced to the 3rd-party provider), fallback-ramp (0→10→50→100%), spike, soak, policy-mix (?upstreamPolicy=all/archive/full), cache-bypass (?disallowCached=true) and strict-routing (?fallbackAll=false). --rps, --duration and --fallback-rate override any preset; --json emits the full summary and --html [path] also writes a standalone visual report (charts + tables).

The summary is built from response headers: X-Pinax-Node-Id says who actually served each request (a node id like dc1-eth-arch1, or fallback / cache / gateway), X-Cache-Status the cache verdict. Notes: scenarios that force fallback route real traffic to the paid provider and therefore require --yes; eth_blockNumber is answered by the gateway itself when rewrite_tags is on (node id gateway), so a forced-fallback request landing on that method is intercepted before it can reach the fallback pool. A --chains entry the registry doesn't know is probed live against the gateway to auto-detect its family (EVM → Solana → Bitcoin head calls); pass --chains mychain=evm to skip detection.

The knobs

| Flag | What it does | |---|---| | --scenario <name> | preset traffic shape; --scenario list shows all 9 | | --chains eth,base,solana | which chains; default = every chain derivable from the registry. Order matters for single-chain (first wins). Unknown labels: mychain=evm | | --rps <n> | total target requests/sec across all chains (each preset has a default) | | --duration <sec> | total run length; multi-stage presets split it (spike = 30/40/30%, ramp = 4 equal steps) | | --fallback-rate 0.3 | force 30% of requests to the fallback provider on every stage (needs --yes, real cost) | | --full-only | drop the archive-only deep-state probes from the EVM mix — for fleets of full nodes, where those failing is by design (see below) | | --max-in-flight <n> | client-side concurrency cap, default 256 — raise for high-RPS runs (see below) | | --timeout <ms> | per-request timeout, default 10000 | | --json | full machine-readable summary instead of tables | | --html [path] | also write a standalone, Pinax-branded HTML report — stat tiles, latency-over-time, latency distribution, per-chain/per-method latency, routing composition and errors grouped by chain/error, with chain + method + backend filter dropdowns and hover tooltips (inline SVG, no CDN, no external font). Opens in your default viewer automatically. Path optional (auto-named if omitted) |

Try each scenario

One runnable command per scenario, each writing an HTML report and opening it. The preset's default --rps / --duration (and --chains where it matters) are written out inline so you can tweak them without checking --scenario list. Swap $BASE for the gateway you're testing (staging is safest); add --api-key $PINAX_KEY if it needs auth, --https if it's TLS.

BASE=rpcx.<cluster>.pinax.io

pinax-test gateway $BASE --discover      # which chains does it serve?
pinax-test gateway $BASE --scenario list  # all 9 presets + defaults

# steady mixed-method load across every chain
pinax-test gateway $BASE --scenario baseline      --rps 5  --duration 30  --html baseline.html
# hammer one chain (first of --chains) with the full method mix
pinax-test gateway $BASE --scenario single-chain  --rps 20 --duration 60  --chains eth --html single-chain.html
# low load, a sudden burst at --rps, then low again (30/40/30% of --duration)
pinax-test gateway $BASE --scenario spike         --rps 50 --duration 60  --chains eth,base,arbone --html spike.html
# sustained moderate load for a long window (default 600s)
pinax-test gateway $BASE --scenario soak          --rps 10 --duration 600 --html soak.html
# 25% each of upstreamPolicy all / archive / full (25% unpinned)
pinax-test gateway $BASE --scenario policy-mix    --rps 12 --duration 60  --html policy-mix.html
# 30% of requests bypass the rpc_cache (?disallowCached=true)
pinax-test gateway $BASE --scenario cache-bypass  --rps 10 --duration 60  --html cache-bypass.html
# every request with fallbackAll=false — selection failures surface as errors
pinax-test gateway $BASE --scenario strict-routing --rps 10 --duration 60 --html strict-routing.html

# ⚠ these route real traffic to the paid 3rd-party fallback — need --yes (real cost):
# steady load, 20% forced to the fallback provider
pinax-test gateway $BASE --scenario fallback      --rps 10 --duration 60  --yes --html fallback.html
# step the forced-fallback share 0% → 10% → 50% → 100% (4 equal stages)
pinax-test gateway $BASE --scenario fallback-ramp --rps 10 --duration 120 --yes --html fallback-ramp.html

(Not bun linked? Replace pinax-test with bun run scripts/cli.ts.)

Stepping up the load

The sender is open-loop: it fires at the target RPS no matter how slowly the gateway answers. Slow responses show up as rising latency and eventually shed requests, instead of the tool politely backing off — so finding the knee is just stepping --rps:

for rps in 25 50 100 200 400; do
  pinax-test gateway rpcx.<cluster>.pinax.io --scenario single-chain \
    --chains eth --rps $rps --duration 30 --json > eth-$rps.json
  jq -r '[.sent,.ok,.errors,.shed,.actualRps,.latency.p50,.latency.p95,.latency.p99] | @tsv' eth-$rps.json
done

Three saturation signals, in the order they appear:

1. actualRps < target — the sender can't keep pace. First suspect is the client itself: raise --max-in-flight (in-flight ≈ rps × p95-latency-in-seconds, so 200 rps at 500 ms p95 needs ~100+; give it headroom).
2. p95/p99 climbing while p50 holds — upstream queueing is starting; this is the interesting zone.
3. shed > 0, errors or timeouts — past the knee. shed means the client refused to send because the in-flight cap was hit (responses stopped coming back) — at a generous cap that is itself the signal the gateway stopped keeping up.

Other shapes: baseline --rps 100 spreads 100 rps across all chains at once; soak --duration 3600 for an hour-long steadiness check; fallback-ramp --yes steps forced-fallback 0→10→50→100% so per-stage latency compares in-house nodes vs the 3rd-party provider under an identical method mix.

Reading the output

While running (TTY only), a live line ticks every second with counts, in-flight and p50/p95/p99/max over a rolling ~10 s window — so a burst stage shows its own latency immediately rather than being averaged into the whole run. The final summary:

• Load line: sent / ok / errors / shed, achieved rps, wall time.
• Served line: distinct backing nodes, plus two special shares: fallback (the 3rd-party pool — includes organic fallback the gateway chose itself, not just forced) and gateway-intercepted (eth_blockNumber answered without touching any upstream).
• Per stage table: where ramps/spikes tell their story — compare p50/p95 and fallback share across stages under an identical method mix.
• Per chain table: top nodes shows the actual load-balancing spread (node-ids with the random suffix stripped). A chain with fallback(N) high in that list when nothing was forced = no healthy in-house node serving it.
• Per node table (printed when --chains is given, or force with --by-node; always in --json as perNode): served/ok/err + latency percentiles per backing node — pinpoints a sick node behind a chain-level error rate, and puts the fallback provider's latency side-by-side with in-house nodes. Transport failures (timeouts) carry no node id and appear only in the chain totals.
• Exit code is 0 unless >50% of requests failed, so runs are CI-able; --json carries everything (per-chain node maps, cache statuses, error samples).

Archive verification

The EVM mix carries two archive-only deep-state probes (~20% of EVM traffic): eth_getBalance(deep) and eth_getCode(deep) read the zero address's state at a random block in the first half of the chain (≥1024 behind head). Block bodies survive on every full node, so eth_getBlockByNumber(historical) can't tell full from archive — state can: geth/nethermind/besu keep only ~128 blocks of it, so on anything but a true archive these probes fail with a missing trie node / state not available style error. That makes an ordinary run double as fleet verification: a node tagged arch showing errors concentrated on the (deep) methods in the per-node table has pruned state. Caveats: a custom stack that silently returns empty state for pruned blocks (zkSync Era does) passes undetected; and when the fleet under test is supposed to be full nodes, pass --full-only so expected failures don't drown the report. Requests pinned ?upstreamPolicy=full (the policy-mix scenario) never carry an archive probe — failing there would be by design, not a finding.

Cost warning: only forceFallback scenarios are gated behind --yes, but organic fallback also hits the paid provider — e.g. policy-mix on a chain with no full nodes quietly routes its upstreamPolicy=full share to the fallback pool. If a run's fallback share surprises you, find the leaking chain before scaling the RPS up.

Tweaking beyond the flags

Presets live in src/lib/rpc/gateway-stress.ts → GATEWAY_SCENARIOS; each is a function returning stages — add your own and it appears in --scenario list. Method mixes and weights are right above in METHOD_MIXES (e.g. crank the eth_getLogs weight to make each request heavier). For anything the flags can't express — different flag rates per stage, custom chain weights, programmatic sweeps — use the library directly; the CLI is sugar:

// bench.ts — bun run bench.ts
import { runGatewayStress } from "@/lib/rpc";

const summary = await runGatewayStress({
  base: "rpcx.<cluster>.pinax.io",
  chains: [
    { name: "eth", family: "evm", weight: 3 },   // eth gets 3× base's traffic
    { name: "base", family: "evm", weight: 1 },
  ],
  stages: [
    { label: "warm", durationSec: 30, rps: 20 },
    { label: "strict + bypass", durationSec: 60, rps: 100,
      flags: { fallbackAllOff: 0.5, disallowCached: 0.3 } },
  ],
  maxInFlight: 512,
});
console.log(JSON.stringify(summary.perStage, null, 2));

Watching the server side

The client summary shows what callers experienced; the gateway's Prometheus metrics (central Grafana, cluster label selects the deployment) show what the selector did while you loaded it:

rate(rpc_upstream_forced_fallback_total{cluster="<cluster>"}[1m])   # ?forceFallback traffic
rate(rpc_upstream_fallback_total{cluster="<cluster>"}[1m])          # organic fallback (selection failed)
rate(rpc_upstream_selection_errors_total{cluster="<cluster>"}[1m])
sum by(policy) (rate(rpc_upstream_selection_policy_total{cluster="<cluster>"}[1m]))
rpc_upstream_selection_node_count{cluster="<cluster>", network="eth"}  # pool size the selector sees

A good ritual for a serious run: start a soak or fallback-ramp in one terminal and watch fallback_total + selection_errors_total rates in another. Client-side latency degradation with no server-side error movement points at network/proxy, not selection.

Layout

src/lib/
  rpc/            general RPC tester (probe, tests, chains, benchmarks, poi-trace, stress, gateway-stress, arc, index)
  firehose.ts     Firehose tester
  registry.ts     chain registry client (The Graph Networks Registry, Pinax-filtered)
  node-health.ts  indexer node head-lag classification
  poi-peers.ts    allocation POI cross-check
  network.ts      The Graph network-subgraph types/client (used by the POI suite)
  providers.ts    chain-provider discovery from TOML config
  format.ts  timeout.ts  types.ts   small leaf helpers
data/registry.json  cached Pinax chain list (refreshed daily by a workflow)
tests/            the eight suites above
scripts/          cli.ts (pinax-test CLI) · fetch-registry.ts (registry cache)

Import everything RPC from the @/lib/rpc barrel. The src/lib test surface is framework-free (no Next.js / React) and uses only Bun's built-ins; the only runtime dependency is commander, used by the pinax-test CLI.

Provenance: extracted from subgraphs-dashboard's src/lib; the dashboard remains the upstream for these modules.