da-reader

Read and decode the perps chain's Data Availability records from TypeScript, in real time. Zero runtime dependencies.

The chain's DA layer is an S3 bucket of block-inputs/<n> objects (gzipped JSON wrapping SBE binary frames), where n is exactly the on-chain block height (genesis = 0). This package handles the whole read path:

| layer | module | what it does | |---|---|---| | storage | S3DaReader | SigV4-signed GET/HEAD (no @aws-sdk), gunzip, JSON | | frames | readHeader, decodeFillLeg, decodeTransferOp, … | SBE decode of fill legs (template 3) and transfers (template 60) | | blocks | processBlockInputs, iterateFrames, … | wrapper walk, oracle prices, per-block stats | | stream | followBlockInputs, findTip | ordered, gap-free, self-healing real-time tail | | units | PERP_MARKETS, priceTicksToUsd, … | market dimension + exact unit conversions |

Frame offsets are append-only-stable: the protocol's schema rules only ever add fields at the tail, so a decoder pinned to today's offsets keeps working as the schema grows.

Quickstart

import {
  S3DaReader, s3DaConfigFromEnv, followBlockInputs,
  marketSymbol, priceTicksToUsd, quoteQuantumsToUsd, sizeQuantumsToBase,
} from "@arcus-xyz/da-reader";

const reader = new S3DaReader(s3DaConfigFromEnv()); // DA_S3_* env vars (see .env.example)

for await (const { blockNumber, block } of followBlockInputs(reader, { start: "tip" })) {
  for (const f of block.fills) {
    console.log(
      `block ${blockNumber}: ${marketSymbol(f.marketId)} ` +
      `${sizeQuantumsToBase(f.fillSize, f.marketId)} @ $${priceTicksToUsd(f.fillPrice, f.marketId)} ` +
      `($${quoteQuantumsToUsd(f.notionalQq).toFixed(2)})`,
    );
  }
}

Runnable versions: examples/print-fills.ts (real-time printer) and examples/indexer.ts (durable-cursor indexer skeleton — the pattern for a service that persists blocks).

Following in real time

followBlockInputs(source, opts) is an async generator that yields { blockNumber, atTip, block } strictly in order, never skipping a block:

• start: number (first block yielded), "tip" (resolve the current tip via findTip, yield it first), or { after: n } — resume from a persisted cursor verbatim; the +1 lives in the library so there is no off-by-one to remember.
• Catch-up fetches concurrency blocks in parallel (default 8); a 404 marks the tip (onCaughtUp fires once), after which it polls every pollIntervalMs (default 3s) and drains bursts without sleeping.
• Errors are classified (see below): transient ones retry forever with capped jittered backoff (onError sees every attempt), fatal ones throw out of the generator. Corrupt bodies (bad gzip) get corruptRetries refetches.
• Shutdown: pass an AbortSignal; the generator returns cleanly mid-anything. Breaking out of a for await loop also closes it.
• Gap alarm: if the tail stalls on a missing block while later blocks exist (a writer failure — should never happen), onGap fires after stallProbeAfterMs (default 2 min) so it alarms instead of looking like a quiet chain. The stream still never skips.

Durable-cursor pattern: persist blockNumber after processing each block (ideally in the same transaction as your writes), and restart with start: { after: persisted }. Re-processing a block after a crash-before-persist is possible — make writes idempotent per (blockNumber, fill.msgIndex).

Error taxonomy

| error | meaning | follower behavior | |---|---|---| | null return | 404 — block doesn't exist (the tip) | switch to tip polling | | S3RequestError .isTransient (5xx, 429, SlowDown) | S3 blip | retry with backoff, forever | | network TypeError / TimeoutError | connection trouble | retry with backoff, forever | | S3RequestError 403/400/301 | bad creds / wrong region — never self-heals | throw immediately | | CorruptObjectError | body failed gunzip/JSON (truncated read?) | bounded refetch, then throw | | decode throws (bad schemaId, truncated frame) | schema violation | throw immediately — fail loud |

S3RequestError carries .status and .awsCode (parsed from the S3 XML error body). Clock skew is self-healing: on RequestTimeTooSkewed the reader learns the server offset and re-signs once.

Credentials

Read-only credentials for the DA bucket, via s3DaConfigFromEnv() (DA_S3_*, see .env.example) or an explicit S3DaConfig. No bucket coordinates are baked into the package — supply the bucket, prefix, and region you read from. STS/temporary credentials are supported via sessionToken.

⚠️ The IAM policy should grant s3:GetObject and s3:ListBucket. Without ListBucket, S3 answers 403 (not 404) for missing keys, which breaks "404 = tip" — the follower would treat every tip poll as a fatal error.

Units (exact math)

Wire integers are bigint and stay exact; convert at the display boundary only:

| wire value | unit | to display | |---|---|---| | fillPrice, oracle prices | ticks | priceTicksToUsd(ticks, marketId) | | fillSize | quantums | sizeQuantumsToBase(q, marketId) | | fee, notionalQq, closedPnl, netQuoteBalance | quote quantums | quoteQuantumsToUsd(qq) (= /1e9) |

Per-fill notional = fillSize × fillPrice in quote quantums, exactly, for every market (the wire invariant tickSize × quantumSize = 1e-9). Do not use the frame's filledNotional field for this — it is cumulative per order.

Bigint gotcha: JSON.stringify throws on bigints, so raw ProcessedBlocks can't go straight into a log line or HTTP body. Use processedBlockToJson / fillToJson / blockStatsToJson (bigints → lossless decimal strings).

Aggregating fills correctly

A taker crossing N maker orders emits one aggregated taker leg + N maker legs, and tradeId is not a taker↔maker join key. So: count trades and sum volume from maker legs (role === ROLE.MAKER), sum fees over all legs, and treat deriveBlockStats as the reference implementation of those rules. Addresses come out lowercase-hex; EIP-55 checksumming is left to consumers (needs keccak, which we don't depend on).

Deposits and withdrawals (template 60)

AccountTransferUpdateOperation is the only frame carrying money into or out of an account. decodeTransferOp(frame) decodes one; pair it with iterateFrames, which processBlockInputs does not filter for you:

import { iterateFrames, decodeTransferOp, TEMPLATE, quoteQuantumsToUsd } from "@arcus-xyz/da-reader";

for (const slot of iterateFrames(rawBlockJson)) {
  if (slot.templateId !== TEMPLATE.ACCOUNT_TRANSFER_UPDATE_OPERATION || slot.buf === null) continue;
  const op = decodeTransferOp(slot.buf);
  if (op.movement === null) continue; // op.movementSkipped says why
  const { kind, account, subaccount, amountQq, netQuoteBalanceQq } = op.movement;
  console.log(
    `${kind} $${quoteQuantumsToUsd(amountQq)} -> ${account}/${subaccount}` +
    ` (balance now $${quoteQuantumsToUsd(netQuoteBalanceQq)})` +
    (op.hasRootchainProvenance ? ` [rootchain inbox #${op.rootchainQueueIndex}]` : ""),
  );
}

op is the faithful decode — both account slots, both balances, status, sequence numbers, provenance. op.movement is the derived one-account view, non-null only for an applied deposit or withdrawal; op.movementSkipped names the case when it is null (rejected, not_a_credit, unreadable_account, party_mismatch, non_positive_amount). Three traps this handles that a hand-rolled parser typically does not:

• status is not decoration. Ten rejection codes exist, and when status !== 0 the operation was not applied — its balance and sequence fields describe pre-op state. Crediting one debits an account the exchange never debited. movement is null for those; transferStatusName(op.status) names the rejection for a log line.
• There are two account slots. A deposit is chain→account, a withdrawal is account→chain, so which slot holds the trading account — and which of the two netQuoteBalance fields is its balance — depends on the operation type. The other slot holds the ROBINHOOD_CHAIN sentinel (source/destination report it as { kind: "chain" }); a frame where it is missing is one we have misidentified, and yields party_mismatch rather than a plausible-looking amount.
• rootchainQueueIndex cannot signal its own absence. The rootchain inbox is zero-indexed, so 0 is a valid slot. Provenance comes from a non-zero rootchainPayloadHash; the index is therefore null unless hasRootchainProvenance.

netQuoteBalanceQq is the account's authoritative post-op balance, stamped at apply time — the same class of signal as a fill leg's netQuoteBalance. Resync to it rather than accumulating ±amountQq.

Offsets are derived from the protocol schema and checked against frames produced by the exchange's own reference encoder (tests/t60GoldenFrames.ts), so the offset table is tested against something other than our own reading of it.

Runnable version: examples/print-transfers.ts.

Decoding other templates

processBlockInputs surfaces fills (template 3). Block-inputs also carry templates 23, 28, 31 (PositionUpdate), 32 (FundingTick) and 36. To decode those without forking the wrapper-walking logic:

for (const slot of iterateFrames(rawBlockJson)) {
  if (slot.templateId === TEMPLATE.FUNDING_TICK) {
    // slot.buf is the full frame (64-byte header + body); bring your own offsets
  }
}

readIdString, decodeAccountIdAt and isChainSentinelAt read a char[36] idString slot at any offset, which is most of what a new template needs.

Development

npm install
npm test          # node:test via tsx; fixtures are synthetic frames
npm run typecheck
npm run build     # tsup -> dist (ESM + d.ts)

The package is ESM-only with a Node 22.12 floor — that version can require() an ESM graph, so CommonJS consumers can require("@arcus-xyz/da-reader") without a separate CJS build. Installing straight from the git repo works too; a prepare script builds dist/ on install.

Licensed under Apache-2.0.