@nodii/retry

The retry / exponential-backoff / jitter policy for the Nodii stack, as a real package.

Retry+backoff+jitter was hand-rolled 14 times inside ts/* itself — not in the services, in the shared libraries. Every service inherited whichever variant its lib happened to implement, so two consumers of the same fleet behaved differently under the same failure.

Zero runtime dependencies. Injectable clock and RNG throughout — a suite exercising a 30s cap finishes in microseconds.

The survey

Every row is a real implementation on origin/main @ 8af8e137. FLEET_POLICIES in src/fleet-policies.ts encodes each one, and tests/fleet-parity.test.ts asserts this package reproduces it.

| # | Site | Base | Mult | Cap | Jitter | Max attempts | Retryable | |---|------|------|------|-----|--------|--------------|-----------| | 1 | approval/consumer.ts:161 (BullMQ backoffStrategy) | 250ms | ×2 | 60s | none | 5 (consumerMaxRetries) | any handler throw | | 2 | approval/streams-reader.ts:389 | 200ms | — | — | none | ∞ | any loop throw | | 3 | onboarding/submit/client.ts:37 | 1000ms | ×2 (literal [1000,2000]) | — | none | 3 | network/timeout + 5xx | | 4 | onboarding/submit/signup-client.ts:44 | 1000ms | ×2 (literal [1000,2000]) | — | none | 3 | network/timeout + 5xx | | 5 | outbox-dispatcher/sql-builders.ts:39 (SQL cooldown) | 60s | linear ×n | — | none | 8 (publishMaxAttempts) | every publish failure | | 6 | outbox-dispatcher/drainer.ts:421 | 200ms | — | — | none | ∞ | any drain throw | | 7 | replica-consumer/start.ts:242 (cold start) | 1000ms | ×2 | 30s | ±20% | 0 = indefinite | every bootstrap failure | | 8 | replica-consumer/define-replica.ts:294 + init.ts:17 | 100ms | ×2 | 30s | jitterPct: 25 | 5 | n/a — DEAD CONFIG | | 9 | replica-consumer/worker/streams-worker.ts:1120 | — (PEL redelivery) | — | — | none | 5 | all apply errors → DLQ | | 10 | role-catalog/publish.ts:203 | 100ms | ×2 | 30s | ±12.5% | 30 | gRPC UNAVAILABLE only | | 11 | saga/participant-worker.ts:351 | 250ms | — | — | none | ∞ | any loop throw | | 12 | telemetry/worker/streams-worker.ts:776 | — (PEL redelivery) | — | — | none | 3 | poison-tagged only | | 13 | telemetry/worker/outbox-dispatcher.ts:262 | 200ms | — | — | none | ∞ | any drain throw | | 14 | telemetry/worker/single-active-lease.ts:714 | max(1s, ttl/2) | — | — | none | ∞ | lease not held / Redis down |

The divergences

Accidental

1. jitterPct carries two different UNITS inside one package. replica-consumer/start.ts writes 0.2 (a fraction); define-replica.ts and init.ts write 25 (a percent) for a field of the same name in the same library. Read as a fraction, 25 is ±2500% jitter. This package rejects pct > 1 outright so the ambiguity cannot survive migration.
2. retryBackoff is DEAD CONFIG. git grep retryBackoff -- ts/ returns three hits: two constructors and one type. Nothing reads it. replica-consumer declares an exponential 100ms→30s ±25% apply backoff that has never executed — the real spacing is Redis PEL redelivery plus the XCLAIM reaper. Do not "migrate" it; migrating would ACTIVATE a delay that has never run in production.
3. Three structurally identical loop-error sleeps, two constants. approval/streams-reader, outbox-dispatcher/drainer and telemetry/worker/outbox-dispatcher all sleep 200ms; saga/ participant-worker sleeps 250ms. Same failure mode (Redis blip in a stream-read loop), same remedy, arbitrary difference. None has jitter or growth, so a sustained Redis outage means every pod issues 4–5 commands/second forever.
4. Both jitter sites can exceed their own declared cap. Both apply jitter AFTER the cap and never re-clamp: role-catalog sleeps up to 33 750ms against a declared 30 000ms cap; replica-consumer cold start up to 36 000ms against the same declared cap. A cap the code may exceed is not a cap.
5. approval's declared 250ms base is never slept. 2 ** attemptsMade * 250 against BullMQ's post-incremented counter means the first retry waits 500ms. Off by one doubling.
6. Three jitter magnitudes (12.5% / 20% / 25%) for one concept, none of them derived from anything.
7. single-active-lease self-heals on a fixed ttl/2 tick with zero jitter. This is the one genuine thundering-herd shape in the survey: after a Redis blip every replica of a service retries the acquire path in lockstep.

Deliberate (leave alone)

1. replica-consumer cold start retries indefinitely (maxAttempts: 0). Giving up means silently dropping a backfill. Correct.
2. outbox-dispatcher's cooldown is LINEAR, not exponential — and publishMaxAttempts caps RE-CLAIM, never deletes the row (D401). A row is source of truth; the 24h sweep raises the M17 page. Correct, and the reason growth: "linear" exists in this package.
3. onboarding does not auto-retry 429. Honouring a Retry-After is the caller's decision. Correct.
4. role-catalog retries UNAVAILABLE only, fail-fast on FAILED_PRECONDITION. A version mismatch will never heal by waiting. Correct.
5. telemetry's retry ledger is bumped only for poison-tagged errors; infra errors (fence lost, store down) leave the entry in the PEL without spending budget. Correct — otherwise five Redis blips silently dead-letter a good event.
6. replica-consumer treats UnknownAggregateKind as process-fatal, not retryable. A silent drop there is data loss. Correct.

Why equal jitter is the default

"equal" — d/2 + U(0, d/2).

• Full jitter (U(0, d)) minimizes contention but has no floor: any attempt can return ≈0. In this fleet almost every retry waits on a single dependency that is booting (role-catalog publishing at boot against a cold tenant-service; replica cold-start backfill against the snapshot producer), not on a contended shared resource. Full jitter's benefit is herd dispersal; its cost — losing the monotonically growing floor — is what actually bites, because a near-zero delay both burns an attempt from a finite budget and adds load to something already struggling.
• Equal jitter keeps an exponentially growing floor (d/2) while still dispersing enough to break lockstep between the N pods of a service that boot together — which is exactly the herd case here (an ECS rolling deploy). AWS's own analysis finds full and equal near-identical on total work and completion time; equal keeps the floor.
• Decorrelated jitter (min(cap, U(base, 3·prev))) is the strongest for pure contention, but it is stateful and also non-monotone, and nothing in the fleet does it today — defaulting to it would silently change all 14 sites. Offered, never defaulted. (bullmqBackoffStrategy rejects it outright: BullMQ's backoffStrategy is stateless and has no previous delay to thread.)
• { kind: "proportional", pct } exists because it is what the two live jitter sites actually do. role-catalog's randomFn()*0.25 + 0.875 is exactly 1 + 0.125·(2r−1), so their migration is byte-preserving for the same RNG draw.
• "none" exists because 11 sites have no jitter and their migration must be behaviour-preserving.

Order of operations — the part the fleet gets wrong:

grow → cap → jitter → clamp (unless allowJitterAboveCap) → floor at 0 → round to integer ms

allowJitterAboveCap: true reproduces the live overshoot exactly, for a byte-preserving migration. Default false, i.e. the declared cap holds.

The BullMQ off-by-one

BullMQ increments job.attemptsMade before the backoff is computed and before failed fires. So after the k-th attempt fails, attemptsMade === k, and "was that the last one?" is:

isFinalAttemptFromAttemptsMade(attemptsMade, maxAttempts); // attemptsMade + 1 >= maxAttempts

The naive attemptsMade >= maxAttempts disagrees at exactly one point (attemptsMade === max - 1) and runs one attempt too many. The 1-based in-process ledger convention (retries.inc(id)) is a different function:

isFinalAttempt(attempt, maxAttempts); // attempt >= maxAttempts

Two named functions, deliberately — there is no single isLastAttempt(n, max) for a call site to guess at. maxAttempts === 0 means unbounded; nothing is ever final.

The boundary with @nodii/idempotency

They answer different questions and must not be merged:

| | @nodii/idempotency (D242) | @nodii/retry | |---|---|---| | Question | should the KEY be cached or released? | should the CALLER re-invoke? | | Kind | storage / replay | control flow | | Side | server | client | | Keyed by | the completed OUTCOME | the thrown ERROR |

transport is a superset of "retryable": RESOURCE_EXHAUSTED is transport, yet a caller honouring a Retry-After may decline to retry (@nodii/onboarding does exactly this for 429). And CANCELLED / UNKNOWN / UNIMPLEMENTED / DATA_LOSS are deliberately unmapped by D242 and fall back to defaultClass — a default that exists to answer the storage question and must not silently become a retry verdict.

So this package ships no gRPC-status table, no HTTP-status table, and no dependency on @nodii/idempotency — a change to D242's map cannot silently change anyone's retry behaviour. Coupling is explicit and one-way:

shouldRetry: shouldRetryFromOutcomeClassifier(myD242Classifier); // transport ⇒ may retry

The reverse direction is forbidden and not offered: declining to retry does not make an outcome deterministic, and caching a key on that basis would replay a transport failure as a business result. The bridge fails closed on any unrecognised class.

Usage

import { retry, NonRetryableError, bullmqBackoffStrategy } from "@nodii/retry";

const res = await retry(
  async (ctx) => {
    const r = await fetch(url);
    if (r.status === 403) throw new NonRetryableError("bot challenge failed");
    if (r.status >= 500) throw new Error(`server ${r.status}`);
    return r.json();
  },
  {
    policy: { maxAttempts: 3, baseDelayMs: 1000, multiplier: 2, jitter: "equal" },
    shouldRetry: (err) => !(err instanceof TypeError),
    onRetry: ({ attempt, delayMs, error }) =>
      logger.warn("retrying", { attempt, delay_ms: delayMs, error: String(error) }),
  },
);

// BullMQ — the post-increment handled in one place
new Worker(queue, handler, {
  settings: {
    backoffStrategy: bullmqBackoffStrategy({
      baseDelayMs: 500, multiplier: 2, maxDelayMs: 60_000, jitter: "equal",
    }),
  },
});

Everything is injectable: sleep, random, signal. backoffSequence(policy, n) gives the deterministic schedule for tests and for operators reasoning about "how long until we give up".

Migration

Call sites are not migrated in this release. FLEET_POLICIES[<id>].migration states, per site, whether the migration is behaviour-preserving or a deliberate change, and tests/fleet-parity. test.ts proves the preserving ones really are.