@energy8platform/stake-math-tools

Node-only dev-time utilities for building Engine lookup tables (force matrices) from raw simulation output. Compresses millions of source simulations into a small weighted table that passes Stake's publish-time validation gates (Liability Limits, Gaps in Hit Rate Table, Unique Events). Companion to @energy8platform/stake-bridge.

Why

Engine games ship a pre-built weighted lookup table: each row is (sim_id, weight, payout_cents) and the RGS samples a row at runtime to decide each round's outcome. The math team's job is to compress millions of raw simulations down to a much smaller weighted table whose aggregate distribution still hits the design's target RTP / volatility / hit-rate under a hard capMaxWin ceiling, and passes Stake's risk-management checks.

This package does that compression in one call.

Two algorithms, one entry point

optimizeLookupTable(rows, params)
       │
       ├─ algorithm: 'tiered'  (default, recommended for Stake)
       │   └─ tier rows by payout magnitude; cap+large rows get weight 1;
       │      small rows get weight W calibrated to preserve cap rate.
       │      Three refinement passes — composition (hit-rate),
       │      RTP-aware partition (mean), Σ-preserving 2-swap (variance).
       │      Stake-Liability-safe by design.
       │
       └─ algorithm: 'nnls'    (legacy, exact target-fitting)
           └─ Lawson–Hanson NNLS over sampled candidates.
              Hits RTP/CV/hit-rate exactly but tends to concentrate
              weight on few rows — typically fails Stake's
              "Within Liability Limits" check on volatile games.

The default is 'tiered'. Pick 'nnls' only when Stake-compatibility is not a concern (custom RGS, internal tooling, etc.).

Architecture (tiered, default)

raw simulations (1M–10M rows)                          lookup table (10K–100K rows)
        │                                                       ▲
        ▼                                                       │
filter (payout ≤ capMaxWin)                                     │
        │                                                       │
        ▼                                                       │
classify by payout multiplier:                                  │
  cap   (pm ≥ capPmThreshold)        weight = 1                 │
  large (largePm ≤ pm < cap)         weight = 1   ◄── rare      │
  small (zero + bulk)                weight = W                 │
        │                                                       │
        ▼                                                       │
sample composition biased by targetHitRate                      │
(n_nonzero / n_zero proportion in small tier)                   │
        │                                                       │
        ▼                                                       │
RTP-aware partition of non-zero small:                          │
   solve  n_high·μ_high + n_low·μ_low = n_B · μ_target          │
   then stratified log-payout sample within each side           │
        │                                                       │
        ▼                                                       │
refineRtpBySwap  — single-row in↔out swaps close the residual   │
                    RTP gap within toleranceRTP budget          │
        │                                                       │
        ▼                                                       │
refineCvBySwap   — Σ-preserving 2-swaps adjust Σ payout² toward │
                    target without disturbing the RTP we just   │
                    achieved (Σ-drift bounded by toleranceRTP)  │
        │                                                       │
        ▼                                                       │
fillStakeRangeGaps — for each Stake distribution range up to    │
                     maxPayout that's empty but source has rows,│
                     swap in a source row. Prevents "Gaps in    │
                     the Hit Rate Table" rejection.             │
        │                                                       │
        ▼                                                       │
diversifyPayouts   — if uniqueEvents < minUniqueEventsRate ×    │
                     nRowsOut, swap duplicate-payout rows for   │
                     source rows with new payout values until   │
                     target unique count reached or RTP budget  │
                     exhausted. Prevents "Insufficient Unique   │
                     Events" rejection.                         │
        │                                                       │
        ▼                                                       │
W = n_high·(1 − target_cap_rate) / (n_small · target_cap_rate)  │
        │                                                       │
        ▼                                                       │
compute stakeReport (top-K, distribution, unique events) ───────┘

Determinism is preserved through a single seed parameter that threads every RNG call.

Install

The package is a monorepo workspace member; consumers inside the repo just import it. It is not published to npm.

Quick start

import { optimizeLookupTable, type LookupRow } from '@energy8platform/stake-math-tools';

// 1. Parse simulation dump (CSV → array). No CSV parser is included on purpose —
//    the math team's pipeline already has one. The input is just Iterable<LookupRow>.
const rows: LookupRow[] = parseCsv('./sim_output.csv');

// 2. Compress.
const result = optimizeLookupTable(rows, {
  targetRTP: 0.96,        toleranceRTP: 0.005,
  targetCV: 8.0,          toleranceCV: 1.0,
  targetHitRate: 0.30,    toleranceHitRate: 0.01,
  capMaxWin: 5_000_000,   // payout cents (50000.00x bet)
  nRowsOut: 100_000,

  // Stake-tuning knobs (recommended for production):
  largePmThreshold: 50,   // pm ≥ 50 → large tier (weight=1). Lower = lower concentration,
                          //   slower convergence. 50–500 is a typical range.
});

// 3. Inspect.
console.log(result.achieved);          // { rtp, cv, hitRate, maxPayout, totalWeight }
console.log(result.toleranceMet);      // booleans per target
console.log(result.maxRowRtpShare);    // top-1 RTP share — Stake Liability indicator
console.log(result.stakeReport);       // full Stake-style report (see below)
if (result.warnings.length) console.warn(result.warnings);

// 4. Write rows out in the format Stake expects: (sim_id, weight, payoutCents)
writeCsv('./lookUpTable_BASE_0.csv', result.rows);

Public API

| Export | Purpose | |---|---| | optimizeLookupTable(rows, params) | Main entry. Dispatches to tiered or nnls. | | buildTieredLookup(rows, params) | Tier-based algorithm directly (bypasses dispatcher). | | transformJsonlZst(opts) | Streaming *.jsonl.zst → *.jsonl.zst transformer with optional line/buffer mapper. Constant memory regardless of input size. See Streaming books rewriter below. | | computeStakeReport(rows, achieved, betCostCents, costMultiplier?) | Compute Stake-style report from a built table. Pass costMultiplier for cost-scaled P(5K)/P(10K) and the ETL>40×cost threshold. | | detectHitRateGaps(distribution) | Find intermediate empty buckets in the hit-rate table. | | computeMetrics(rows) | Weighted RTP / CV / hit-rate / maxPayout. BigInt-safe accumulators. | | bucketize(rows, opts) | Zero / log-spaced / near-max payout partition. | | mulberry32(seed) | Tiny deterministic PRNG. | | weightedReservoirSample(indices, weights, k, rng) | Algorithm A-Res. | | solveNNLS(A, b, opts?) | Lawson–Hanson NNLS with Tikhonov regularization. | | solveQP(A, b, opts) | FISTA + simplex projection (alternative QP solver). | | quantizeWeights(weights, total) | Largest-remainder, wᵢ ≥ 1, exact Σ = total. |

Full types in src/types.ts. Internal helpers (lawsonHansonNNLS, solveLS, …) are not exported.

optimizeLookupTable(rows, params)

Required

| Param | Type | Description | |---|---|---| | targetRTP | number | LUT-RTP target (Σ(w·payout) / (Σw · betCostCents)). E.g. 0.96. For buy-bonus modes, set to gameRtp × cost. | | toleranceRTP | number | Tight tolerance drives refinement-loop precision. E.g. 0.001. | | targetCV | number | Coefficient of variation (volatility). | | toleranceCV | number | Exits CV refinement when gap drops below this. | | targetHitRate | number | Fraction of weighted output landing on payout > 0. | | toleranceHitRate | number | | | capMaxWin | number | Hard cap in payout cents. Rows above are dropped. | | nRowsOut | number | Exact output row count. |

Tier-based knobs (recommended for Stake)

| Param | Default | Description | |---|---|---| | algorithm | 'tiered' | 'tiered' or 'nnls'. | | capPmThreshold | 0.95 × maxPm | pm ≥ this → cap tier (weight 1). | | largePmThreshold | undefined | pm in [largePm, cap) → large tier (weight 1). Set this to lower the top-K RTP share and improve Stake-Liability margin. Typical: 50–500. | | largeTarget | natural rate | Effective P(cap+large) in output. Override with Stake's per-tier limits if needed. | | betCostCents | 100 | Bet cost (1 bet = 100 cents). Used for pm = payoutCents / betCostCents. | | ensureRangeCoverage | true | Run a 4th refinement pass that guarantees every Stake distribution range up to actual maxPayout has ≥ 1 output row when source has rows in it. Prevents "Gaps in the Hit Rate Table" rejection. Set to false to disable. | | minUniqueEventsRate | 0.01 | Minimum fraction of nRowsOut that must be distinct payoutCents values. Stake rejects "Insufficient Unique Events" when too few outcomes exist. 100K output → ≥1K unique. 300K → ≥3K. Set to 0 to disable. When source can't supply enough new payouts, optimizer maximizes under budget and emits a warning. |

Distribution shape

Tail buckets ([2000, ∞)) tend to collapse to …18 → 1 → 1 → 1 → 4 — a starve in middle ranges and a cap-row wall at the top — which makes a reviewer raise eyebrows even though it passes Stake's hard gates. The shape knobs reshape the high-tier sampling so per-bucket row counts follow a log-decay curve.

| Param | Default | Description | |---|---|---| | shapeDistribution | false | When true, the high-tier sample uses bucket-decay-by-Stake-range (each higher bucket targets ratio × previous). When largePmThreshold is unset and this flag is on, the optimizer auto-derives one at max(50, capPmThreshold / 20) so the decay covers multiple buckets. | | shapeDecayRatio | 0.5 | Ratio between adjacent buckets. Lower = steeper tail = lower CV (fewer high-payout rows in the tail). Honest trade-off — typical sweet spot 0.3–0.5. | | shapeAutoMatchCV | false | Implies shapeDistribution=true. Auto-picks shapeDecayRatio via 5-point coarse sweep + bisection refinement so achieved CV lands at targetCV ± toleranceCV. CV(ratio) is U-shaped (very low ratios shrink T → variance climbs back up), so the search isn't a plain bisection. Costs 5–7 full pipeline runs. |

Stake "Within Probability Limits"

| Param | Default | Description | |---|---|---| | maxProb5K | undefined | Maximum allowed P(payout ≥ 5000 × bet) after cost-scaling. When set, the optimizer wraps the pipeline in an auto-retry loop that shrinks largeTarget (and defaults largePmThreshold = 5000 if unset) until the cost-scaled probability lands under the cap. Up to 4 retries. | | maxProb10K | undefined | Same but for the 10000× threshold. Both limits are enforced jointly — each retry shrinks largeTarget by the worse of the two overshoots. | | costMultiplier | 1 | Game/mode cost multiplier (e.g. 250 for BONUS_ADEPT). Applies Stake's leniency scaling to the prob limits (c≥1000 → ×0.2, 500≤c<1000 → ×0.5, 200≤c<500 → ×0.8, else ×1.0) and sets the threshold for the ETL>40×cost check. |

verif.md risk-suite gating

When starRating is set, the optimizer populates Stake's full verif.md limit set as defaults — explicit overrides on the individual fields still win. Currently CVaR / ETL / baseStd / payoutMul / costMul are reported and gated (ToleranceMet.{cvar,etl40xCost,etlP10000,baseStdRange,payoutMultiplierCap,costMultiplierCap} flip false when violated) but not auto-enforced — fixing requires structural changes to the high-tier sample. P(5K)/P(10K) DO auto-enforce via the maxProb5K/maxProb10K retry loop above.

| Param | Default | Description | |---|---|---| | starRating | undefined | 2 or 3. Populates verif.md defaults: maxCVaRNormalized, maxEtl40xCost, maxEtlP10000, minBaseStd/maxBaseStd, maxPayoutMultiplier, maxCostMultiplier, maxProb5K/maxProb10K. 2-star tightens P(10K) less but tightens almost everything else; 3-star is mostly more lenient except a tighter P(10K) and ETL10K. | | maxCVaRNormalized | star default (700 / 800) | Conditional Value at Risk cap — CVaR / betCost ≤ this. CVaR = expected payout in the worst-0.1% tail. | | maxEtl40xCost | star default (0.8 / 0.9) | ETL cap with threshold 40 × costMultiplier × bet. ETL = share of total RTP from rows above the threshold. | | maxEtlP10000 | star default (0.6 / 0.8) | ETL cap with threshold 10000 × bet. | | minBaseStd / maxBaseStd | star default (0.6, 50 / 60) | Bounds on baseStd — only checked when costMultiplier === 1. | | maxPayoutMultiplier | star default (25000 / 100000) | Cap on the maximum payout multiplier in the output. | | maxCostMultiplier | star default (1000 / 1500) | Cap on costMultiplier (game-level — caller's responsibility to set, just gated here). |

Output sizing

| Param | Default | Description | |---|---|---| | requireMaxReached | true | Force ≥ 1 output row close to capMaxWin. | | maxReachedFraction | 0.95 | What counts as "close". | | totalWeightOut | nRowsOut × 1_000_000 | Sum of integer output weights. | | seed | 0xC0FFEE | Deterministic seed for all RNG. |

NNLS-only knobs

| Param | Default | Description | |---|---|---| | maxIterations | 5 | Expand-and-retry attempts on tolerance miss. | | bucketCount | 100 | Log-buckets between min-nonzero and cap. | | minPerBucket | 3 | Min sample slots per non-empty non-zero bucket. | | maxRowRtpShare | 0.05 | Per-row cap on RTP contribution (iterative cap-and-resolve). | | maxWeightPerRow | 10 | Per-row weight ≤ N × uniform-prior. |

Returns

{
  rows: LookupRow[],                  // exactly nRowsOut rows; sim_id preserved
  achieved: {
    rtp, cv, hitRate, maxPayout, totalWeight
  },
  toleranceMet: {
    rtp, cv, hitRate, maxReached,
    rtpConcentration, weightCap,       // NNLS-only constraints
    prob5K, prob10K,                   // verif.md: scaled probability limits
    cvar, etl40xCost, etlP10000,       // verif.md: CVaR + ETL caps
    baseStdRange,                      // baseStd ∈ [min, max] (cost=1 only)
    payoutMultiplierCap,               // payoutMultMax ≤ cap
    costMultiplierCap                  // costMultiplier ≤ cap
  },
  maxRowRtpShare: number,              // largest single-row RTP fraction
  maxWeightRatio: number,              // max weight / uniform-prior
  refinement: {                        // per-pass swap counters
    rtpSwaps,                          // refineRtpBySwap iterations
    cvSwaps,                           // refineCvBySwap (Σ-preserving 2-swaps)
    gapFillSwaps,                      // ensureRangeCoverage swaps
    diversifySwaps,                    // minUniqueEventsRate swaps
    gapsUnfillable,                    // ranges source couldn't fill
  },
  warnings: string[],                  // human-readable issues (gaps, target misses, …)
  stakeReport: {                       // Stake-publish-UI-equivalent metrics
    payoutMultMax,                     // ≡ Stake's "Payout Mult"
    baseStd,                           // ≡ Stake's "Base STD"
    prob5K, prob10K,                   // raw P(payout ≥ 5K / 10K × bet)
    prob5KScaled, prob10KScaled,       // cost-scaled per verif.md — what limits compare against
    costMultiplier,                    // echoed for clarity
    cvarNormalized,                    // expected payout in worst-0.1% tail / bet
    cvarAbsoluteCents,                 // same CVaR but in cents
    etl40xCost,                        // share of RTP from payouts ≥ 40 × costMultiplier × bet
    etlP10000,                         // share of RTP from payouts ≥ 10000 × bet
    topKShare: [{k: 1, share}, …],     // top-1/5/10/100 RTP shares
    hitRateDistribution: HitRateBucket[],  // 16-bucket pm table mirroring Stake's UI
    uniqueEvents: number,              // distinct payoutCents — ≡ "Insufficient Unique Events"
    nonZeroPayouts: number,            // rows with payoutCents > 0 — ≡ "Reasonable Portion of Paying Results"
    betCostCents
  }
}

Never throws on tolerance miss — returns the best-effort result with warnings. Only throws when the filtered input has fewer than nRowsOut rows.

Determinism: same (rows, params) → bit-identical output.

Hit-rate distribution table

result.stakeReport.hitRateDistribution mirrors what Engine shows in the publish UI. 16 payout-multiplier buckets:

[0, 0.1)   [0.1, 1)   [1, 2)   [2, 5)   [5, 10)   [10, 20)
[20, 50)   [50, 100)  [100, 200)  [200, 500)
[500, 1000)  [1000, 2000)  [2000, 5000)  [5000, 10000)
[10000, 20000)  [20000, ∞)

For each bucket: count (rows in range), effectiveHitRate (Σ weight in range / total weight).

detectHitRateGaps(distribution) returns the intermediate empty buckets (sandwiched between non-empty ones) — these are what Stake's "Gaps in the Hit Rate Table" check flags. Empty buckets at the tail (above the highest non-empty bucket) are natural and not flagged.

The optimizer proactively prevents intermediate gaps via the ensureRangeCoverage pass (default on for tier-based): after RTP+CV refinement, any empty intermediate bucket gets a row swapped in from source. If a range can't be filled (source has no rows in that pm range), a warning is emitted — that's a game-design issue your simulation needs to address.

Stake publish-UI mapping

| Stake / verif.md metric | result.stakeReport field | Notes | |---|---|---| | Payout Mult | payoutMultMax | max payout / bet. Capped via maxPayoutMultiplier. | | Base STD | baseStd | stddev in bet units. Gated against minBaseStd/maxBaseStd when costMultiplier === 1. | | Within 5K Probability Limit | prob5KScaled ≤ maxProb5K | scaled value (with cost-multiplier leniency) is what Stake checks. Auto-enforced via largeTarget retry. | | Within 10K Probability Limit | prob10KScaled ≤ maxProb10K | same — scaled, auto-enforced jointly with 5K. | | Within Liability Limits | topKShare[0] (top-1) | usually < 0.05 with largePmThreshold set. | | Risk Limits (CVaR) | cvarNormalized ≤ maxCVaRNormalized | expected payout in worst-0.1% tail. Reported + gated, not yet auto-enforced. | | Liability (ETL, >40× Cost) | etl40xCost ≤ maxEtl40xCost | share of total RTP from rows above 40 × costMultiplier × bet. | | Liability (ETL, P>10000) | etlP10000 ≤ maxEtlP10000 | share of total RTP from rows above 10000 × bet. | | Maximum Cost Multiplier | costMultiplier ≤ maxCostMultiplier | game-level cap (gated only). | | Hit-Rate Distribution table | hitRateDistribution | full match by range; shape control via shapeDistribution + shapeDecayRatio. | | Insufficient Unique Events | uniqueEvents | distinct payoutCents. Auto-driven to minUniqueEventsRate × nRowsOut. | | Reasonable Portion of Paying Results | nonZeroPayouts | rows with payoutCents > 0. | | Gaps in Hit Rate Table | detectHitRateGaps(...) returns [] | tail empties are natural. |

How tolerance flows

Both refinement passes derive their per-iteration Σ-drift budget from params.toleranceRTP so the user's tolerance* values actually drive the precision:

• refineRtpBySwap uses 0.5 × toleranceRTP × T × 100 / W cents of Σ-drift budget.
• refineCvBySwap uses the other 0.5 × toleranceRTP × …, and exits when |Σ²_achieved − Σ²_target| ≤ 2 × targetCV × mean² × T × toleranceCV / W.

Tighten toleranceRTP for sub-percent precision; loosen toleranceCV to let CV refinement exit earlier when the source distribution can't reach the target.

Streaming books rewriter

A pool of raw round books (books_<MODE>.jsonl.zst) typically dwarfs the curated LUT — it's the whole simulation, not just the rows we kept. After the optimizer picks N rows, you need to extract the matching books, optionally renumber their ids into 0..N-1 (Stake convention), and re-compress. transformJsonlZst is the streaming primitive for that step.

import { transformJsonlZst, optimizeLookupTable } from '@energy8platform/stake-math-tools';

const opt = optimizeLookupTable(poolRows, { /* … */ });
const wanted = new Set(opt.rows.map((r) => r.sim));
const idPrefix = /^\{"id":(\d+),/;

const result = await transformJsonlZst({
  inputPath:  'pool/books_BASE.jsonl.zst',          // 1M rounds, possibly multi-GB
  outputPath: 'stake-math/books_BASE.jsonl.zst',    // becomes N rounds
  zstdLevel: 9,
  binaryMapper: (lineBuf) => {
    // Peek only the first ~32 bytes — the id-prefix the simulator writes is
    // deterministic, so we never need to decode the full line. Works for
    // arbitrarily large lines (above V8's ~512 MB string limit too).
    const head = lineBuf.subarray(0, 32).toString('utf8');
    const m = idPrefix.exec(head);
    if (!m) return null;
    if (!wanted.has(Number(m[1]))) return null;
    return lineBuf;                                  // pass through unchanged
  },
});
console.log(`kept ${result.linesWritten}/${result.linesRead}`);

Pipes zstd -dc → mapper → zstd -<level> so the working set is one line — gigabyte books files run with kilobytes of RAM. Spawns the zstd binary (must be on PATH).

| Option | Type | Description | |---|---|---| | inputPath | string | Source *.jsonl.zst. | | outputPath | string | Destination *.jsonl.zst (overwritten). | | mapper | (line: string, i) => string | string[] | null | UTF-8-decoded line. Mutually exclusive with binaryMapper. Throws on lines above V8's ~512 MB string limit. | | binaryMapper | (line: Buffer, i) => Buffer | string | (Buffer | string)[] | null | Raw bytes. Use this when individual book lines may exceed the string limit (bonus games with massive event arrays). Mutually exclusive with mapper. | | zstdLevel | number | 1–22. Default 9 (matches the kitsune pipeline). | | onProgress | (read, written) => void | Fired every progressEveryLines (default 100 000). |

In identity mode (no mapper) the implementation is a pure byte pipe — fastest path (~25 MB/s compressed input on a single core). With mapper / binaryMapper it splits on LF via Buffer.indexOf and runs ~20–25 MB/s for regex prefixes, ~6 MB/s for full JSON.parse rewrites.

Math runtime: SpinML (Rust e8)

e8-math runs game math through the Rust e8 engine (SpinML, Cranelift JIT; fetched by install-e8.mjs from the game-engine repo's Releases, override with E8_BINARY=/path/to/e8):

export default {
  runtime: 'spin',              // default and only supported runtime
  model,
  // raw SpinML source — declarations live inside the .spin file
  luaScript: readFileSync(new URL('./src/game/script.spin', import.meta.url), 'utf8'),
  modes: { BASE: { sim: { iterations: 1_000_000 } } },
} satisfies MathConfig;

Same flag dialect, stdout report, and per-round -dump JSONL as the old Go simulate binary — pool and curate run unchanged, ~25× faster. Master-seeded runs are deterministic independent of the host's core count (rounds are bound to 64 seed lanes), and any dumped round replays bit-for-bit via (rng.server_seed, rng.client_seed, spins[0].nonce).

runtime: 'lua' is rejected with a migration hint — legacy Lua games pin stake-math-tools ≤ 0.8.x. Porting: see docs/lua-to-spin-migration.md.

The two-pass build

The pipeline runs in three stages, and the split is the single biggest thing it does for wall clock:

e8-math pool     --config ./math.config.ts   # light: round,payoutCents per round
e8-math curate   --config ./math.config.ts   # optimize → lookUpTable + selection
e8-math books    --config ./math.config.ts   # heavy: serialize ONLY the selected rounds
e8-math all      --config ./math.config.ts   # all three, end to end

On a real game ~5% of simulated rounds reach the published books. The old one-pass build paid full JSON serialization for 100% of them: 14M rounds became 51 GB that was written, compressed, read back, parsed, and thrown away. Math was never the bottleneck — on kitsune, 1M rounds cost 0.66 s to compute and 4.7 s to print.

So the pool stage no longer prints anything. e8 simulate -dump-csv writes one round,payoutCents line per round — everything curate needs to choose rows, and nothing else. curate optimizes off that and records which pool rounds its rows came from (selected_<MODE>.txt). Only then does the books stage re-run the simulation with -only-rounds, serializing the survivors.

The books stage does not stop at re-simulating: the engine writes the finished Stake books itself (-books-map → {id,payoutMultiplier,events,criteria}, zstd-compressed as it writes). The pipeline used to parse that output and re-serialize every book, which on 200k books cost 1.0 s of parsing and ~2.8 s of JSON.stringify — the data was being serialized twice, the second time in JS.

Re-running is safe because a round is a pure function of (master seed, round index): the lane is round % 64, its seed sha256(master:lane), its nonce round / 64 + 1. Nothing depends on the host's core count or on which rounds were serialized. The engine test asserts a selectively-dumped round is byte-identical to the same round in a full dump, and this package's e2e asserts that a two-pass build produces byte-identical lookUpTable_<MODE>_0.csv and books_<MODE>.jsonl.zst to a one-pass build.

Measured on kitsune, 4M rounds → 200k books, one mode — e8-math all:

| | one-pass (--full) | two-pass (default) | |---|---|---| | wall clock | 63.6 s | 5.5 s | | of which the books stage | — | 0.76 s | | pool on disk | 745 MB | 19 MB | | artifacts | — | byte-identical (200k books, 208 MB) |

Every stage prints what it cost, so a slow run says where it went:

  ⏱  [BASE] pool: 4,000,000 rounds — 2.42s
  ⏱  curate: 1 mode(s) — 1.62s
  ⏱  [BASE] books — 0.76s
  [BASE] books: 200000 row(s) from 200000 round(s), written by the engine; …
Done in 5.47s.

Curation runs in the engine

The optimizer is where this pipeline runs out of memory. Measured at ~200 bytes per pool row — the source array of {sim, weight, payoutCents} objects is 56 of them, the rest is the optimizer's working copies — which puts a 100M-round pool at ~20 GB, past any Node heap.

So curate hands the work to the engine (e8 curate run), which holds the same pool in three integer vectors. It is a byte-for-byte port of tiered.ts, and this package's implementation stays the reference: the two are compared on real pools across all five branches (main path, shapeDistribution, shapeAutoMatchCV, the probability-limit retry loop, the starRating preset), including the curated table, the book map and every number in the report.

| kitsune, 4M pool → 200k books | JS optimizer | engine | |---|---|---| | curate stage | 1.62 s | 0.34 s | | peak RSS of the whole run | 842 MB | 285 MB |

Nothing to configure. An engine without the subcommand falls back to the TypeScript path silently, and E8_CURATE=js forces it — useful when comparing the two. E8_CURATE=engine turns the fallback into an error instead, for a CI that wants to know the fast path is actually running.

NNLS/QP (algorithm: 'nnls') was not ported: every shipped game uses tiered. Those modes stay on the TypeScript path.

transformEvents and the size gate

A transformEvents hook in math.config.ts shapes each book's events before serialization. Applying a JS function to a book means parsing and re-serializing it, so a hook normally puts the mode back on the slow path — every book through Node.

Most such hooks only care about the rare oversized book. paper-duel's deflates spin payloads for books that would breach Stake's 1 MiB events cap, which is a handful out of 400k. Declare that threshold and the books stage skips the rest without parsing them:

transformEvents: transform,
transformEventsWhenBytesOver: 980 * 1024,   // below this, the transform is a no-op

The stage then copies engine-written lines through byte for byte and only opens the ones above the gate (re-deriving their criteria, since a transform may change the events it is derived from). Unset, the hook still works exactly as before — every book parsed, transformed, re-serialized. The console line names which route ran: written by the engine, written by the engine, N rewritten by transformEvents, or assembled here (ungated transformEvents).

The pool manifest

pool writes pool_<MODE>.manifest.json next to the CSV: the master seed the run actually used (math.config.ts need not pin sim.seed — the engine invents one per run, and the heavy pass has to reuse it), the rng profile, action, bet, iterations, params, and a sha256 of the .spin source.

books refuses to run when any of that drifted:

[BASE] the pool was simulated from a different run than the config now describes
(the .spin source changed; iterations 6000000 → 4000000). The books stage would
serialize rounds the lookUpTable never saw. Re-run `e8-math pool --mode BASE`.

Without the check the failure is invisible: the artifacts stay valid JSON, every size and tolerance check passes, and the game deals grids nobody simulated.

--full — the one-pass pool

e8-math pool --full (and all --full) serializes every round up front, as before. Slower and much bigger, but the pool is self-contained: curate alone then yields publishable books, no books stage involved. Use it when you want the whole simulation on disk to inspect, or with an engine that predates -dump-csv. Setting POOL_ZSTD_LEVEL implies --full (it archives a dump, and the light pool has none).

books --keep-dump keeps the heavy pass's filtered dump in the pool instead of deleting it — a few GB at most, since it holds only the selected rounds.

Pipeline throughput

The stages above remove most of the I/O. What is left:

--jobs N — curate several modes at once. Modes are independent from pool onward, so the curate stage runs up to N of them concurrently; reports still print in config order, and the artifacts are identical either way.

e8-math curate --config ./math.config.ts --jobs 4

The default is 1 because the limit is memory, not CPU: each in-flight mode holds its whole source LUT plus the optimizer's working copies (several hundred MB for a 6M-round BASE). Raise it as far as the machine allows — the win is capped by the heaviest mode either way. In all, the pool stage stays sequential: the engine's workers already saturate the CPU, so running modes side by side there measures ~1.0×.

Whatever the engine dumps, it dumps compressed. A -dump path ending in .gz is gzipped as it is written — each engine worker compresses its own part in parallel, and the raw file never exists on disk. This is what the books stage writes, and what --full writes for the whole pool. Compression changes nothing about the run: same seed, same rounds, same bytes once inflated.

Everything downstream reads the archive in-process via zlib — no external binary, and a multi-GB pool never lands on disk inflated. curate accepts books_<MODE>.jsonl, books_<MODE>.jsonl.gz and books_<MODE>.jsonl.zst, in that order, so pools from any era still curate.

An engine that predates gzipped dumps writes plain JSONL to the .gz path. That is detected by the file's magic bytes (not its name), the dump is renamed back to books_<MODE>.jsonl, and the legacy path below takes over with a note on stderr. Update the e8 binary to get the fast path.

POOL_ZSTD_LEVEL — the legacy archive path. Setting it at all opts out of gzipped dumps: the engine writes the raw .jsonl and zstd -<level> packs it afterwards, exactly as before. =0 skips compression and keeps the raw file.

POOL_ZSTD_LEVEL=19 e8-math all --config ./math.config.ts  # pool as a .zst archive
POOL_ZSTD_LEVEL=0  e8-math all --config ./math.config.ts  # pool stays .jsonl

Reach for it when the pool matters as a long-lived archive and the extra ratio is worth the wall clock — zstd -T0 does not parallelize a single stream (~100 MB/s), so on a 50+ GB pool that pass costs ~10 minutes for a file curate reads once. Either way index.json names whichever file was actually written.

A dump the engine already gzipped is never re-packed, even with POOL_ZSTD_LEVEL set: rewriting a multi-GB archive to change its container buys nothing.

The pool and the lookUpTable must describe the same run. A curated row's sim is the dump's line index, so the two files are coupled positionally. curate fails if a selected round has no line in the dump:

[BASE] pool is missing 30 of the 40 rounds the lookUpTable selected (30 curated
row(s) affected) — books_BASE.jsonl[.gz|.zst] does not describe the same run as
lookUpTable_BASE_0.csv.

Without that check those rows ship as books with an empty events array — valid JSON, accepted by every size and tolerance check, and dead on screen. A damaged archive cannot slip through either: a truncated or non-gzip .gz raises unexpected end of file / incorrect header check rather than yielding a short read, and concatenated gzip members are read in full.

Curating off a bare lookUpTable, with no dump in the pool at all, still works — useful while tuning weights — but every book comes out event-less, so the result carries a warning saying exactly that.

Scripts

npm test          # vitest run — full suite (~15s)
npm run typecheck # tsc --noEmit

Design history

• docs/superpowers/specs/2026-05-08-stake-lookup-optimizer-design.md — original NNLS-based design.
• Subsequent commits added the tiered algorithm in response to Stake's "Within Liability Limits" rejection of the NNLS-concentrated output. The tier-based approach is what Stake's reference implementations use; we converged independently on the same algorithm via empirical iteration.

License

MIT