@recoengine/eval

Offline evaluation for recoengine. The half of "tune the weights yourself" that the other packages leave to you.

The engine explains why an item is where it is. That answers what happened. It does not answer the question you actually act on: is this configuration better than the last one? Weights are set by hand here — that is the design — and setting them by hand without a number to check them against is guessing with extra steps.

npm i -D @recoengine/eval

A development tool. It is deliberately not re-exported from the recoengine facade: it belongs in devDependencies, not in the bundle you ship to a device. The facade does depend on it for one thing — the recoengine try CLI — and nothing reachable from import … from 'recoengine' touches it, so no bundler that follows your imports will include a line of it.

Use

import { CLOCK, createEngine, timestamp } from 'recoengine'
import {
  averagePrecision, catalogueCoverage, evaluate, formatReport,
  ndcg, precisionAt, recallAt, replayClock, splitByTime,
} from '@recoengine/eval'

// 1. Split the event log by time. Never at random — see below.
const data = splitByTime(events, { at: timestamp(Date.parse('2026-06-01')) })

// 2. Build the engine on a clock the evaluation controls.
const clock = replayClock(data.at)
const engine = createEngine<Track>().use(/* … */).provide(CLOCK, clock).build()

// 3. Replay and score.
const report = await evaluate(engine, data, {
  k: 10,
  metrics: [ndcg(), precisionAt(), recallAt(), averagePrecision()],
  corpusMetrics: [catalogueCoverage()],
  clock,
})

console.log(formatReport(report))

users 4820  ·  excluded 391  ·  catalogue 12043
  ndcg@10        0.3142
  precision@10   0.0871
  recall@10      0.2265
  map@10         0.1904
  coverage@10    0.4417

The cold-start slice

One mean hides its own worst case. Pass segments and the users the engine barely knows get their own row:

import { coldStart, warm } from '@recoengine/eval'

// Keep the users with no history at all — the default split drops them.
const data = splitByTime(events, { at, minTrain: 0 })

const report = await evaluate(engine, data, {
  k: 10,
  metrics: [ndcg(), recallAt()],
  segments: [coldStart(), warm()],
  clock,
})

users 4820  ·  excluded 12  ·  catalogue 12043
  ndcg@10        0.3142
  recall@10      0.2265
  cold_start  ·  611 users
    ndcg@10      0.0904
    recall@10    0.0712
  warm  ·  4209 users
    ndcg@10      0.3467
    recall@10    0.2491

Cold start is the part of an algorithmic recommender that gets assumed to work rather than shown to: popularity is in the mix, so an empty history probably still produces a sensible page. Probably is not a contract. The slice turns it into a number that moves when a weight changes and that a regression cannot cross quietly.

Two things decide whether the row means anything:

• minTrain: 0. The default split drops users with no train events — the genuinely cold ones — before the segment ever sees them, which makes the cold-start row look better than the product does.
• An empty slice keeps its row. cold_start · 0 users says the split kept nobody cold, which must not read the same as "cold start is fine".

coldStart() takes users under five train events, warm() the rest; coldStart({ under: 20 }) moves the line where the domain puts it. byHistoryLength(id, { min, max }) slices anywhere — the bounds are half-open, so adjacent segments tile the population exactly once. Coverage inside a segment still divides by the whole catalogue: the question is what share of everything available these users were shown.

Metrics

| | What it answers | | --- | --- | | ndcg() | Graded relevance and position. The one metric that sees both. | | precisionAt() | How much of the page was relevant. Divided by k, so unfilled slots count against you. | | recallAt() | How much of what the user went on to like the page managed to surface. | | averagePrecision() | Precision recomputed at every hit. Averaged across users, this is MAP. | | reciprocalRank() | Where the first hit landed. Averaged, MRR. For feeds where the user acts once. | | hitRate() | Did the page contain anything relevant at all. | | catalogueCoverage() | Share of the catalogue the engine was ever willing to show. | | intraListDiversity({ similarity }) | How unlike each other one page's items are. |

Keep an accuracy metric and catalogueCoverage() side by side. A recommender that shows everyone the same fifty popular items scores respectably on nDCG and has coverage near zero — and accuracy alone cannot tell it apart from a good one.

Which strategy is this number

attributeNdcg(report) breaks the metric down by the strategies that earned it:

import { attributeNdcg, formatAttribution } from '@recoengine/eval'

console.log(formatAttribution(attributeNdcg(report)))

  ndcg@10            0.3142
    history          0.1912   61%
    co_occurrence    0.0885   28%
    popularity       0.0410   13%
    fatigue         -0.0096   -3%
    unattributed     0.0031    1%

This is the report a learned model cannot print: it produces a score and keeps no account of what the score was made of. Here every score is a fold over named contributions and there is no way to record one without the other, so the metric and its causes come out of the same run — and the run costs nothing extra, because explain: 'none' already carries the contributions.

Each relevant item on a page holds a slice of that user's nDCG (gain × discount(rank) / idealDcg), and the slice is split among the strategies by their share of the item's final score, following the fold in ARCHITECTURE.md §11.2. Additive strategies split base; multiplicative modifiers own base × (Π m − 1), which is what fatigue cost; boosts own their number outright. The rows add back up to the metric, and the tests assert it rather than hope for it. unattributed is nDCG earned by items whose score no strategy owns — a vetoed page still counts for the metric, and dropping that quietly would leave the rows summing to less than the number above them with no sign of why.

It is accounting, not a counterfactual. It says who paid for the page you got, not what the page would look like without them: a strategy can hold 60% of the score for items that would have ranked top anyway. To answer that, zero the weight and run again. Read a small share as "not carrying this page", never as "safe to delete".

Tuning the weights, and the slice that keeps it honest

tuneWeights walks the weights one at a time and keeps what helps — coordinate ascent over a fixed grid, seeded, so the same call gives the same answer.

import { evaluate, tuneWeights, formatTuning, weightsUnderTest } from '@recoengine/eval'

// The counterpart of `replayClock`: the layer the search writes candidates into.
const weights = weightsUnderTest()
const engine = createEngine<Track>().use(weights).provide(CLOCK, clock).use(/* … */).build()

const result = await tuneWeights({
  engine,
  weights,
  start: { history: 1, popularity: 0.3, recency: 0.4 },
  train: splitByTime(tuningEvents, { at: tuningBoundary }),
  validate: splitByTime(events, { at }),   // never optimised against
  k: 10,
  seed: 'tune-1',
  clock,
})

console.log(formatTuning(result))

  tune  ndcg@10   0.3142 → 0.3826   +21.8%   34 runs, seed 'tune-1'
    history       1 → 1.5
    popularity    0.3 → 0.1
    recency       0.4 → 0
    validation    0.3050 → 0.3512   +15.1%

Three things worth knowing before quoting that percentage:

• The gain is measured on the split it optimised. Coordinate ascent will happily find the weights that suit one dataset's noise. validate is scored twice — once at the start, once at the end — and never used to choose, so a gain that exists only on the tuning split shows up as a validation row that disagrees. Without it, the run prints a number about one dataset and it gets quoted as a number about the product.
• A weight of 0 is a real answer. The grid starts at zero on purpose: a strategy the search turns off is one worth deleting, and learning that is worth more than a third of a percent of nDCG.
• It costs one engine run per candidate, and runs is in the result. Keep the tuning split smaller than the one you report on.

Weights are divided by Σ weight when they fold, so only their ratios matter — which is why the grid is absolute values rather than multipliers, and why history: 1 → 1.5 and halving everything else are the same move.

Three things this package will not do quietly

Each is a standard way to produce a number that means nothing.

It will not split at random. A random split lets the engine see Friday while predicting Thursday, and every metric comes out optimistic by an amount nobody can estimate afterwards. Only splitByTime exists.

It will not average "not applicable" as zero. A user with nothing held out has no recall. Scoring them 0 drags the mean down in proportion to how strict the split was, which is how a dataset change gets mistaken for a quality regression. Metrics return undefined for those users, and each summary reports how many contributed.

It will not let the clock drift. Every recency-shaped component reads ctx.now: interactionRecencyExtractor decays by it, recencyStrategy ranks by it, fatigueModifier recovers by it. Evaluate a split from last June against the system clock and every train event looks eleven months stale — recency collapses for everyone and the run measures an engine nobody will deploy. Nothing throws. Pass a replayClock and the engine stands where the split does.

The one option to get right

seenInTrain decides whether an item the user already touched can count as a hit.

• 'exclude' (default) — the safe reading. For a shop or a news feed, re-recommending something already bought or read is not a hit, and counting it turns the metric into a memory test historyStrategy wins by construction.
• 'keep' — the honest reading for repeat consumption. In music, replaying a track is the signal; excluding it throws away most of the ground truth.

There is no default that fits both, so the option is named after the decision rather than being a boolean you can set without noticing.

Relevance is graded, not binary

gain turns a user's held-out events on one item into a number. The default is 1 per item — binary relevance. countGain compresses repeats with log2(1 + n), the same compression nDCG's own discount uses:

splitByTime(events, { at, gain: countGain })

// Or your own: a purchase outweighs a view, a skip is not relevance at all.
splitByTime(events, {
  at,
  gain: (list) => list.filter((e) => e.type === 'purchase').length * 5
                + list.filter((e) => e.type === 'view').length,
})

A gain of zero or below drops the item from the truth entirely, rather than leaving it in recall's denominator where it would silently cap the metric below 1.

License

MIT