@ordinatio/timeline

Merge several newest-first sources into ONE paginated feed, with an exact cursor: nothing repeated, nothing skipped, even when many rows share one millisecond. Pure TypeScript, no database, no framework. Ships a contract test kit (@ordinatio/timeline/testing) so any storage can prove it is a correct source.

The idea

• A source (TimelineSource) returns its own rows in strict order: time descending, then id descending. It is asked for "everything after this bound", where the bound is { position: { at, id } } (strictly after that row), { floor: at } (every row at or below that time, for a source that has shown nothing yet) or null (from the newest).
• createTimeline({ sources, scope, sourceTimeoutMs }).page({ cursor, limit }) asks every source for limit + 1 rows in parallel, checks each answer, merges by (time desc, kind rank, id desc), cuts at limit and returns { ok: true, entries, nextCursor } or { ok: false, reason } (bad-cursor, bad-limit, source-failed, source-timeout, source-misbehaved). It never throws for an expected outcome. page(null) is the same as page().
• The cursor holds the last shown row of each source plus one floor. It is opaque, strict and scope-checked, but NOT signed: a truncated, malformed, non-canonical or other-scope cursor is bad-cursor, but a client can hand-build a perfectly valid cursor for any scope (encodeCursor is exported). The scope only stops accidental reuse of a cursor across feeds. Authorization and tenancy must live in every source's own filter, never in the cursor.
• A failing, slow or lying source fails the whole page, so the cursor can never move past rows that were not read.
• keysetAfter gives the plain-object where fragment a database source needs; createMemorySource is the reference implementation.
• createTimeline throws at creation for a mis-built configuration, including a scope that is not a string of at most 255 characters.

Writing a source

A source MUST:

1. Return at most limit rows, and the service passes limit + 1 (page size plus ONE extra), so the extra row tells exactly whether there is more. Return up to the whole limit it is given.
2. Honour { floor } as at <= floor, INCLUSIVE, with NO id condition (rows at exactly the floor time are included).
3. Honour { position } as strictly after (at, id) in (at desc, id desc) order: older, or the same time with a smaller id.
4. Order and compare ids by code unit (binary / C collation), never a locale collation.
5. Set every row's kind to the source's own kind.
6. Throw SourceTimeoutError to signal a timeout (for example a cancelled statement); any other throw is source-failed.
7. Respect context.deadline (absolute time on the service's clock, after which the service stops waiting) and stop work when it passes.
8. Apply its own tenancy and authorization filter on every query.

keysetAfter emits the OR form (time < t OR (time = t AND id < i)), which a database cannot always use an index for; a SQL author may use the equivalent row comparison (time, id) < (t, i).

import { createTimeline, keysetAfter, SourceTimeoutError, type TimelineEntry, type TimelineSource } from '@ordinatio/timeline';

type Note = TimelineEntry & { text: string };

function notesSource(db: NotesDb, personId: string): TimelineSource<Note> {
  return {
    kind: 'note',
    async fetchPage(bound, limit, { deadline }) {
      const rows = await db.notes.findMany({
        where: { personId, ...keysetAfter({ timeField: 'createdAt', idField: 'id' }, bound) }, // personId: the source's own scoping
        orderBy: [{ createdAt: 'desc' }, { id: 'desc' }],
        take: limit, // the service already asked for page size + 1
        timeoutMs: Math.max(1, deadline - Date.now()), // hypothetical: stop when the service stops waiting
      }).catch((error) => { throw error.code === 'TIMEOUT' ? new SourceTimeoutError() : error; });
      return rows.map((r) => ({ kind: 'note', id: r.id, occurredAt: r.createdAt.getTime(), text: r.text }));
    },
  };
}

const timeline = createTimeline({ sources: [notesSource(db, 'person-1')], scope: 'person:person-1' });
const first = await timeline.page({ limit: 20 });
if (first.ok && first.nextCursor) {
  const second = await timeline.page({ cursor: first.nextCursor, limit: 20 }); // "Show more"
}

Proving your source: the contract kit

@ordinatio/timeline/testing exports defineTimelineSourceContract(api, options). It registers its checks with the describe, it and expect of YOUR runner (vitest, jest, anything with those three), so the same suite proves the in-memory source and your database source (run the database one in an opt-in test against a scratch scope):

import { describe, it, expect } from 'vitest';
import { defineTimelineSourceContract } from '@ordinatio/timeline/testing';

defineTimelineSourceContract({ describe, it, expect }, {
  name: 'notes source',
  kind: 'note',
  foreignRows: 'seeded', // REQUIRED: 'seeded' (give seedForeign below) or null = "this source has no foreign rows to prove" (the check then reports itself as skipped)
  hiddenRows: 'seeded',  // REQUIRED, same for rows the source must hide
  // Called once per check: a fresh, EMPTY, isolated copy (own scope or organization, own parent).
  async makeSource() {
    const scope = await createScratchScope();
    return {
      source: notesSource(db, scope),
      seed: (rows) => insertNotes(scope, rows),               // rows are { id, kind, occurredAt }
      seedForeign: (rows) => insertNotes(otherScope(), rows),  // needed for foreignRows: 'seeded': another tenant's rows, must never appear
      seedHidden: (rows) => insertTrashedNotes(scope, rows),   // needed for hiddenRows: 'seeded': trashed or removed rows, must never appear
      seedOutOfRange: (rows) => insertNotes(scope, rows),      // optional: only for a source that EXCLUDES years outside 1 to 9999
      cleanup: () => dropScope(scope),
    };
  },
  makeTimingOutSource: async () => notesSourceWithTinyStatementTimeout(db), // optional: must throw SourceTimeoutError
  // maxLimit: 50,      // optional: the maxLimit of your createTimeline (default 50); the kit seeds 2 x (maxLimit + 1) + 3 rows and needs maxLimit + 1 back in ONE call
  // tieIds: ['...'],  // optional: replace the default tricky ids if your id column cannot hold them (for example a uuid column)
  // plainId: (n) => ..., // optional: the other fixture ids; a uuid or cuid id column breaks the default `row-0001` ids, so the fixture ids must fit the column
});

What it checks, against its own independent model (not the package's helpers): the three bound shapes (null; { position } strictly after, in time descending then id descending; { floor } inclusive, time only, with rows exactly at the floor and no id needed), limit and limit + 1 exactness (also a feed of more than twice maxLimit + 1 rows, so a source with its own row cap below maxLimit + 1 is caught), strict order with many rows at ONE millisecond and ids that sort differently by code unit and under a locale collation, every row's kind, no duplicates, empty and past-the-end answers, the smallest and largest representable times (years 1 and 9999), rows out of that range (absent, never a crash), the context deadline (honoured, or a correct answer, or SourceTimeoutError, nothing else), the timeout signal, stability of repeated calls, rows of another scope and hidden rows (foreignRows and hiddenRows), that the service's own answer check accepts every answer, and finally paging the whole feed through createTimeline with random page sizes over your REAL source against a sorted oracle. The checks are proved able to fail: this package's tests run the kit against deliberately wrong sources, at least one for every class of flaw the list above names, and each must be caught by a named check (including ones that only the whole-feed oracle can catch, such as a source that leaks foreign or hidden rows only when asked for exactly maxLimit + 1 rows). Not every check has a wrong source of its own: a few are guarded only because the same flaw also breaks a neighbouring check (the list is in the header of source-contract.test.ts).

Not checked (test them yourself): that a query uses an index or is cheap on a deep page, the real cost of a timeout, and ids in a database whose column collation differs from code-unit order (the kit proves id order only with the ids it is given). foreignRows and hiddenRows are required options; null is a deliberate opt-out and the kit reports the check as "skipped by the caller". Supply both for any source over shared storage. The kit tests the behaviour of single calls to your source. That a row deleted between two pages never causes a repeat or a skip is proved by the service oracle in this package (service.model.test.ts), not by the kit. Some checks make hundreds of calls: on a database over a network raise your runner's per-test timeout.

What an open feed promises

Rows that existed from the first page to the last are never repeated and never skipped. A row that appears later and is older than its source's place shows up once, possibly out of order; a row newer than the page end does not join (reload shows it).

Known limits

• The cursor is not signed (see above): tenancy is the sources' job.
• kindOrder (and the default source order) is not carried in the cursor. Keep it stable while cursors are live; a deploy that changes it makes in-flight "Show more" clicks inconsistent (rows tied at the boundary may repeat or be skipped). Acceptable: the user reloads.
• Exported for the contract kit and tests (not part of the everyday API): encodeCursor, decodeCursor, validateSourcePage, TIMELINE_TIME_MIN, TIMELINE_TIME_MAX.
• Times are whole milliseconds in years 1 to 9999 (TIMELINE_TIME_MIN to TIMELINE_TIME_MAX). A source over data that can hold other dates must exclude them in its own query; the kit's seedOutOfRange hook proves it.
• The package has no runtime dependencies. The /testing entry has none either (it imports no test runner: you pass yours).
• Version 1.0.0 is the first release. The cursor format is v1.: a change that makes old cursors unreadable will be a new major version, and decoding refuses any other version (the page answers bad-cursor).

License

MIT