@shapeshift-labs/frontier-event-log

v0.1.3

Published

4 months ago

Bounded in-memory event logs, replay cursors, compaction, and Frontier patch events.

0High
0Medium
0Low

shapeshift-labs

frontier event-log replay cursor compaction state

Frontier Event Log

Bounded in-memory event logs, replay cursors, key compaction, and Frontier patch events.

This package sits beside @shapeshift-labs/frontier, the small JSON diff/apply core package. It keeps event replay, cursor ownership, retention, and compaction out of state/cache packages while still using core JSON clone and patch types.

npm: @shapeshift-labs/frontier-event-log
source: siliconjungle/-shapeshift-labs-frontier-event-log
license: MIT

Related Packages

The published Frontier package family is generated from one shared package catalog so READMEs stay in sync across packages:

@shapeshift-labs/frontier: Core JSON diff/apply, compact patch tuples, JSON Pointer, equality, clone, validation, Unicode helpers, and tiny dependency-free runtime budget/scheduler primitives.
@shapeshift-labs/frontier-query: Shared query-key, selector path, condition, entity identity, and table-shape primitives.
@shapeshift-labs/frontier-codec: Patch serialization, binary frames, canonical JSON, and patch-history codecs.
@shapeshift-labs/frontier-engine: Stateful planned diff engine, adaptive profiles, schema plans, and engine-level history helpers.
@shapeshift-labs/frontier-state: Patch-routed app-state subscriptions, owned commits, maintained views, and path mapping.
@shapeshift-labs/frontier-dataflow: Serializable incremental dataflow and materialized-view graphs for Frontier apps, including selectors, dependency DAGs, filters, joins, aggregations, stale paths, recompute budgets, output patches, provenance records, and proof of why derived views changed.
@shapeshift-labs/frontier-state-cache: Normalized query-result cache with entity/query watchers, persistence, change logs, optimistic layers, scheduled persistence, and mutation bridge.
@shapeshift-labs/frontier-state-cache-idb: IndexedDB persistence adapter for Frontier state-cache snapshots and durable change logs.
@shapeshift-labs/frontier-state-cache-file: Structured file persistence adapter for Frontier state-cache snapshots and change logs.
@shapeshift-labs/frontier-state-cache-sql: SQL persistence adapter for Frontier state-cache snapshots and change logs.
@shapeshift-labs/frontier-schema: JSON Schema validation, Frontier profile generation, CloudEvent envelopes, and query/table schema helpers.
@shapeshift-labs/frontier-migrations: Boundary-first data migrations, import normalization, plugin/API version mapping, versioned envelopes, graph diagnostics, patch path rewrites, dry-run reports, and current-shape rehydration.
@shapeshift-labs/frontier-inspect: Cross-package inspection/evidence bundles, registry graph snapshots, feature/resource impact reports, timeline/event normalization, redaction, JSONL import/export, and AI-readable app feature maps.
@shapeshift-labs/frontier-scheduler: Deterministic work scheduling, lanes, cancellation, backpressure, frame policies, replay snapshots, and work graphs.
@shapeshift-labs/frontier-logging: Opt-in structured logging, browser telemetry, scheduled sinks, file sinks, exporters, benchmark traces, and Frontier patch/update summaries.
@shapeshift-labs/frontier-mutation: Explicit mutation and selector plans compiled to Frontier patches or CRDT operations.
@shapeshift-labs/frontier-effects: Serializable effect descriptors and resource graphs for Frontier apps, including fetch, storage, timers, navigation, workers, clipboard, broadcast, WebSocket, stream, policy metadata, runtime records, redaction, JSONL, proof helpers, and registry graph output.
@shapeshift-labs/frontier-policy: Serializable policy and capability decisions for Frontier apps, effects, views, sync, routes, traces, and AI tools.
@shapeshift-labs/frontier-tools: Serializable app action/tool manifests for AI-operable Frontier apps, including availability, validation, dry-run plans, patch previews, effect/tool constraints, execution records, rollback links, and registry graph output.
@shapeshift-labs/frontier-sandbox: Runtime-agnostic sandbox contracts for Frontier patch-producing actions, including manifests, declared reads/writes/capabilities, host-validated patch/effect/event/log results, dynamic source modules, source event replay, and structural runtime adapters.
@shapeshift-labs/frontier-sandbox-quickjs: QuickJS/WebAssembly runtime adapter for Frontier sandbox actions, including invocation/runtime isolation modes, deadline and memory limits, dynamic source execution, and patch/effect result normalization.
@shapeshift-labs/frontier-workflow: Serializable durable workflow/process manifests for Frontier apps, including steps, waits, approvals, timers, retries, expected patches, compensation, records, timelines, and registry graph output.
@shapeshift-labs/frontier-worker: Serializable worker and edge task descriptors for Frontier apps, including queues, idempotency keys, retry and timeout policy, declared reads/writes/effects, snapshots, patch outputs, produced assets, execution records, logs, trace links, proof hashes, dedupe indexes, and registry graph output.
@shapeshift-labs/frontier-assets: Serializable asset and content provenance graphs for Frontier apps, including source files, generated variants, thumbnails, LOD chunks, shader/material dependencies, transforms, hashes, owners, runtime consumers, review plans, registry graph output, and impact queries.
@shapeshift-labs/frontier-triggers: Capability-gated event trigger registry, scoped event envelopes, listener/reaction rules, structured rejection, deterministic event-to-action scheduling, replay/provenance records, and registry graph output.
@shapeshift-labs/frontier-virtual: DOM-neutral virtualization, layout providers, range materialization, grids, spatial/frustum indexes, patch invalidation, camera anchors, and serializable layout state.
@shapeshift-labs/frontier-scene: Patch-native 2D/3D scene graph, transform propagation, bounds queries, virtual/culling adapters, spatial invalidation, and camera/frustum materialization.
@shapeshift-labs/frontier-pathfinding: Patch-native grid pathfinding, typed-array A*/Dijkstra search, flow fields, connected components, line-of-sight smoothing, dirty-cell invalidation, and scheduler-friendly path jobs.
@shapeshift-labs/frontier-lod: Patch-native level-of-detail and significance selection for rendering and computation workloads, compact typed hot paths, multi-observer selection, budget degradation, materialization frames, and scheduler work plans.
@shapeshift-labs/frontier-route: DOM-neutral app/game route resources, route and scene manifests, match/resolve/transition planning, dependency metadata, sessions, registry graph output, and impact queries.
@shapeshift-labs/frontier-trace: Serializable traces, spans, events, causal links, W3C trace context helpers, timeline/resource/path queries, critical-path analysis, registry graph output, JSONL/proof helpers, Chrome trace export, and redaction for app-wide feature observability.
@shapeshift-labs/frontier-manifest: Build/static feature manifests for owners, routes, actions, states, migrations, tests, source files, assets, resources, tasks, dependency metadata, registry graph output, feature maps, JSONL export, and impact queries.
@shapeshift-labs/frontier-view: Renderer-neutral view manifests, type defaults, validation frames, action bindings, visual channels, virtual/LOD hints, and data-to-representation mapping for Frontier apps.
@shapeshift-labs/frontier-dom: Patch-native DOM and host renderer bindings, manifest hydration, JSX runtime/compiler helpers, SSR, devtools, and logging bridges.
@shapeshift-labs/frontier-playwright: Playwright/headless automation probes for Frontier state, DOM, devtools, marks, and timeline queries.
@shapeshift-labs/frontier-test: Serializable test/spec evidence manifests for Frontier apps, including fixtures, commands, expected patches/effects/routes/policies, coverage declarations, run plans, run records, report adapters, replay proofs, fuzzers, benchmarks, registry graph output, and impact queries.
@shapeshift-labs/frontier-history: Serializable temporal explanation and causality records for Frontier apps, including field-change explanations, action/workflow/policy/effect/trace/test provenance, audit windows, undo planning, registry/provenance graph output, JSONL replay bundles, and proof hashes.
@shapeshift-labs/frontier-application: Serializable whole-application graph and impact queries for Frontier apps, including features, owners, packages, routes, views, actions, mutations, state paths, effects, workers, assets, tests, traces, policies, workflows, migrations, benchmarks, registry graph output, feature maps, JSONL bundles, and proof hashes.
@shapeshift-labs/frontier-linter: Serializable Frontier lint rules, diagnostics, fixes, reports, and fast rule execution for package catalogs, registry graphs, application maps, manifests, traces, policies, workflows, workers, assets, tests, benchmarks, and source snippets.
@shapeshift-labs/frontier-crdt: Native CRDT documents, update tooling, awareness, branches, conflict introspection, version frames, and undo.
@shapeshift-labs/frontier-crdt-sync: CRDT sync endpoints, repo/storage/provider contracts, scheduled sync work, document URLs, local networks, model checking, forensics, and text binding contracts.
@shapeshift-labs/frontier-crdt-websocket: WebSocket client/server transports for Frontier CRDT sync providers.
@shapeshift-labs/frontier-react: React external-store hooks and adapters for Frontier state, cache, and CRDT surfaces.
@shapeshift-labs/frontier-richtext: Rich text Delta normalization/application, marks, embeds, ranges, and cursor/selection transforms for local editor integrations.
@shapeshift-labs/frontier-realtime: Shared realtime command, tick, snapshot, prediction, reconciliation, interpolation, rollback, message, and delta primitives.
@shapeshift-labs/frontier-realtime-server: Authoritative realtime room, tick, command validation, rate-limit, session, and snapshot-history runtime.
@shapeshift-labs/frontier-realtime-websocket: WebSocket client, wire, and Node room-server transport for Frontier realtime.
@shapeshift-labs/frontier-game: Game-facing entity, component, player, room, ownership, spatial interest, rollback, physics, and replication helpers above realtime.

Package source repositories:

Install

npm install @shapeshift-labs/frontier @shapeshift-labs/frontier-event-log

Usage

import { diff } from '@shapeshift-labs/frontier';
import { appendPatchEvent, createEventLog } from '@shapeshift-labs/frontier-event-log';

const log = createEventLog({
  capacity: 1000,
  compactByKey: true,
  dropTombstones: true
});

log.append({
  key: 'todo:a',
  value: { type: 'todo.updated', id: 'a', done: true }
});

const patch = diff(
  { todos: [{ id: 'a', done: false }] },
  { todos: [{ id: 'a', done: true }] },
  { arrayKey: 'id' }
);

appendPatchEvent(log, patch, {
  key: 'todos',
  metadata: { source: 'cache-write' }
});

const replay = log.read(0, { limit: 32 });
console.log(replay.records, replay.cursor);

API

import {
  applyPatchEventRecord,
  appendPatchEvent,
  createEventLogCheckpoint,
  createEventLog,
  createEventLogReplayStorage,
  diffBetweenTimes,
  replayEventLog,
  stateAtTime,
  type EventLog,
  type EventLogCheckpoint,
  type EventLogConsumer,
  type EventLogCursor,
  type EventLogRecord,
  type EventLogReplayStorage,
  type PatchEventLogValue
} from '@shapeshift-labs/frontier-event-log';

`createEventLog(options?)`

Creates an in-memory log with monotonically increasing offsets.

Useful options:

capacity: maximum retained record count.
discard: oldest drops old records when full; new rejects new appends.
compactByKey: enables key-based compaction.
compactOnAppend: compacts after each append when compactByKey is enabled.
dropTombstones: removes latest null keyed records during compaction.
scheduler: optional structural scheduler for queued append compaction.
initialOffset: starting offset.
now: timestamp supplier for deterministic tests.

Appending And Reading

log.append(input) appends or throws if rejected.
log.tryAppend(input) returns { accepted, record?, reason? }.
log.appendBatch(inputs, { maxRecords?, maxBytes? }) appends a bounded batch.
log.read(cursor?, { limit?, maxBytes? }) returns cloned records plus a cursor.
log.truncateBefore(cursor) drops retained records before a checkpoint cursor.
log.clear() removes retained records without resetting the next offset.

Consumers

const consumer = log.createConsumer('worker-a');
const result = consumer.read({ limit: 100 });
consumer.ack(result.cursor);

Consumers own a read cursor and a committed cursor. They are useful for replay windows, durable checkpoints, and independent application workers.

Checkpoints And Replay

const checkpoint = createEventLogCheckpoint(log, { count: 2 });
const result = replayEventLog(log, checkpoint, (state, record) => ({
  count: state.count + Number(record.value.delta || 0)
}));

createEventLogCheckpoint() captures an application snapshot plus an event-log cursor. replayEventLog() resumes from that cursor in bounded batches and returns the replayed state, cursor, replay count, and a fresh checkpoint. createEventLogReplayStorage() provides the same snapshot-plus-bounded-change-log shape used by state-cache persistence without making state-cache depend on event-log.

Temporal State And Diff

stateAtTime() materializes state at an offset, cursor, high-watermark, or timestamp by replaying from a checkpoint. diffBetweenTimes() materializes two temporal states and returns a Frontier patch between them. For patch-event logs, applyPatchEventRecord() is the reducer.

const atCursor = stateAtTime(log, checkpoint, applyPatchEventRecord, {
  at: { offset: 128 }
});

const change = diffBetweenTimes(log, checkpoint, applyPatchEventRecord, {
  from: { offset: 64 },
  to: { timestamp: Date.now() }
});

Patch Events

appendPatchEvent(log, patch, {
  key: 'doc:1',
  metadata: { actor: 'alice' }
});

Patch events store { kind: 'patch', patch, metadata? } values in the log. They are ordinary event-log records and can be read, compacted, or retained like any other keyed event.

Subpath Imports

import { createEventLog } from '@shapeshift-labs/frontier-event-log';
import { createEventLog as createEventLogSubpath } from '@shapeshift-labs/frontier-event-log/event-log';

Both imports expose the same event-log API.

Package Scope

This package owns:

in-memory event logs,
snapshot checkpoints and bounded replay helpers,
temporal state-at-time and diff-between-times helpers,
generic replay storage for snapshot plus change-log adapters,
append batching and bounded replay windows,
consumer cursors and acknowledgements,
capacity retention policies,
keyed compaction and tombstone dropping,
Frontier patch event records.

It does not own:

diff/apply primitives,
binary patch codecs,
app-state subscriptions,
normalized query caches,
structured telemetry sinks,
CRDT documents, branches, sync, awareness, or rich text.

TypeScript

The package ships ESM JavaScript plus .d.ts declarations for the root export and ./event-log subpath. The package-local TypeScript source lives in src/ and compiles directly to dist/.

Validation

npm test
npm run fuzz
npm run bench
npm run pack:dry

The package test suite covers root and subpath imports, append/read behavior, clone isolation, retention policies, keyed compaction, batch limits, consumers, checkpoint replay, temporal state/diff, replay storage, patch events, and randomized operation sequences.

Benchmarks

Run the package-local benchmark:

npm run bench

Latest local package benchmark on Node v26.1.0, darwin arm64, 9 rounds:

| Fixture | Median | p95 | | --- | ---: | ---: | | Append keyed JSON event | 3.99 us | 4.65 us | | Read replay window, 32 records | 1.55 us | 1.69 us | | Consumer read and ack | 0.43 us | 0.46 us | | Compact keyed log, 1k records | 156.31 us | 181.40 us | | Append batch compactOnAppend, 1k records | 100.82 us | 105.50 us | | Replay from checkpoint, 64 records | 25.43 us | 28.23 us | | State at offset, 64 patch events | 15.96 us | 16.14 us | | Diff between offsets, 64 patch events | 26.64 us | 28.25 us | | Replay storage append/read checkpoint | 11.16 us | 11.51 us | | Append Frontier patch event | 2.34 us | 2.99 us |

These are Frontier-only package measurements, not competitor comparisons. Replay and consumer fixtures use preseeded retained logs so the timed work is read/cursor behavior, not fixture construction.

License

MIT. See LICENSE.