ArrowBase

ArrowBase — columnar Arrow store with @tanstack/db-compatible API.

ArrowBase is a local-first TypeScript database built on Arrow-style column buffers. It keeps the TanStack DB mental model—collections, transactions, live queries, persistence, and React hooks—while adding vectorized filters, spatial indexes, Arrow IPC, snapshots, and cross-tab sync.

v0.1.1: the compatibility catalog is 294/294 green against @tanstack/[email protected]. Exact public-export and operator-manifest tests guard that surface. Pre-1.0 means the API can still evolve between minor versions; documented tolerances cover fixed capacity, safe-integer key normalization, and floating-point aggregates.

Open the live playground · Read the migration guide · Inspect compatibility

Project docs:

• COMPATIBILITY.md — all 294 audited upstream exports.
• MIGRATING.md — side-by-side TanStack DB replacements.
• docs/showcase.md — playground and benchmark contract.
• docs/ivm.md — incremental query engine boundaries.
• SECURITY.md and CONTRIBUTING.md.
• RELEASE_NOTES.md and CHANGELOG.md.

Requirements

• Node.js 20 or newer.
• Modern browsers. ArrowBase uses SharedArrayBuffer when cross-origin isolation is available and safely falls back to ArrayBuffer otherwise.

Install

npm install arrowbase

Optional integrations are installed only when you use their subpaths:

npm install @tanstack/db   # arrowbase/tanstack
npm install apache-arrow   # arrowbase/arrow
npm install react          # arrowbase/react

Playground

The hosted ArrowBase City Ops Console is an interactive ArrowBase vs TanStack DB playground. It runs deterministic spatial, reactive, transactional, and IPC workloads in browser workers. A correctness hash must match before timing or memory ratios unlock.

See docs/showcase.md for the benchmark contract, URL controls, operation registry, and local or tunnel workflows.

Development

pnpm install --frozen-lockfile
pnpm check                  # typecheck + 700+ behavioral tests
pnpm playground:check       # UI typecheck + tests + production build
pnpm playground:e2e         # desktop and mobile Playwright smoke
pnpm bench:vs-tanstack      # head-to-head benchmark
pnpm build
pnpm verify:package-exports

Use these focused playground commands during UI work:

pnpm playground:dev
pnpm playground:typecheck
pnpm playground:test
pnpm playground:build
pnpm playground:e2e

Performance ratios are evidence, not promises: the showcase compares equal result hashes before presenting speed or memory measurements.

What's in the box

Phase 1 — Core primitives

Fixed-width columns (int8..int64, uint8..uint64, float32/64, bool) backed by a 64-byte-aligned SAB bump allocator. Dense uint32 primary key, soft-delete, per-column + global version counters, bounded undo ring with mark/rollback.

import { Collection, defineSchema } from 'arrowbase';

const schema = defineSchema({
  name: 'features',
  columns: {
    __id: { type: 'uint32' },
    __deleted: { type: 'bool' },
    speed: { type: 'int32', nullable: true },
    weight: { type: 'float64' },
  },
});

const features = Collection.create(schema, { capacity: 10_000 });
features.insert({ __id: 1, __deleted: false, speed: 50, weight: 1.5 });
features.update(1, { speed: 60 });
features.delete(1); // soft; becomes hard in hard-delete mode

const mark = features.beginTransaction();
features.update(1, { speed: 999 });
features.rollback(mark);

Phase 2 — Reactive queries

Immutable query builder (where/select/orderBy/limit/offset/deps). QueryExecutor caches snapshots keyed on observed-column versions. A passive change listener invalidates the cache on insert/delete/compact/ rollback; updates that touch only unobserved columns keep the cached snapshot identity.

import { query, run } from 'arrowbase';

const q = query(features)
  .where((r) => r.speed !== null && (r.speed as number) > 50)
  .orderBy('weight', 'desc')
  .select(['__id', 'speed'])
  .limit(20);

const exec = run(q);
const unsub = exec.subscribe((snap) => console.log(snap.rows));
// unsub() when done; exec.dispose() to detach the passive listener.

Predicate deps are inferred via a tracking Proxy. Override with .deps(['surface', 'speed']) when inference is incomplete.

Phase 3 — Variable-width + dict columns

utf8, binary, and dictionary-encoded dict_utf8. Utf8/binary use a ranged-offsets layout (Int32Array(capacity * 2) pairs + append-only ValuesHeap) so updates are O(1) at the cost of leaked bytes until compact() runs.

const schema = defineSchema({
  name: 'road',
  columns: {
    __id: { type: 'uint32' },
    name: { type: 'utf8', nullable: true, valuesHeapBytes: 512 * 1024 },
    surface: { type: 'dict_utf8' },
    blob: { type: 'binary' },
  },
});

Deferred cleanup — compaction

Collection.compact() reclaims leaked utf8/binary heap bytes and hard- deleted row slots, rebuilds the row index, clears the undo ring, and fires a compact ChangeEvent so reactive queries recompute.

const { reclaimedRows, reclaimedHeapBytes } = features.compact();

Phase 4 — TanStack DB adapter

arrowbase/tanstack exposes an ArrowBase collection as a first-class TanStack DB collection. Use @tanstack/db's native live-query builder to join ArrowBase with native TanStack collections.

This adapter prioritizes drop-in native interoperability. TanStack's public collection boundary exchanges JavaScript row objects, so this route does not preserve zero-copy source reads or push ArrowBase spatial indexes into TanStack's native query compiler. Use ArrowBase's root createLiveQueryCollection API when columnar differential execution and spatial pushdown matter; it keeps the same builder shape and runs structural queries through @tanstack/db-ivm. See docs/ivm.md for the exact supported/fallback boundary.

import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db';
import { toTanstackCollection } from 'arrowbase/tanstack';

const abFeatures = Collection.create(/* ... */);
const features = createCollection(toTanstackCollection(abFeatures));

const tags = createCollection({ /* native TS collection with tags */ });

const joined = createLiveQueryCollection({
  query: (q) =>
    q
      .from({ t: tags })
      .join({ f: features }, ({ t, f }) => eq(t.featureId, f.__id))
      .select(({ t, f }) => ({ label: t.label, name: f.name })),
});

onInsert/onUpdate/onDelete forward TanStack mutations into ArrowBase; the sync listener mirrors ArrowBase changes back into TanStack. compact and rollback trigger a truncate() + replay for full consistency.

Options:

• id?: string — TanStack collection id. Default 'arrowbase:<schemaName>'.
• includeSoftDeleted?: boolean — mirror tombstones as regular rows.

Phase 5 — Snapshot + sync

Custom binary snapshot (ARBS magic, format v1) preserves schema, data, dictionaries, and globalVersion across round-trips. Sync adapter interface + in-process MemorySyncAdapter for tests; ChangeLog records outgoing deltas; applyDeltas handles inbound with upsert promotion.

import {
  exportSnapshot,
  importSnapshot,
  ChangeLog,
  MemorySyncAdapter,
  SharedDeltaLog,
  applyDeltas,
} from 'arrowbase';

// Persistence
const buf = exportSnapshot(features, { tag: 'nightly' });
const restored = importSnapshot(buf, (schema, capacity) =>
  Collection.create(schema, { capacity }),
);

// Sync
const log = new SharedDeltaLog();
const outbound = new ChangeLog(features);
const adapter = new MemorySyncAdapter(features, log, outbound);

const { deltas, upTo } = await adapter.pull(0);
await adapter.apply(deltas);

Phase 6 — Devtools + subset push-down

describeCollection(abColl) returns a framework-free JSON snapshot of schema + per-column metadata (version, nullCount, heap usage, dict sample). pushDown(expr, knownColumns) translates TanStack DB BasicExpression ASTs (eq, and, or, not, boolean ref) into an ArrowBase Predicate; unsupported expressions return null so the caller falls back gracefully.

import { describeCollection, pushDown } from 'arrowbase';

console.log(describeCollection(features));
// → { schemaName, primaryKey, rowCount, liveRowCount, globalVersion,
//     hasSoftDelete, schemaFingerprint, columns: [...] }

Phase 7 — Vectorized filter + sort kernels

.filter(expr) takes a column-op DSL expression that compiles to a row-test closure reading directly from typed-array column buffers. No per-row proxy, no row-object allocation during filter, numeric sort via a Float64Array shadow-key buffer, materialization only of the [offset, offset+limit) window. The fn-predicate path via .where(fn) still works and can be combined with .filter() — the kernel runs first, the fn runs only on survivors.

import { query, run, f, fEq, fGt, fAnd } from 'arrowbase';

const q = query(features)
  .filter(fAnd(fEq(f('surface'), 'asphalt'), fGt(f('speed'), 50)))
  .orderBy('weight', 'desc')
  .select(['__id', 'speed', 'weight'])
  .limit(20);

const snap = run(q).snapshot();

Operators: fEq, fNeq, fGt, fGte, fLt, fLte, fAnd, fOr, fNot, fIsNull, fIsNotNull. Supported types: all fixed-width numerics, bool, int64/uint64 (with number→bigint literal promotion), utf8 (eq/neq), dict_utf8 (eq/neq via O(1) dictionary code lookup).

String predicates. fContains, fStartsWith, fEndsWith, fLike, fMatches work on utf8 and dict_utf8 columns. On dict-encoded columns the kernel walks the dictionary once up-front to build a Set<code> of matching entries, then each row is a single Set.has() — ~4× faster than per-row utf8 substring search on 10k rows.

import {
  query, run, f,
  fContains, fStartsWith, fEndsWith, fLike, fMatches, fAnd,
} from 'arrowbase';

// Case-insensitive substring (ASCII-fold).
query(c).filter(fContains(f('tag'), 'asp', { caseInsensitive: true }));

// SQL LIKE. `%` = any sequence, `_` = any single char, `\` escapes.
query(c).filter(fLike(f('path'), '/users/%/active'));

// Raw regex (pattern string or RegExp).
query(c).filter(fMatches(f('url'), /^https?:/));

// Composable, negatable.
query(c).filter(fAnd(
  fStartsWith(f('tag'), 'A'),
  fContains(f('text'), 'road', { caseInsensitive: true, negated: true }),
));

The fn-compiler auto-lowers the three common JS method forms into the equivalent DSL node:

// Both run through the vectorized kernel:
query(c).where(r => r.tag.includes('asphalt'));      // → fContains
query(c).where(r => r.tag.startsWith('A'));           // → fStartsWith
query(c).where(r => r.tag.endsWith('vel'));           // → fEndsWith

Phase 8 — List + Struct column types

Two nested column shapes landed, rounding out the type system:

• list<T> — each cell is an array of T. Arrow-style offsets + child buffer; O(1) updates with leak-until-compact semantics. v1 item types: every leaf (all primitives, bool, utf8, dict_utf8, binary).
• struct<{f1, f2, ...}> — each cell is a record of fixed named fields. Parent validity bitmap in SAB + one regular child column per field. v1 field types: leaves only.

Nested list-in-list, struct-in-struct, list-in-struct, struct-in-list are rejected at schema compile time. Snapshot format bumped to v2 so list offsets + child buffers and per-field struct buffers round-trip.

const schema = defineSchema({
  name: 'features',
  columns: {
    __id: { type: 'uint32' },
    tags: { type: 'list', items: { type: 'utf8' } },
    geo: {
      type: 'struct',
      fields: {
        lat: { type: 'float64' },
        lng: { type: 'float64' },
        label: { type: 'utf8', nullable: true },
      },
    },
    attrs: {
      type: 'list',
      items: { type: 'int32', nullable: true }, // list items may be null
    },
  },
});
const c = Collection.create(schema, { capacity: 1_000 });
c.insert({
  __id: 1,
  tags: ['road', 'highway'],
  geo: { lat: 51.5, lng: -0.12, label: 'London' },
  attrs: [10, null, 30],
});
// update replaces the full list/struct cell
c.update(1, { tags: ['updated'] });
// compact reclaims leaked list child bytes + soft-delete slots
const { reclaimedHeapBytes } = c.compact();

List child storage lives in JS heap (outside the SAB segment) because its size is unbounded at schema-compile time. This is a deliberate v1 tradeoff — zero-copy cross-worker sharing for list columns is deferred. Struct field buffers and parent validity stay in the SAB segment.

Auto-lowering .where(fn) to the vectorized kernel

.where(fn) attempts to compile the predicate closure into a FilterExpr tree via a tiny recursive-descent parser over Function.prototype.toString(). When it succeeds, the fn is discarded and the vectorized kernel runs — same fast path as .filter(expr), no caller-side rewrite needed:

// Both forms take the vectorized path; second one compiles the closure
// into the equivalent FilterExpr at query-build time.
query(c).filter(fAnd(fEq(f('active'), true), fGt(f('age'), 18)));
query(c).where((r) => r.active === true && r.age > 18);

Supported shapes: ===/!==/==/!=/>/>=/</<=/&&/||/!, param property or bracket access (r.col or r['col']), string/number/ bool/null literals, row.col == null → isNull. Anything else (captures, method calls, arithmetic, nested field access, ternary, etc.) falls back silently to the legacy row-scan path:

// Falls back — outer variable is a capture the compiler can't inline.
const min = 18;
query(c).where((r) => r.age > min);

The end-to-end speed-up on a 10k row filter+sort+paginate is ~5× (0.5 ms vs 2.5 ms); reactive notifications drop from ~300 µs to ~60 µs. See Benchmarks.

Hash joins

Inner, left, right, and full outer joins between two collections. The right side is hashed; the left side (post-filter) probes into the hash. Nested row shape — the right row is attached at row[as] — so column-name collisions never matter.

import { query, join, runJoin, f, fEq } from 'arrowbase';

const plan = join(
  query(orders).filter(fEq(f('status'), 'paid')),
  users,
  { on: { left: 'userId', right: '__id' }, as: 'user', type: 'left' },
)
  .where((r) => r.user === null || r.user.region === 'us')
  .orderBy('user.name')
  .select(['__id', 'amount', 'user.name']);

const exec = runJoin(plan);
exec.subscribe((snap) => render(snap.rows));
// Re-runs on mutations to EITHER collection.

• Join types — 'inner' (default), 'left', 'right', 'full'.
  • left / full — unmatched left rows emit {...leftRow, [as]: null}. Detect via row[as] === null.
  • right / full — unmatched right rows emit {[as]: rightRow} with left columns absent (not null — undefined), matching SQL projection semantics.
• Key normalization. Safe-range BigInts are unified with Numbers during hashing, so an int64 foreign-key column on one side joins cleanly against a uint32 PK on the other.
• Null keys never join (SQL semantics). For left / full, a left row with a null key is still emitted as unmatched.
• Left-side filters push down. Any .filter() / .where() on the left query runs through the vectorized kernel before the probe — the hash only sees survivors.
• Chainable builder. query(orders).join(users, …) is equivalent to the standalone join(…) function; both return a JoinQuery supporting the full post-join surface (where / select / orderBy / limit / offset).
• Paths. Selection and ordering accept plain column names (left side) or '<as>.<col>' dotted paths (right side). Dotted access into a null namespace yields undefined, so nulls sort last by default.
• Reactivity. The executor subscribes to both collections; mutations on either trigger a recompute. Cache keyed by (leftVersion, rightVersion) — calling snapshot() twice with no mutations returns the same object.

Limitations. Single-column equi-key per side (no composite keys), full recompute on mutation (no incremental update). The kernel is scalar JS, not SIMD — 1k × 1k runs in ~0.8 ms on the reference machine.

GeoArrowBase — geo columns, predicates, and spatial indexes

ArrowBase has dedicated geometry columns for local-first map UIs: point, linestring, and polygon. Coordinates are [lng, lat] in EPSG:4326. Bbox predicates use planar lon/lat bboxes; point dwithin uses Haversine meters. Geo helpers are ArrowBase extensions and do not change the TanStack-compatible operators / comparisonFunctions export lists.

import {
  Collection,
  bboxIntersects,
  bboxWithin,
  createEffect,
  createTransaction,
  defineSchema,
  dwithin,
  queryOnce,
  within,
} from 'arrowbase';
import { exportArrowIPC, importArrowIPC } from 'arrowbase/arrow';

const featureSchema = defineSchema({
  name: 'geo_features',
  columns: {
    __id: { type: 'uint32' },
    name: { type: 'utf8', valuesHeapBytes: 64 * 1024 },
    geometry: { type: 'point', nullable: true },
    route: { type: 'linestring', nullable: true },
    footprint: { type: 'polygon', nullable: true, autoCloseRings: true },
  },
});

const features = Collection.create(featureSchema, {
  capacity: 100_000,
  spatialIndex: { column: 'geometry', type: 'rtree', eager: true },
});

features.insert({
  __id: 1,
  name: 'Singapore depot',
  geometry: [103.8198, 1.3521],
  route: [
    [103.72, 1.30],
    [103.82, 1.35],
    [103.92, 1.40],
  ],
  footprint: [[
    [103.80, 1.30],
    [103.90, 1.30],
    [103.90, 1.40],
    [103.80, 1.40],
    // autoCloseRings adds the closing coordinate.
  ]],
});

Spatial query helpers work in queryOnce(...), live queries, and effects:

const viewport = [103.6, 1.2, 104.0, 1.5] as const;
const singapore = [103.8198, 1.3521] as const;

const visible = await queryOnce((q) =>
  q
    .from({ feature: features })
    .where(({ feature }) => bboxIntersects(feature.geometry, viewport))
    .select(({ feature }) => ({ id: feature.__id, name: feature.name }))
    .orderBy(({ $selected }) => $selected.id, 'asc'),
);

const fullyInsideBbox = await queryOnce((q) =>
  q
    .from({ feature: features })
    .where(({ feature }) => bboxWithin(feature.geometry, viewport))
    .select(({ feature }) => ({ id: feature.__id, name: feature.name }))
    .orderBy(({ $selected }) => $selected.id, 'asc'),
);

const nearby = await queryOnce((q) =>
  q
    .from({ feature: features })
    .where(({ feature }) => dwithin(feature.geometry, singapore, 30_000))
    .select(({ feature }) => ({ id: feature.__id, name: feature.name }))
    .orderBy(({ $selected }) => $selected.id, 'asc'),
);

const pointsInsideTheirPolygon = await queryOnce((q) =>
  q
    .from({ feature: features })
    .where(({ feature }) => within(feature.geometry, feature.footprint))
    .select(({ feature }) => ({ id: feature.__id, name: feature.name })),
);

Use createEffect for viewport enter/update/exit deltas:

const viewportEffect = createEffect({
  id: 'visible-features',
  query: (q) =>
    q
      .from({ feature: features })
      .where(({ feature }) => bboxIntersects(feature.geometry, viewport))
      .select(({ feature }) => ({
        id: feature.__id,
        name: feature.name,
        geometry: feature.geometry,
      }))
      .orderBy(({ $selected }) => $selected.id, 'asc'),
  onEnter: (event) => console.log('entered viewport', event.key),
  onUpdate: (event) => console.log('moved in viewport', event.previousValue, event.value),
  onExit: (event) => console.log('left viewport', event.key),
  onBatch: (events, ctx) => console.log(ctx.effectId, events.length),
});

await viewportEffect.dispose();

Optimistic edits use the same transaction shape as non-geo rows. Rollback restores geometry values, bbox cache, dirty index overlay state, and live-query/effect deltas:

const tx = createTransaction({
  mutationFn: async ({ transaction }) => {
    const res = await fetch('/api/features/sync', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify(transaction.mutations),
    });
    if (!res.ok) throw new Error('sync failed');
  },
});

tx.mutate(() => {
  features.update(1, (draft) => {
    draft.geometry = [103.85, 1.31];
  });
});

await tx.isPersisted.promise;

spatialIndex supports point, linestring, and polygon envelopes. It maintains a compact Flatbush base plus immutable mini-tree segments and a small dirty overlay, so sustained inserts/updates do not degrade into a full linear dirty scan and do not require compact(). Indexed bboxIntersects, bboxWithin, and dwithin queries still run the original predicate as a residual exact filter after candidate lookup.

Arrow IPC is not Parquet. exportArrowIPC() and importArrowIPC() read and write Arrow IPC streams/files with GeoArrow extension metadata (geoarrow.point, geoarrow.linestring, geoarrow.polygon). GeoParquet would be a separate export path; do not treat these bytes as Parquet.

const bytes = await exportArrowIPC(features);
const restored = await importArrowIPC(bytes, { name: 'restored_features' });

Apache Arrow IPC — arrowbase/arrow

Optional subpath for interop with the broader Arrow ecosystem (arrow-js, DuckDB-WASM, Polars-WASM, pyarrow, Arrow Flight…). Install the peer dependency once:

pnpm add apache-arrow

Export a collection as an IPC byte stream or file:

import {
  exportArrowIPC,
  importArrowIPC,
  exportArrowTable,
} from 'arrowbase/arrow';

const bytes = await exportArrowIPC(collection);           // stream (default)
const file  = await exportArrowIPC(collection, { format: 'file' });
const table = await exportArrowTable(collection);         // arrow.Table directly

// Anywhere downstream — arrow-js, DuckDB-WASM, pyarrow, Arrow Flight:
// tableFromIPC(bytes) consumes this byte stream.

const rehydrated = await importArrowIPC(bytes);

Supported type mapping (both directions):

| ArrowBase | arrow-js | |---------------------|-------------------------| | bool | Bool | | int8/16/32/64 | Int8/16/32/64 | | uint8/16/32/64 | Uint8/16/32/64 | | float32/64 | Float32/Float64 | | utf8 | Utf8 | | dict_utf8 | Dictionary<Int32,Utf8>| | binary | Binary | | list<T> | List<map(T)> | | struct<…> | Struct<…> | | point | geoarrow.point (FixedSizeList<Float64>[2]) | | linestring | geoarrow.linestring (List<FixedSizeList<Float64>[2]>) | | polygon | geoarrow.polygon (List<List<FixedSizeList<Float64>[2]>>) |

Geo columns use GeoArrow extension metadata with { crs: 'EPSG:4326', edges: 'planar' }. Arrow IPC is not Parquet: this subpath serializes Arrow streams/files only, not GeoParquet files.

Internal columns. __deleted is stripped from the Arrow schema by default and soft-deleted rows are dropped. Pass { includeInternalColumns: true } for a loss-less round-trip.

Import inference. importArrowIPC() synthesizes an ArrowBase schema from the Arrow Schema. Columns using Timestamp, Decimal, Date_, Map, etc. throw with a clear error — those map-to-ArrowBase semantics are out of scope for v1. Pass { capacity, name } to size or label the rehydrated collection.

Not zero-copy. Export materializes rows and feeds vectorFromArray; import walks table.toArray(). ArrowBase uses ranged offsets for O(1) updates and keeps dict tables in JS heap, so wire compatibility with Arrow's dense-offset layout needs a rebuild. This is a serialization path, not a hot one.

IndexedDB persistence — arrowbase/idb

Durable storage for long-lived client apps. Loads a snapshot on boot, auto-saves with debounce on every mutation, flushes on demand.

import { defineSchema } from 'arrowbase';
import { IdbPersistence } from 'arrowbase/idb';

const schema = defineSchema({ /* ... */ });

const persist = await IdbPersistence.open(schema, {
  dbName: 'my-app',
  key: 'features',
  capacity: 10_000,
  // format: 'arbs' (default) or 'arrow-ipc' (requires apache-arrow).
});

const collection = persist.collection; // either restored or fresh
persist.start({ debounceMs: 500 });

window.addEventListener('beforeunload', () => { persist.flush(); });

Snapshot format is pluggable:

• 'arbs' (default) — native binary snapshot. Fast, preserves ranged offsets, round-trips soft-deletes byte-for-byte.
• 'arrow-ipc' — Apache Arrow IPC file, portable across arrow-js, DuckDB-WASM, pyarrow, etc. Requires the apache-arrow peer dep (dynamically imported only when this format is selected).

Storage model: one IDB object store (default 'snapshots'), caller supplies a key per collection. Each record holds { format, bytes, version, savedAt }, keeping snapshots self-describing so IdbPersistence.open() picks the right decoder. Schema-fingerprint mismatches are rejected with an actionable error.

Lifecycle:

• start({ debounceMs, saveImmediately }) — subscribe to the collection and auto-save. Bursts of mutations coalesce into a single IDB write per window (default 250 ms).
• flush() — bypass the debounce and await the IDB transaction.
• stop() — cancel any pending write + unsubscribe. Does not close the underlying IDB connection.
• close() — stop() + release the IDB connection. Idempotent.

For cache-wipe UX:

import { deleteStoredSnapshot } from 'arrowbase/idb';
await deleteStoredSnapshot('features', { dbName: 'my-app' });

Cross-tab sync — arrowbase/broadcast

Propagate mutations between same-origin tabs (or iframes, or shared workers) via the platform BroadcastChannel API. Every instance on the same channel name shares state; no server round-trip.

import { Collection, defineSchema } from 'arrowbase';
import { BroadcastSync } from 'arrowbase/broadcast';

const schema = defineSchema({ /* ... */ });
const collection = Collection.create(schema, { capacity: 10_000 });

const sync = new BroadcastSync(collection, { channelName: 'users' });
sync.start();
// …mutations here are mirrored to every other tab in real time…

// Teardown:
sync.stop();

Protocol: four message types (delta, sync-request, sync-response, resync) tagged with a per-instance nodeId. Self-echoes are filtered; re-broadcast of inbound deltas is suppressed by a reentrancy counter; a per-peer cursor dedupes deltas that arrive both via the live stream and a racing sync-response. Late joiners automatically send a sync-request(version=0) on start() and peers reply with the relevant log slice — pass { requestSyncOnStart: false } to opt out.

Compact / rollback events publish a resync sentinel (those break incremental delta coherence). Peers receive via the onResync hook and should refetch a snapshot out of band, typically via IdbPersistence, then rejoin:

const sync = new BroadcastSync(collection, {
  channelName: 'users',
  onResync: async ({ fromNodeId, version }) => {
    // Rehydrate from IDB, then rejoin.
    await persist.restore();
    sync.requestSync(collection.globalVersion);
  },
});

Non-goals. Convergence under concurrent edits (last-writer-wins, no CRDT merge), delivery guarantees (BroadcastChannel is best-effort), and security (every same-origin document on the channel sees every message — don't broadcast secrets).

React — arrowbase/react

Three hooks, zero boilerplate, concurrent-mode safe (useSyncExternalStore):

• useLiveQuery(query) — subscribe to a Query, returns the latest { rows, version } snapshot, re-renders only when an observed column moves.
• useLiveQueryResult(collection, build, deps) — builder-closure variant that rebuilds the query when deps change. Saves callers from memoizing the Query object themselves.
• useCollectionVersion(collection) — thin subscription to collection.globalVersion; useful as a cache-bust key.

React is an optional peerDependency (^18 or ^19). Add it to your app dependencies and import from the subpath:

import { useLiveQuery, useLiveQueryResult, useCollectionVersion } from 'arrowbase/react';
import { query, f, fEq } from 'arrowbase';

function ActiveUsers({ users }) {
  const { rows } = useLiveQueryResult(
    users,
    (q) => q.filter(fEq(f('active'), true)).orderBy('name'),
  );
  return <ul>{rows.map((r) => <li key={r.__id}>{r.name}</li>)}</ul>;
}

Re-render economy is inherited from the executor's dep-tracking: a mutation to an unobserved column produces zero extra renders.

Benchmarks

pnpm bench runs the regression benches and fails if any exceeds its soft ceiling. pnpm bench:operations runs a manifest-checked matrix of all structural query operators, relational stages, join modes, GeoArrow predicates, differential backend shapes, and public CRUD operations. pnpm bench:all runs every benchmark suite in sequence. pnpm bench:spatial runs bounded GeoArrowBase scan, indexed small-candidate, and dirty-overlay query benches. pnpm bench:vs-tanstack runs the published head-to-head release bench against @tanstack/db for aggregate scan and memory-footprint comparisons. Latest local validation (Node 24):

| bench | mean | budget | |-----------------------------------------------------|----------|--------| | insert 100k int32 rows | 60.5 ms | 100 ms | | insert 10k utf8 rows | 12.7 ms | 50 ms | | filter+sort+paginate 10k (fn predicate, auto) | 0.76 ms | 1 ms | | filter+sort+paginate 10k (fn predicate, fallback) | 2.84 ms | 5 ms | | filter+sort+paginate 10k (vectorized DSL) | 0.56 ms | 1 ms | | reactive notify (fn predicate, auto) | 116 µs | 150 µs | | reactive notify (fn predicate, fallback) | 328 µs | 500 µs | | reactive notify (vectorized DSL) | 94 µs | 150 µs | | string contains 10k (dict_utf8, pre-resolved) | 0.67 ms | 1 ms | | string contains 10k (utf8, per-row decode) | 2.74 ms | 5 ms | | string LIKE 10k (utf8, simple-pattern specialization) | 3.22 ms | 5 ms | | string startsWith 10k (dict_utf8) | 0.58 ms | 1 ms | | snapshot export + import 10k | 101.1 ms | info |

Insert benches use the raw column-buffer path so the regression guard tracks storage-kernel throughput separately from TanStack-compatible optimistic transaction overhead, which is covered by the transaction suites.

Vectorized filter+sort and reactive-notify paths beat the PRD v2.1 §10 aspirational targets (1 ms filter+sort, 100 µs reactive notify), while auto-lowered .where(fn) remains inside the release budget. Auto-lowering keeps the gap between common .where(fn) predicates and hand-written DSL small enough for ergonomic app code. On dict-encoded columns, the string predicates pre-resolve matching codes once and drop to a single Set.has() per row — ~4× faster than the utf8 per-row decode path.

Non-goals (see PRD.md §2.1)

• Cross-tab shared memory (SAB is same-document only). Cross-tab state replication is available via arrowbase/broadcast (message-based, last-writer-wins).
• Server-side durable SQL storage (use IndexedDB/localStorage client persistence, or bring your own server layer).
• Full SQL (query surface is TanStack-DB-shaped).
• Server-side use (browser + Node worker_threads only).
• Arrow compute kernels (possible later, out of scope v1).

Future work

• Nested list<struct> / list<list> / struct<struct>.
• SAB-resident dictionary + list child for cross-worker visibility.

License

MIT. See LICENSE.