@dmytromykhailiuk/data-store

Offline-first IndexedDB datastore with a persistent outbox, push/pull sync, conflict resolution, multi-table transactions and a typed filter DSL — framework-agnostic, no vendor lock-in.

Full documentation: open Docs in a browser — every option, with examples, a table of contents and cross-links. This README is the short form.

Built for apps that must keep working with no connection at all — and reconcile honestly when it returns. IndexedDB is the single source of truth, every mutation commits atomically with its queued upload, and everything the server has not acknowledged survives reloads, crashes and going offline for a week. The transport is yours — plain push/pull functions per table; swap GraphQL for REST without touching store code.

Install

npm i @dmytromykhailiuk/data-store

The push/pull engines read the online state from @dmytromykhailiuk/network-connection — NetworkConnection.init() must run before store.start(), otherwise start() rejects immediately with a clear error.

Quick start

import { NetworkConnection } from "@dmytromykhailiuk/network-connection";
import { ConflictError, createDataStore, defineTable } from "@dmytromykhailiuk/data-store";

// 1. Declare tables: model type + the slice of sync params they consume.
const store = createDataStore<{ eventId?: string }>({
  name: "InspectorDataStore",
  schemaVersion: 1,
  ignoreFields: ["_version", "updatedAt"], // server-managed fields
  tables: {
    Issue: defineTable<Issue, { eventId?: string }>({
      primaryKey: "id",
      indexes: { byEvent: { key: "eventId" } },
      scope: {
        key: (p) => p.eventId ?? null,                    // which slice to mirror
        keep: (issue, p) => issue.eventId === p.eventId,  // what survives a switch
      },
      push: {
        create: async (issue) => (await api.createIssue(issue)).data,
        update: async (issue) => {
          try { return (await api.updateIssue(issue)).data; }
          catch (e) {
            if (api.isVersionMismatch(e)) throw new ConflictError({ remote: api.remoteOf(e) });
            throw e; // transport errors retry with backoff + offline parking
          }
        },
        delete: async (issue) => { await api.deleteIssue(issue.id, issue._version); },
      },
      pull: {
        fetch: async ({ params, checkpoint, signal }) => {
          const page = await api.issuesByEvent(params.eventId!, { since: checkpoint, signal });
          return { items: page.items, checkpoint: page.startedAt, done: !page.nextToken };
        },
        fetchOne: (id) => api.getIssue(id),
      },
      merge: (local, remote) => ({ ...remote, status: local.status }),
    }),
    Draft: defineTable<DraftNote>({ primaryKey: "id", local: true }), // local-only
  },
});

// 2. Start: opens IndexedDB (running migrations), restores the outbox, pulls.
await NetworkConnection.init("/healthcheck.txt");
await store.start({ params: { eventId: route.eventId } });
await store.whenSynced();

// 3. Work with tables — everything survives reloads and offline.
const issues = store.table("Issue");
await issues.put({ id: crypto.randomUUID(), eventId, severity: 3, status: "OPEN" });
await issues.update(id, { status: "RESOLVED" });
const open = await issues.query({ status: { eq: "OPEN" }, severity: { ge: 2 } });

// 4. React.
issues.subscribe({ eventId: { eq: eventId } }, ({ type, item, origin }) => render(item));
await store.whenUploaded(); // outbox drained — safe to log out

How it works

• IndexedDB is the single source of truth — no in-memory mirror to drift out of sync; reads are consistent snapshots.
• Records are stored wrapped — { key, data, meta }; your fields never mix with bookkeeping (state, revision counter, sync scope, last error).
• One write path — every mutation takes the table's FIFO lock (execution-blocker) and runs one readwrite transaction spanning the data and the outbox. Locks are never held across the network.
• The outbox is data — queued uploads commit atomically with the records they belong to and are rebuilt from disk on every start. Mutations of one record coalesce (create+update→create, create+delete→nothing, delete+put→update) without losing their FIFO position.
• The transport is yours — the store understands exactly two special errors: ConflictError (merge → one re-push → surface) and FatalPushError (straight to the error state). Everything else is a transport failure: exponential backoff that parks while offline.

Record lifecycle: pending → pushing → synced, with error for surfaced failures (revive via retryFailed() / resolveFailed() / a newer save) and local for records that never push. A deleted server-known record is a hidden tombstone until the server confirms; a save mid-push can never be lost — the pipeline re-checks the revision counter after every network call.

When a pull meets a record with unpushed changes, it rebases it by default: the table's merge(local, remote) runs and the record stays queued with the merged data — the eventual push carries fresh server-managed fields instead of a stale base (important on last-write-wins backends). Opt out per table with pull.mergePending: false; tombstones and error-state records are never rebased.

Data arriving outside the pull loop (WebSocket snapshots, SSR payloads) is applied imperatively with the exact same semantics — no pull config required:

socket.on("issues", (items) => void store.table("Issue").applyRemote(items));

Transactions

await store.transaction(["Event", "Issue"], async (tx) => {
  await tx.table("Event").update(eventId, { status: "COMPLETED" });
  for (const issue of resolved) await tx.table("Issue").delete(issue.id);
});

One IndexedDB transaction over the listed tables and the outbox: everything commits or rolls back together, events fire only after the commit, locks are acquired in sorted order (no deadlocks). Don't await the outside world inside — IndexedDB auto-commits (you'll get a TransactionInactiveError explaining this).

Queries

Both a typed, serializable DSL and plain predicates, always over full records:

await issues.query({
  and: [
    { status: { in: ["OPEN", "TRIAGED"] } },
    { severity: { between: [2, 4] } },
    { or: [{ title: { contains: q } }, { id: { beginsWith: q } }] },
  ],
}, { limit: 50 });

await issues.query((issue) => issue.tags.length > 3);

Operators are type-checked per field ({ severity: { beginsWith } } doesn't compile); invalid conditions throw a loud SchemaError instead of silently matching everything.

Queries are purely local by default. With a pull.query handler declared, { remote: true } fetches first, persists the result with full pull semantics, then answers locally:

await issues.query({ status: { eq: "OPEN" } }, { remote: true });

Conflicts

A push handler throws ConflictError({ remote? }) → the store merges (merge(local, remote), default: local wins except ignoreFields, which come from the remote — list your version field there), re-pushes once, and surfaces a second conflict as an error-state record:

await issues.resolveFailed(key, (local, remote) => {
  if (!remote) return "discard-local";
  return { ...remote, note: local.note }; // or "keep-local"
});

Sync scopes & params

await store.setParams({ eventId: next });

Tables whose scope.key(params) changed abort their pull, evict records failing scope.keep (never records with unpushed changes), and pull the new scope — resuming from its persisted checkpoint if it synced before. Untouched scopes stay marked synced: navigating back is instant.

Migrations

createDataStore({
  schemaVersion: 3, // structure (tables/indexes) reconciles automatically on bump
  migrations: {
    3: async ({ table }) => {
      await table("Issue").updateEach((i) => (i.status === "OPENED" ? { ...i, status: "OPEN" } : i));
    },
  },
  ...
});

A migration that throws aborts the whole upgrade — nothing half-commits. Adding a table without bumping the version fails fast with a SchemaError.

Binary fields (the iOS Blob bug)

Some WebKit builds throw DataCloneError when a Blob hits IndexedDB. Declare where your blobs live — on affected browsers they are transparently stored as ArrayBuffers and come back as real Blobs on every read:

createDataStore({
  binary: { mode: "auto" }, // feature-probe at start(); "always"/"never" to force
  tables: {
    Photo: defineTable<Photo>({
      primaryKey: "id",
      blobPaths: ["preview", "attachment.file", "frames"], // typed dot-paths, arrays ok
    }),
  },
});

Reads, events, push handlers and merge always see real Blobs (merge must treat them as opaque — pick a side whole). Inside store.transaction() live Blobs are rejected on every platform — pre-encode with await table.encodeBlobs(item).

Testing

Runs unmodified on fake-indexeddb; alias @dmytromykhailiuk/network-connection to a controllable double (inline @dmytromykhailiuk/retry-request so it sees the same mock). This package's own 145-test suite is written exactly that way and doubles as a cookbook.

TypeScript

Everything is inferred from defineTable<Model, ParamsSlice>(): store.table("Issue") is a TableStore<Issue>, filters are checked against Issue's fields, transaction views only accept the declared tables. Errors form one hierarchy (DataStoreError with a stable code) — branch on instanceof, never on message text.

License

MIT