nosql-sync-kit

Offline-first document sync for browsers. Your app reads and writes a local store, and the SDK syncs it in the background with a remote database:

• CouchDB, directly from the browser over its HTTP API, using CouchDB's own revisions.
• MongoDB (or any other store), through a small server that the SDK provides. Browsers can't speak the MongoDB wire protocol, so the server is the bridge.
• Your own backend, by implementing four methods on the server side and/or a custom adapter in the browser.

The package has three entry points:

| Import | Use it in | Contents | |---|---|---| | nosql-sync-kit/client | browsers | createClient() and the lower-level engine, stores, and adapters | | nosql-sync-kit/server | Node.js 18+ | createSyncServer(), createRequestHandler(), storage backends | | nosql-sync-kit/testing | Node tests | runStorageConformance() for checking your own storage backend |

It also ships a nosql-sync-kit command that runs the server from environment variables.

Install

npm install nosql-sync-kit
npm install mongodb   # only if you use the MongoDB storage

Client

import { createClient } from 'nosql-sync-kit/client';

const client = await createClient({
  name: 'notes',                                   // local database name
  remote: { type: 'gateway', url: '', token: '' }, // see "Remotes" below
});

await client.put('note:1', { title: 'Groceries' });
const doc = await client.get('note:1');            // { _id, data, synced } or null
const all = await client.all();                    // every document that isn't deleted
await client.remove('note:1');

client.on('change', ({ id, origin }) => render());
client.on('status', (s) => console.log(s.state, s.pending));

createClient(options)

| Option | Default | Meaning | |---|---|---| | name | 'browser-sync' | Name of the IndexedDB database. Use a different name per app or user. | | remote | null | null for offline only, or a remote (see below). | | conflict | 'local' | 'local', 'remote', or (local, remote) => { data, deleted }. | | pollInterval | 3000 | Milliseconds between pulls. 0 turns polling off. | | pageSize | 100 | Documents per request. | | validate | none | (id, data) => void. Throw to reject a put(). | | store | IndexedDB | Your own local store. Falls back to memory when IndexedDB is unavailable. | | crossTab | true | Tell other tabs about changes with BroadcastChannel. |

Client methods

| Method | Returns | |---|---| | get(id) | { _id, data, synced } or null | | all() | array of { _id, data, synced } | | put(id, data) | the saved document. Runs validate first. | | remove(id) | undefined. The delete syncs as a tombstone. | | sync() | a promise for the status after syncing. Rejects if the remote can't be reached. | | status() | { state, lastSyncAt, pending, error } | | on(type, fn) | an unsubscribe function | | close() | stops polling and listeners. Local data stays on disk. |

synced is false while a local change is waiting to be pushed.

Events

| Event | Payload | When | |---|---|---| | change | { id, origin, deleted? }. origin is 'local', 'remote', 'conflict', or 'other-tab'. | A document changed. | | status | { state, lastSyncAt, pending, error }. state is 'local-only', 'idle', 'syncing', or 'error'. | Sync state changed. | | conflict | { id, local, remote, resolved }, where each is { data, deleted }. | A remote edit and an unsynced local edit collided. | | error | { id, error } | The remote rejected one document. |

Remotes

// browser-sync server (MongoDB or any server-side storage)
{ type: 'gateway', url: 'https://sync.example.com', token: 'optional-bearer-token' }

// CouchDB, directly
{ type: 'couchdb', url: 'http://localhost:5984', database: 'notes', username: 'app', password: '...' }

// Your own adapter object (see "Writing a remote adapter")
{ info, changes, bulk, fetch }

Conflicts

Writes carry the revision they were based on. If the remote has moved on, the write is a conflict. The SDK fetches the remote version and applies your conflict strategy:

• 'local' (default): keep the local edit and push it on top of the remote version.
• 'remote': drop the local edit.
• A function: merge however you like.

createClient({
  remote,
  conflict: (local, remote) => ({
    data: { ...remote.data, ...local.data, tags: [...new Set([...remote.data.tags, ...local.data.tags])] },
    deleted: false,
  }),
});

Each conflict emits a conflict event, so you can tell the user.

Lower-level pieces

SyncEngine, LocalDB, openIdbStore, createMemoryStore, createCouchRemote, and createGatewayRemote are exported too, for apps that build their own wiring.

Server

import { createSyncServer, createMemoryStorage, createMongoStorage } from 'nosql-sync-kit/server';

const storage = await createMongoStorage({
  url: 'mongodb://127.0.0.1:27017',
  dbName: 'myapp',
});

createSyncServer({
  storage,
  token: process.env.SYNC_TOKEN,                  // optional bearer token for /api
  staticMounts: [{ prefix: '/', dir: './dist' }], // optional: serve your web app too
}).listen(8787);

createSyncServer(options) and createRequestHandler(options)

createSyncServer returns a Node http.Server. createRequestHandler returns a (req, res) function, for mounting in an existing server.

| Option | Default | Meaning | |---|---|---| | storage | required | Any object with info, changes, bulk, fetch. | | token | '' | If set, every API request needs Authorization: Bearer <token>. Compared in constant time. | | basePath | '/api' | Where the API is mounted. | | cors | '*' | Access-Control-Allow-Origin value, or false to disable. | | maxBodyBytes | 5 MB | Largest accepted request body. | | maxBulk | 500 | Most documents per bulk or fetch call. | | staticMounts | [] | [{ prefix, dir, index? }]. Matched by longest prefix. | | onError | console.error | Called for unexpected errors. |

The handler rejects field names that start with $, because MongoDB treats them as operators.

HTTP API

| Method and path | Body or query | Response | |---|---|---| | GET /api/info | | storage info | | GET /api/changes?since=&limit= | limit 1–500 | { changes: [{ _id, rev, deleted, data, seq }], seq, more } | | POST /api/bulk | { docs: [{ _id, baseRev?, deleted?, data? }] } | { results: [{ _id, ok, rev } \| { _id, conflict: true }] } | | POST /api/fetch | { ids: [...] } | { docs: [{ _id, rev, deleted, data }] } |

Command line

STORAGE=mongodb MONGODB_URL=mongodb://127.0.0.1:27017 SYNC_TOKEN=change-me \
STATIC_DIR=./dist npx nosql-sync-kit

Variables: PORT (8787), STORAGE (memory or mongodb), MONGODB_URL, MONGODB_DB, MONGODB_COLLECTION, SYNC_TOKEN, CORS_ORIGIN, STATIC_DIR.

Storage backends

Storage is the server-side half of a remote. It implements four methods:

info()                    // -> { name, ... }
changes({ since, limit }) // -> { changes: [{ _id, rev, deleted, data, seq }], seq, more }
bulk(docs)                // docs: [{ _id, baseRev, deleted, data }]
                          // -> [{ _id, ok: true, rev } | { _id, conflict: true } | { _id, error }]
fetch(ids)                // -> [{ _id, rev, deleted, data }]  (missing ids are omitted)

• bulk must accept a write only when baseRev equals the stored revision, or when there is no stored document and no baseRev. That compare-and-set is what makes conflicts work.
• seq must increase with each write, and changes({ since }) must return only entries after since. A document written several times should appear once, with its latest state.
• Deletes are tombstones: deleted: true, with data cleared. Keep them, so other clients learn about the delete.

Bundled backends: createMemoryStorage() (process memory) and createMongoStorage() (MongoDB collection plus a counter document for seq).

Writing a storage backend

1. Implement the four methods above.
2. Check it with the conformance suite:

// test/redis-storage.test.js
import { runStorageConformance } from 'nosql-sync-kit/testing';
import { createRedisStorage } from '../src/redis-storage.js';

runStorageConformance({
  name: 'redis',
  create: () => createRedisStorage({ url: process.env.REDIS_URL, prefix: `test:${Date.now()}` }),
  cleanup: (storage) => storage.close(),
  skip: !process.env.REDIS_URL && 'set REDIS_URL to run',
});

The suite covers creates, conflicts, stale writes, tombstones, fetch, ordering, paging, repeated writes, and partial batch failures.

| Store | How to do the compare-and-set | |---|---| | MongoDB | updateOne({ _id, rev: baseRev }), or insertOne when there is no base (a duplicate _id is a conflict) | | PostgreSQL | UPDATE ... WHERE id = $1 AND rev = $2 | | Redis | WATCH and MULTI, or a Lua script | | DynamoDB | ConditionExpression: rev = :base | | Firestore | a transaction that reads rev and compares it |

Writing a remote adapter

To sync with a backend that has no server piece, implement the same four methods in the browser and pass the object as remote. createClient checks that all four are present.

Testing your app

Use createMemoryStorage() as the remote, and createMemoryStore() as the local store, with crossTab: false and pollInterval: 0. Then call client.sync() explicitly. The SDK's own tests do this; see test/client.test.js.

CouchDB setup

The browser calls CouchDB directly, so CouchDB must allow CORS for your app's origin. In local.ini:

[httpd]
enable_cors = true

[cors]
origins = http://localhost:8787
credentials = true
headers = accept, authorization, content-type
methods = GET, POST, PUT, DELETE, OPTIONS

Use a dedicated CouchDB user with access only to the app's database. Anyone who can read the browser profile can read the password that the app stores in localStorage (the example app does this, for convenience).

Limitations

• Polling, not push. Changes arrive within pollInterval. Use pollInterval: 0 and call sync() yourself if you want to control timing.
• MongoDB sequence gaps. With concurrent writers, a lower seq can commit after a higher one has already been read, so a reader can skip it. The planned fix is to re-read a small overlap window. Applying a revision you have already seen is harmless, so the overlap is safe.
• Attachments are not supported. Documents are JSON.
• One winning revision. CouchDB conflicts from other writers are resolved to the winning revision (style=main_only).
• Authentication is minimal. The server uses one shared bearer token. Add per-user auth in front of it for multi-user deployments.

Development

npm test          # from the repo root
MONGODB_URL=mongodb://127.0.0.1:27017 npm test   # also runs the MongoDB conformance suite

License

Apache License 2.0. See LICENSE.