backless-core

Core library for local-first SQLite sync via cloud storage.

Installation

npm install backless-core backless-google-drive   # or backless-onedrive

Usage

1. Define your schema

import type { DatabaseSchema, DB } from "backless-core";

export const mySchema: DatabaseSchema = {
  version: 1,
  syncedTables: new Set(["todos"]),

  async create(db: DB) {
    await db.exec(`
      CREATE TABLE todos (
        id TEXT PRIMARY KEY NOT NULL DEFAULT (uuid()),
        title TEXT NOT NULL DEFAULT '',
        done INTEGER NOT NULL DEFAULT 0
      )
    `);
  },

  async migrate(db: DB, from: number, to: number) {
    if (from < 2 && to >= 2) {
      await db.exec("ALTER TABLE todos ADD COLUMN priority TEXT NOT NULL DEFAULT 'normal'");
    }
  },

  async clear(db: DB) {
    // Do NOT call db.exec("BEGIN") here — clear() runs inside a transaction
    // started by backless.clearAllData(). Nesting BEGIN will throw.
    await db.exec("DELETE FROM todos");
    await db.exec("DELETE FROM todos__crsql_clock");
  },
};

syncedTables controls which tables get CRDT sync enabled. Only tables listed here are synced across devices — all others (cache tables, local UI state, etc.) are left as plain SQLite. Backless.init() calls crsql_as_crr on each declared table at startup. If a listed table doesn't exist or lacks a non-nullable primary key, startup throws immediately with a clear error.

2. Initialize Backless

Vite required for WASM — backless-core loads cr-sqlite via a ?url import (import wasmUrl from "...crsqlite.wasm?url"). This is a Vite-specific feature. If you use a different bundler (webpack, esbuild, Rollup), you need to copy the .wasm file to your public directory and provide its URL manually via the wasmUrl option in Backless.init.

import { Backless, VersionMismatchBehavior } from "backless-core";
import { GoogleDriveApi } from "backless-google-drive";
import { mySchema } from "./schema.js";

const backless = await Backless.init({
  schema: mySchema,
  databaseName: "myapp.db",
  versionMismatch: VersionMismatchBehavior.APPLY_COMPATIBLE,
  appFolderName: "MyApp",
});

const db = backless.database;

3. Read and write

await db.exec(
  "INSERT INTO todos (id, title) VALUES (?, ?)",
  [crypto.randomUUID(), "Buy milk"]
);

const todos = await db.execO<{ id: string; title: string; done: number }>(
  "SELECT id, title, done FROM todos ORDER BY rowid"
);

4. Sync

const drive = new GoogleDriveApi(async () => myGetAccessToken());

const result = await backless.sync(drive);
console.log(`Pulled ${result.pulled}, pushed ${result.pushed}`);
// result: { pulled, pushed, mediaUploaded, mediaDeleted, warnings, errors }

5. Sign in / sign out

Sign in — tell backless which account is active. If a different account was previously stored, all local data and media cache are cleared automatically before the new account is set:

await backless.signIn("google", "[email protected]");

Sign out — deactivates the account and wipes all local data and media cache by default:

await backless.signOut();

Pass options to suppress either clearing step (e.g. for session-expiry flows where you want to keep local data and only re-authenticate):

await backless.signOut({ clearData: false, clearMedia: false });

The account is always deactivated regardless of these flags.

Expired session — the user's cloud token expired but they haven't signed out. No backless call needed — just re-authenticate with your OAuth flow to get a fresh token. The database, cursors, and sync state are all still valid. Call sync() once you have a new token.

How sync works

Initialisation

Backless.init() does the following before returning:

1. Opens (or creates) the SQLite database file.
2. Creates two internal tables if they don't exist: _sync_meta and _sync_cursors.
3. Generates and persists a device_id (a UUID) if this is a new database. The device ID is stable for the lifetime of the database — it identifies this device's changeset folder in the cloud.
4. Runs schema.create() on a fresh database, or schema.migrate() if the stored schema_version is older than the current one.

The device ID and schema version are stored as rows in _sync_meta. They survive clearAllData() (so the device remains consistently identified after sign-out) but are permanently removed if the database file itself is deleted.

Cloud folder structure

All data lives under a single app folder in the user's cloud storage:

<appFolderName>/
├── README.txt                     ← dropped on first sync
├── changesets/
│   ├── device-<uuid-A>/
│   │   ├── snapshot-00100.json    ← full-state snapshot at sequence 100
│   │   ├── cs-00101.json
│   │   └── cs-00102.json
│   └── device-<uuid-B>/
│       └── cs-00001.json
└── media/                         ← binary attachments (if MediaSchema configured)

Each device writes only to its own folder. Other devices never write to each other's folders, so there are no cloud-level write conflicts.

A sync cycle: pull → push

sync() always pulls first, then pushes. A no-change sync with 3 devices typically costs 2-3 API calls thanks to folder caching and skip-unchanged optimizations.

Pull — download changesets from all remote devices in parallel, then apply them sequentially to avoid SQLite transaction conflicts. Unchanged device folders (same modifiedTime since last sync) are skipped entirely.

Push — extract local changes since the last push, upload as a new cs-NNNNN.json. Bails early with zero API calls if nothing changed.

After pull and push, sync() also runs stale device cleanup (prunes cloud folders inactive for 20+ days), pruned-device recovery (re-bootstraps if another device deleted our folder), snapshot compaction (creates snapshots every 100 pushes), and media sync (uploads/GCs binary files).

For the complete sync lifecycle, internal state tables, CRDT mechanics, optimizations, and edge cases, see docs/sync-architecture.md.

Conflict resolution

Backless uses cr-sqlite for CRDT semantics — last-writer-wins per column using Lamport timestamps (col_version). Concurrent edits to different columns of the same row are both preserved. Only same-column concurrent edits trigger LWW resolution. Deletes participate in LWW as tombstones. No manual conflict resolution is required.

Bootstrap recovery

After clearAllData() (e.g. sign-out) and re-sign-in, sync() automatically re-downloads this device's own changeset files and restores the full database state. No special app-level handling needed.

Account management

// Sign in — stores the account and clears local data if switching from a different account
await backless.signIn("google", "[email protected]");

// Restore session on page load
const account = await backless.getActiveAccount();
if (account) {
  // { provider: "google", email: "[email protected]" }
}

// Sign out — deactivates the account and clears all local data and media cache
await backless.signOut();

signOut() (and clearAllData()) remove the stored account so getActiveAccount() returns null afterwards.

Snapshot compaction

Backless automatically reduces cloud file accumulation over time. Every 100 pushed changesets a full-state snapshot (snapshot-NNNNN.json) is written to the device's cloud folder. After 200 more pushes the changeset files covered by that snapshot are deleted.

What this means for new devices

A device joining for the first time (cursor = 0) will download the latest snapshot instead of replaying every individual changeset file. This keeps initial sync fast regardless of history length.

What this means for existing devices

A device that hasn't synced in a while may find that changeset files it needs have already been deleted. When this happens sync() / pull() will add a warning to SyncResult.warnings for the affected remote device and skip it for that sync cycle.

const result = await backless.sync(drive);

for (const w of result.warnings) {
  if (w.message.includes("clearAllData")) {
    // This device has fallen too far behind a compacted remote.
    // Wipe local data and re-sync from the remote's latest snapshot.
    await backless.clearAllData();
    await backless.sync(drive);
    break;
  }
}

Constants

| Constant | Value | Meaning | |---|---|---| | SNAPSHOT_THRESHOLD | 100 | A snapshot is created every N pushed changesets | | SNAPSHOT_GRACE | 200 | Covered cs files are deleted this many pushes after the snapshot |

These are not currently configurable. The constants are exported from backless-core if you need to reference them.

Media sync

Backless can sync binary files (images, attachments, documents) alongside your CRDT changesets. Media is handled separately from the database: files are stored in a dedicated media/ subfolder in the cloud and loaded on demand — never eagerly pulled during sync.

Storage: Browser Cache API

Media blobs are stored locally using the browser's Cache API under the cache name "backless-media". Each file is keyed by its SHA-256 hash (https://backless-media/<hash>). This means:

• Persistent across page reloads — cached blobs survive navigation and browser restarts (until explicitly cleared or the browser evicts them under storage pressure).
• No IndexedDB or custom storage needed — the browser manages it natively.
• Deduplicated by content — two attachments with identical bytes share a single cache entry.

Upload flow

When the user attaches a file:

1. Hash it with hashFile(file) to get its SHA-256 content hash.
2. Store it locally with cacheMedia(hash, blob).
3. Record the attachment in your database (hash, filename, MIME type, etc.).

During sync(), backless calls mediaSchema.getUnuploadedMedia() to find attachments not yet in the cloud, then uploads them to <app-folder>/media/<shard>/ in the background. Files are sharded by the first two hex characters of their hash to keep cloud folder sizes manageable.

If a file is already present in the cloud (detected by filename), it is marked uploaded without re-uploading. If the local cache was cleared (e.g. after sign-out) before the upload completed, the item is skipped with a warning and will be retried on the next sync once the cache is populated again.

On-demand loading

Media is never downloaded during sync. When your UI needs to display an attachment, call:

const blob = await backless.resolveMedia(hash, filename, drive);
const url = URL.createObjectURL(blob);
img.src = url;

resolveMedia follows a cache-first strategy:

1. Check the local Cache API — return immediately on hit.
2. On miss, download from the cloud, store in the cache, then return.

Subsequent loads for the same hash are instant (cache hit), even across page reloads.

Sign-out and cache clearing

On explicit sign-out, call signOut() — it clears both the database and the media cache by default:

await backless.signOut(); // wipes SQLite data, sync state, and media cache

On the next sign-in, resolveMedia transparently re-downloads files from the cloud as they are needed. The CRDT sync restores the database; the media cache is rebuilt lazily on demand.

Session expiry (no sign-out) — if only the auth token expires, do not call signOut(). Local data and cached blobs are still valid; just re-authenticate and call sync().

Garbage collection

sync() automatically removes cloud media files whose hashes no longer appear in your mediaSchema.getAllMediaHashes() result — for example after an attachment is deleted. It also prunes the local Cache API of any entries not referenced by the current database state.

Implementing MediaSchema

import type { DatabaseSchema, MediaSchema, MediaItem, DB } from "backless-core";
import { getCachedMedia } from "backless-core";

function createMediaSchema(getDb: () => DB): MediaSchema {
  return {
    // Return attachments not yet uploaded to cloud
    async getUnuploadedMedia(): Promise<MediaItem[]> {
      const rows = await getDb().execO<{
        hash: string; original_filename: string; mime_type: string;
      }>(
        `SELECT a.hash, a.original_filename, a.mime_type
         FROM attachments a
         LEFT JOIN _media_status m ON a.hash = m.hash
         WHERE m.uploaded IS NULL OR m.uploaded = 0`
      );
      return rows.map(r => ({
        hash: r.hash,
        filename: r.original_filename,
        mimeType: r.mime_type,
        data: async () => {
          const blob = await getCachedMedia(r.hash);
          if (!blob) throw new Error(`Media ${r.hash} not in cache`);
          return blob;
        },
      }));
    },

    // Return hashes of all attachments currently in the database
    async getAllMediaHashes(): Promise<Set<string>> {
      const rows = await getDb().execO<{ hash: string }>(
        "SELECT DISTINCT hash FROM attachments"
      );
      return new Set(rows.map(r => r.hash));
    },

    // Called by backless after a file is successfully uploaded
    async markAsUploaded(hash: string): Promise<void> {
      await getDb().exec(
        "INSERT OR REPLACE INTO _media_status (hash, uploaded) VALUES (?, 1)",
        [hash]
      );
    },
  };
}

You also need a _media_status tracking table in your DatabaseSchema.create:

await db.exec(`
  CREATE TABLE IF NOT EXISTS _media_status (
    hash TEXT PRIMARY KEY,
    local_path TEXT,
    uploaded INTEGER DEFAULT 0
  )
`);
// Not listed in syncedTables — Backless leaves it as a plain local-only table

Attaching a file (full example)

import { hashFile, cacheMedia } from "backless-core";

const file = fileInputElement.files[0];
const hash = await hashFile(file);

// 1. Cache locally so the upload step and UI can access it
await cacheMedia(hash, file);

// 2. Record in database (CRDT-synced via changeset)
await db.exec(
  "INSERT INTO attachments (id, event_id, hash, original_filename, mime_type, size) VALUES (?, ?, ?, ?, ?, ?)",
  [crypto.randomUUID(), eventId, hash, file.name, file.type, file.size]
);

// Next sync() call will upload the file to cloud automatically

Backup & restore

Backless provides APIs to export all data and media as a portable archive and to restore from one.

exportData()

async exportData(): Promise<Record<string, Record<string, unknown>[]>>

Reads all rows from every synced table as plain objects (no CRDT metadata). Returns a map of table name → row arrays. Use this to build the data.json payload of a backup.

importData(data)

async importData(data: Record<string, Record<string, unknown>[]>): Promise<void>

Wipes the local database and inserts the provided rows. The database is left in a state where the next sync pushes everything as new changesets. Call clearCloudSync() before syncing to avoid merging stale cloud changesets with the restored data.

getAllMedia(drive, onProgress?, items?)

getAllMedia(
  drive: CloudStorageApi,
  onProgress?: (processed: number, total: number) => void,
  items?: ReadonlyArray<{ hash: string; filename: string; mimeType: string }>,
): AsyncGenerator<{ hash: string; filename: string; mimeType: string; blob: Blob }>

Yields media blobs one at a time (cache-first, cloud fallback). Optimised for bulk exports: checks the local cache in a single caches.open() call, groups downloads by shard folder to minimise listFiles round trips, and uses bounded worker pools for both shard listings and downloads.

Pass items to resolve a specific set of hashes — for example all photos referenced by your data tables — so that media uploaded from other devices is included even if not tracked in the local _media_status table. Omit items to fall back to everything in _media_status.

Items that fail to download are skipped silently. onProgress(processed, total) fires as each item resolves (during downloads) or is yielded (for cache hits).

importMedia(items)

async importMedia(
  items: Array<{ hash: string; filename: string; mimeType: string; blob: Blob }>
): Promise<void>

Bulk-imports media from a backup. Caches each blob locally and records it in _media_status with uploaded=0 so the next sync pushes it to cloud. Trusts the hashes from the backup source (no SHA-256 re-computation).

clearCloudSync(drive)

async clearCloudSync(drive: CloudStorageApi): Promise<void>

Removes all sync state from cloud: deletes every device folder under changesets/ and resets local sync cursors. Does not touch media/ or static exports — those are reconciled on the next sync.

Call this after importData() and before syncing so the next push starts fresh rather than merging with stale changesets.

API

Backless.init(config)

| Option | Type | Default | Description | |---|---|---|---| | schema | DatabaseSchema | required | Your app's schema | | databaseName | string | "backless.db" | SQLite database filename | | versionMismatch | VersionMismatchBehavior | ABORT | How to handle schema version mismatches during sync | | mediaSchema | MediaSchema \| null | null | Optional media sync support | | appFolderName | string | "Backless" | Cloud storage folder name (must match across all platforms) | | folderReadme | string | auto-generated | Custom content for README.txt dropped into cloud app folder on first sync | | autoCleanup | boolean | true | Enable automatic stale-device folder pruning | | cleanupMaxAgeDays | number | 20 | Days of inactivity before a device folder is eligible for pruning |

SyncOptions

All methods that trigger sync (sync(), pull(), push()) accept an optional SyncOptions object:

| Callback | When | |---|---| | signal | AbortSignal to cancel the sync mid-flight | | onStart() | Before any network I/O | | onPulled(n) | After pull completes (n = changes applied) | | onPushed(n) | After push completes (n = changes uploaded) | | onComplete(result) | After all phases complete successfully | | onError(error) | On any phase failure (mutually exclusive with onComplete) | | onRecovery(message) | When recovering from a pruned device folder |

VersionMismatchBehavior

| Value | Behaviour | |---|---| | ABORT | Stop sync if remote schema is newer, warn user to update | | SKIP_DEVICE_UNTIL_NEXT_SYNC | Skip that device this sync cycle, try again next time | | DROP_CHANGESETS | Discard newer changesets, advance cursor | | APPLY_COMPATIBLE | Apply only columns that exist locally, drop unknown ones |

DatabaseSchema interface

interface DatabaseSchema {
  readonly version: number;
  readonly syncedTables: ReadonlySet<string>;  // tables to enable CRDT sync on
  create(db: DB): Promise<void>;
  migrate(db: DB, oldVersion: number, newVersion: number): Promise<void>;
  clear(db: DB): Promise<void>;  // called by backless.clearAllData()
}

Media sync (optional)

Implement MediaSchema to sync binary files alongside your changesets:

interface MediaSchema {
  getUnuploadedMedia(): Promise<MediaItem[]>;
  getAllMediaHashes(): Promise<Set<string>>;
  markAsUploaded(hash: string): Promise<void>;
}

Pass it to Backless.init({ mediaSchema: myMediaSchema }).

Utilities for media handling:

• hashFile(blob) — compute SHA-256 hash for a blob
• cacheMedia(hash, blob) — store a blob in the local cache (call after the user picks a file)
• getCachedMedia(hash) — retrieve a cached blob by hash (use in MediaItem.data callbacks)
• backless.resolveMedia(hash, filename, drive) — cache-first resolution with cloud fallback

Troubleshooting

Table not syncing / "could not find the schema information for table X"

If a table's changes are silently dropped or you see a cr-sqlite error about a missing table, the table is probably not listed in syncedTables.

// ❌ attachments rows are never synced — not in syncedTables
export const mySchema: DatabaseSchema = {
  version: 1,
  syncedTables: new Set(["events"]),   // missing "attachments"
  async create(db) {
    await db.exec("CREATE TABLE events (...)");
    await db.exec("CREATE TABLE attachments (...)");
  },
  ...
};

// ✅ correct
  syncedTables: new Set(["events", "attachments"]),

Table name typo in syncedTables

If a table listed in syncedTables doesn't exist after create() / migrate() runs, Backless.init() throws immediately:

Error: Failed to enable CRDT for table 'evnets'. Make sure the table is created
in schema.create() / schema.migrate() and has a non-nullable primary key.

Fix the typo or ensure create() creates the table before init returns.

schema.clear() must not start its own transaction

backless.clearAllData() wraps the entire clear operation — including the call to schema.clear(db) — in a single BEGIN/COMMIT transaction. If your clear implementation issues its own BEGIN, SQLite will throw because nested transactions are not supported via BEGIN (only savepoints are).

Wrong:

async clear(db: DB) {
  await db.exec("BEGIN");          // ❌ throws — already inside a transaction
  await db.exec("DELETE FROM todos");
  await db.exec("COMMIT");
},

Correct:

async clear(db: DB) {
  await db.exec("DELETE FROM todos");          // ✅ runs inside the outer transaction
  await db.exec("DELETE FROM todos__crsql_clock");
},

WASM loading only works with Vite

backless-core imports the cr-sqlite WASM binary using the Vite ?url suffix:

import wasmUrl from "@vlcn.io/crsqlite-wasm/crsqlite.wasm?url";

This is a Vite-specific feature — other bundlers (webpack, esbuild standalone, Rollup without the correct plugin) do not understand ?url imports and will fail at build time.

If you are not using Vite, copy node_modules/@vlcn.io/crsqlite-wasm/crsqlite.wasm to your public/static directory and pass its URL explicitly:

const backless = await Backless.init({
  schema: mySchema,
  wasmUrl: "/static/crsqlite.wasm",  // served as a static asset
});