@jaaahn/hyper-sync

v1.0.0

Published

a month ago

HyperSync is a lightweight library designed to make data synchronization between clients and servers seamless and efficient.

0High
0Medium
0Low

jaaahn

lightweight real-time sync library client-server

HyperSync

HyperSync is a lightweight library designed to make data synchronization between clients and servers seamless and efficient. It is perfect for building apps that need to work offline and need to synchronize data across multiple devices. With HyperSync, you can:

Easily Sync Data: Automatically push local changes to the server and pull remote updates with minimal setup.
Offline Support: Queue changes locally when offline and sync them once the connection is restored.
Conflict Handling: Resolve data conflicts using timestamps to ensure the latest updates are applied (last-write-wins).
Customizable: Define your own data structures and synchronization logic to fit your app's needs.
Agnostic: Works with any database (e.g. Dexie on the web and SQL on the server) or web framework
Developer-Friendly: Simple APIs for configuring sync, managing data, and handling errors.

HyperSync is built to save you time and effort, so you can focus on building great features for your users.

Note: this documentation was generated with AI and heavily refined by Humans. If you find any unclear sections, mistakes or misspellings, please do not hesistate to contact me.

Important Concepts Before We Start

(You can use any database, both for the client and the server. We chose to write this guide for PostgresDB and Dexie.js)

Required Data Structure for the Server Database

For sync, every row needs at least one primary key (which we consider part of the payload), userId, updatedAt and lastSyncAt

CREATE TABLE IF NOT EXISTS todos (
    todoId TEXT NOT NULL PRIMARY KEY,
    title TEXT NOT NULL,
    done BOOLEAN NOT NULL DEFAULT false,
    deleted BOOLEAN NOT NULL DEFAULT false,

    userId TEXT NOT NULL,
    updatedAt TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    lastSyncAt TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

lastSyncAt however is only used on the server. It does not exist on the client.

Required Data Structure for the Web Database

export const offlineDb = new Dexie("local-db");

offlineDb.version(1).stores({
    todos: "&todoId, title, done, deleted, userId, updatedAt, [todoId+user_id]",
});

Updating updatedAt everytime the data in the web-client changes is important.

"updatedAt" vs. "lastSyncAt"

When using this library, there are moments where updatedAt is used and then there are moments where lastSyncAt is used. It is important for you to understand, when you have to use which of these.

In the server database, each row is associated with two columns containing timestamps (with time zone)

updatedAt → When the data itself last changed (on any client).
lastSyncAt → When the row was last written to the server database.

For synchronization, lastSyncAt is the correct checkpoint.

Reason: A record may have been created or modified offline a long time ago (updatedAt is old), but only much later uploaded to the server. When it is finally uploaded, lastSyncAt is set to NOW().

If clients pull changes using WHERE updatedAt > lastPull they may MISS records that were uploaded recently but whose updatedAt is older than the client's lastPull.

If instead we use: WHERE lastSyncAt > lastPull we reliably fetch everything that was written to the server since the last pull — regardless of when the data originally changed.

Example:

Device A creates an item while offline.
Device B is used regularly and performs pulls (so its lastPull becomes recent).
Much later, Device A comes online and uploads the item.

The item's updatedAt is old, but its lastSyncAt is new. If Device B compares against updatedAt, it will not receive the item. If it compares against lastSyncAt, it will. Conflict resolution (upsert vs discard) should then be decided on the client using updatedAt, because it describes when the data was last touched.

Summary:

updatedAt → when the data changed => set it to new Date() when modifying the local database
lastSyncAt → when the server received it (sync checkpoint) => just accept

Primary Keys / Unique Object Identifiers

For object identifiers, you want to use something like uuid v4 to reliably generate unique and random identifiers on each client without requiring a central orchestrator (e.g. a server). In other cases, you might even want to generate ids deterministically.

Place identifiers, e.g. todoId, inside the payload. This way, you can expect the identifier to be present when a function receives the payload object from HyperSync.

We do however not consider userId and updatedAt as being part of payload, thus there will be separate parameters for that data.

Getting Started

Install

npm install @jaaahn/hyper-sync

Imports

import { configureSync, getSyncEngine, useShape as useClientShape } from "@jaaahn/hyper-sync/web";
import { useExpress, useShape as useServerShape } from "@jaaahn/hyper-sync/server";

1) Web: Configure the Web Client

const sync = configureSync({
    serverEndpoint: "https://api.example.com/sync",
    syncOnWindowFocus: true, // Defaults to true
    syncInterval: 120_000, // Defaults to 2 minutes
});

sync.setUserId("user-123"); // Whenever the user changes, call this function (e.g. from an event listener / watcher)

2) Web: Register a Web Shape

const todosShape = useClientShape({
    shapeKey: "todos",
    syncEngine: sync,
    upsertRemoteChangeFn: async (payload, userId, updatedAt) => {
        // Apply server change to your local store (IndexedDB, state, etc.)
        // Expect "todoId" to be part of "payload"
        await upsertTodoLocally(payload, userId, updatedAt);

        // Throw if payload is invalid; onUpsertErrorFn (if set) receives that error
    },
});

// Queue a local change and trigger sync
await todosShape.pushChange(
    { todoId: "t1", title: "Write docs", description: "README task", done: false, deleted: false }, // payload, including "todoId"
    "user-123", // userId
    new Date() // updatedAt
);

// It's important to tombstone entries via e.g. a "deleted" flag

3) Server: Mount the Express Router

import express from "express";

const app = express();
app.use(express.json());

const todosServerShape = useServerShape({
    shapeKey: "todos",
    upsertChangeFn: async (change) => {
        // Persist incoming change from client

        /**
         * If you throw an error, this change will not be accepted. If one change is rejected, it doesn't break the entire
         * sync layer. If other changes succeed, these will be sent as accepted changes.
         * If a change is rejected, the client will retry it in the next sync event until it is accepted.
         */
        if change.payload.todoId == undefined) throw new Error("invalid-change");

        /**
         * Anatomy of a Change:
         * - changeId
         * - shapeKey
         * - payload (which must include the object identifier, e.g. todoId)
         * - userId (which is going to match the user who has been verified in the request)
         * - updatedAt (when the data was modified on the client)
         */
        await upsertTodoInDb(change); // Store the change (insert new or update existing entry) in the database

        // DO NOT FORGET to set `lastSyncAt` to `NOW()` when upserting data in the server datbase
    },
    pullChangesFn: async (lastPull, syncedAt, userId) => {
        /**
         * Please return all rows that have `todo.lastSyncAt > lastPull`
         * and `todo.lastSyncAt <= syncedAt` and `todo.userId = userId`
         */

        // Return all rows updated in (lastPull, syncedAt] for this user
        const rows = await loadTodoChanges(lastPull, syncedAt, userId);

        // Return the data in the required format
        return rows.map((todo) => ({
            payload: {
                todoId: todo.todoId,
                title: todo.title,
                description: todo.description,
                done: todo.done,
                deleted: todo.deleted,
                // ... or anything you like, this is just an example
            },
            userId: todo.userId,
            updatedAt: todo.updatedAt,
        }));
    },
});

const syncRouter = useExpress({
    shapes: [todosServerShape],
    getUserIdFn: async (req) => req.user.id, // e.g. if the auth library writes the user info to the request
    middlewares: [], // optional, but can be used for authentication tasks
});

app.use("/sync", syncRouter);

How HyperSync Works (with examples)

HyperSync uses a push-then-pull cycle:

Push: queued local changes are sent to POST /push
Pull: server returns remote changes since the last successful pull via GET /pull?lastPull=...

Client Flow Example

User edits a todo in the app
You call pushChange(payload, userId, updatedAt)
HyperSync stores the change in its Dexie queue
Sync engine sends queued changes in batches
Server returns accepted changeIds
Client marks accepted queue items as pushed
Client pulls server-side changes and applies them with upsertRemoteChangeFn

Server Flow Example

POST /push receives { changes: [...] }
Router filters changes so only the authenticated user’s changes are processed
Changes are grouped by shapeKey and passed into each shape’s upsertChangeFn
Response includes acceptedChanges (the changeIds that succeeded)
GET /pull asks each shape for changes between lastPull and syncedAt
Router returns { changes, syncedAt }

Data Model Guidance

For conflict resolution, keep an update timestamp and use tombstones instead of hard deletions:

updatedAt TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
lastSyncAt TIMESTAMP WITH TIME ZONE DEFAULT NOW()

Suggested upsert pattern:

INSERT INTO todos (todoId, userId, done, updatedAt, lastSyncAt)
VALUES ($1, $2, $3, $4, NOW())
ON CONFLICT (todoId)
DO UPDATE SET done = EXCLUDED.done, deleted = EXCLUDED.deleted, updatedAt = EXCLUDED.updatedAt, lastSyncAt = NOW()
WHERE EXCLUDED.updatedAt > todos.updatedAt AND todos.userId = $2

This ensures newer writes win and lastSyncAt records sync time (not logical update time).

API Reference