@rozek/sds-core

Backend-agnostic shared types, base classes, and data-model documentation for the shareable-data-store (SDS) family.

This package contains everything that is common to all SDS CRDT backends: the base entry classes, the error class, the change-set types, and the provider interfaces. It does not contain any CRDT implementation — use one of the backend packages for that:

| Backend | Package | | --- | --- | | json-joy | @rozek/sds-core-jj | | Y.js | @rozek/sds-core-yjs | | Loro | @rozek/sds-core-loro |

Prerequisites

| requirement | details | | --- | --- | | Node.js 22+ | required when using this package in a Node.js project or build toolchain. Download from nodejs.org. | | Modern browser | required when using this package in a web application. Any evergreen browser is supported: Chrome 90+, Firefox 90+, Safari 15+, Edge 90+. |

This package is isomorphic — it contains no native code, no WebAssembly, and no browser-specific APIs.

Installation

pnpm add @rozek/sds-core

Application code typically depends on a backend package directly and does not need to add @rozek/sds-core explicitly — it is a peer dependency of all backend packages.

Concepts

Store

SDS_DataStore is a CRDT-based tree of items. All mutations are tracked as compact binary patches that can be exchanged with remote peers and applied in any order without conflicts.

Entries: items and links

There are two kinds of entries in the tree:

• SDS_Item — a node that can carry a value (string or binary) and contain inner entries
• SDS_Link — a pointer entry that references another item; useful for aliases and cross-references

Every store starts with three well-known, non-deletable items:

| Item | Role | | --- | --- | | RootItem | Root of the user-visible tree | | TrashItem | Deleted entries are moved here | | LostAndFoundItem | Orphaned entries (outer item purged by a remote peer) are rescued here |

Values

An item's value is stored in one of four modes, selected automatically:

| Kind | When used | Storage | | --- | --- | --- | | literal | short strings ≤ LiteralSizeLimit | inline in the CRDT | | literal-reference | strings beyond the literal size limit | hash reference + external blob | | binary | small Uint8Array ≤ 2 KB | inline in the CRDT | | binary-reference | larger Uint8Array | hash reference + external blob |

Literal values support collaborative character-level editing via changeValue().

ChangeSet

Every mutation (or batch of mutations in a transact() block) produces a SDS_ChangeSet delivered to all registered handlers. The ChangeSet maps each affected entry Id to the set of property keys that changed ('Label', 'Value', 'outerItem', 'innerEntryList', 'Info.<key>', …).

API reference

SDS_DataStore

Construction

SDS_DataStore.fromScratch (Options?: SDS_DataStoreOptions):SDS_DataStore
SDS_DataStore.fromBinary (Data:Uint8Array, Options?:SDS_DataStoreOptions):SDS_DataStore
SDS_DataStore.fromJSON (Data:unknown, Options?:SDS_DataStoreOptions):SDS_DataStore

fromScratch creates a new, empty store pre-populated with RootItem, TrashItem, and LostAndFoundItem. fromBinary restores a store from a gzip-compressed snapshot (as produced by store.asBinary()). fromJSON restores a store from a plain JSON object or a JSON string (as produced by store.asJSON()).

interface SDS_DataStoreOptions {
  LiteralSizeLimit?:number      // max inline string length in UTF-16 code units (default 131_072)
  TrashTTLms?:number            // ms after which a trashed entry is eligible for auto-purge (default: 30 days = 2_592_000_000; set to 0 to disable)
  TrashCheckIntervalMs?: number // how often the auto-purge timer fires (default: min(TrashTTLms/4, 3_600_000))
}

Unless TrashTTLms is explicitly set to 0, SDS_DataStore starts an internal setInterval that calls purgeExpiredTrashEntries() at the configured interval. Call dispose() to stop the timer when the store is no longer needed.

Well-known items

readonly RootItem:SDS_Item
readonly TrashItem:SDS_Item
readonly LostAndFoundItem:SDS_Item

Creating entries

newItemAt (
  MIMEType:string | undefined, // MIME type (default 'text/plain')
  outerItem:SDS_Item,
  InsertionIndex?:number       // position within outerItem.innerEntryList (appends if omitted)
):SDS_Item

newLinkAt (
  Target:SDS_Item,
  outerItem:SDS_Item,
  InsertionIndex?:number
):SDS_Link

Looking up entries

EntryWithId (EntryId:string):SDS_Entry | undefined

Importing serialised entries

deserializeItemInto (Data:unknown, outerItem:SDS_Item, InsertionIndex?:number):SDS_Item
deserializeLinkInto (Data:unknown, outerItem:SDS_Item, InsertionIndex?:number):SDS_Link

newEntryFromBinaryAt (Data:Uint8Array, outerItem:SDS_Item, InsertionIndex?:number):SDS_Entry
newEntryFromJSONat   (Data:unknown,    outerItem:SDS_Item, InsertionIndex?:number):SDS_Entry

newEntryFromBinaryAt accepts a gzip-compressed binary snapshot (as produced by entry.asBinary()); newEntryFromJSONat accepts a plain JSON object (as produced by entry.asJSON()). Both methods reconstruct the full entry subtree (including all inner entries) and insert it into outerItem at the given position. Unlike deserializeItemInto/deserializeLinkInto, they automatically detect whether the root is an item or a link.

Moving entries

EntryMayBeMovedTo (Entry:SDS_Entry, outerItem:SDS_Item, InsertionIndex?:number):boolean
moveEntryTo (Entry:SDS_Entry, outerItem:SDS_Item, InsertionIndex?:number):void

Throws SDS_Error('move-would-cycle') if the move would create a cycle in the tree.

Rebalancing order keys

rebalanceInnerEntriesOf (item:SDS_Item):void

Reassigns fresh, evenly-spaced fractional-indexing keys to all direct inner entries of item. This is called automatically whenever a new OrderKey would exceed 200 characters, so manual calls are only needed as a proactive measure before predictably degenerate bulk operations (see Order keys below).

Deleting and purging entries

EntryMayBeDeleted (Entry:SDS_Entry):boolean
deleteEntry (Entry:SDS_Entry):void  // moves to TrashItem; records _trashedAt in Info
purgeEntry (Entry:SDS_Entry):void   // permanently removes from Trash

// purges all direct TrashItem children whose _trashedAt exceeds TTLms;
// returns the count of entries actually removed
purgeExpiredTrashEntries (TTLms?: number):number

dispose ():void  // stops the auto-purge timer (if TrashTTLms was configured)

deleteEntry records a _trashedAt timestamp (milliseconds since epoch) in the entry's Info object. This field is stored in the CRDT and is therefore synced to remote peers.

purgeEntry throws SDS_Error('purge-protected') when the entry (or any descendant) is the target of a link reachable from RootItem; such entries remain in TrashItem.

purgeExpiredTrashEntries skips entries that have no _trashedAt field (e.g. moved to Trash via moveEntryTo) and silently skips protected entries rather than throwing.

Transactions

transact (Callback:() => void):void

Groups multiple mutations into a single CRDT operation and emits exactly one ChangeSet event. Transactions may be nested, but inner ones have no extra effect.

No rollback: If the callback throws, any CRDT mutations already applied within the transaction are not rolled back. Guard against partial writes with explicit precondition checks before mutating.

Events

onChangeInvoke (
  Handler:(Origin:'internal' | 'external', ChangeSet:SDS_ChangeSet) => void
):() => void  // returns an unsubscribe function

'internal' — the mutation originated locally; 'external' — it came from applyRemotePatch.

Sync

get currentCursor ():SDS_SyncCursor              // cursor position after the latest local mutation
exportPatch (since?: SDS_SyncCursor):Uint8Array  // binary CRDT patch since a given cursor
applyRemotePatch (encodedPatch:Uint8Array):void
recoverOrphans ():void  // rescue entries whose outer item no longer exists

Backend compatibility: SDS_SyncCursor values and exportPatch payloads are opaque binary blobs whose format is specific to the backend (YJS, LORO, or JJ). Cursors and patches produced by one backend must never be passed to a store using a different backend.

Serialisation

asBinary ():Uint8Array  // gzip-compressed full snapshot
asJSON ():unknown       // full snapshot as a plain JSON object (binary values are base64-encoded)

SDS_Entry (base class for SDS_Item and SDS_Link)

Identity

readonly Id:string

get isRootItem:boolean
get isTrashItem:boolean
get isLostAndFoundItem:boolean
get isItem:boolean
get isLink:boolean

Hierarchy

get outerItem:SDS_Item | undefined  // direct container
get outerItemId:string | undefined
get outerItemChain:SDS_Item[]       // ancestor chain, innermost first
get outerItemIds:string[]

Label and metadata

get Label:string
set Label(Value:string):void

get Info:Record<string,unknown>  // live proxy; assignments are CRDT mutations

Convenience methods

mayBeMovedTo (outerItem:SDS_Item, InsertionIndex?:number):boolean
moveTo (outerItem:SDS_Item, InsertionIndex?:number):void

get mayBeDeleted:boolean
delete ():void
purge ():void

asBinary ():Uint8Array  // serialises this entry subtree as a gzip-compressed binary snapshot
asJSON ():unknown        // serialises this entry subtree as a plain JSON object

SDS_Item (extends SDS_Entry)

MIME type

get Type:string
set Type(Value:string):void

Value

get ValueKind:'none' | 'literal' | 'literal-reference' | 'binary' | 'binary-reference' | 'pending'
get isLiteral:boolean
get isBinary: boolean

readValue ():Promise<string | Uint8Array | undefined>
writeValue (Value:string | Uint8Array | undefined):void

// character-level collaborative edit on a 'literal' value
changeValue (fromIndex:number, toIndex:number, Replacement:string):void

readValue resolves immediately when the value is stored inline (ValueKind is 'literal' or 'binary'). For reference kinds ('literal-reference', 'binary-reference') it fetches the payload from the configured ValueStore and resolves asynchronously. There is no synchronous alternative. Returns undefined when ValueKind is 'none' or 'pending'.

Throws SDS_Error('change-value-not-literal') if ValueKind !== 'literal'.

Inner entries

get innerEntryList:SDS_Entry[]

SDS_Link (extends SDS_Entry)

A link is a pointer entry that references another item. It is useful for aliases, cross-references, and many-to-one relationships — for example, tagging an item that lives in one part of the tree from multiple other locations.

get Target:SDS_Item  // fixed at creation time; cannot be changed after creation

A link's target is immutable — once created, Target never changes. Deleting a link does not affect the target item. If a link reachable from RootItem points at an item that is in TrashItem, purgeEntry on that item will throw SDS_Error('purge-protected') until the link itself is deleted first.

Exported types

SDS_Error

class SDS_Error extends Error {
  readonly code:string
  constructor (Code:string, Message:string)
}

Thrown by all SDS packages on invalid arguments or invalid state.

Common error codes: 'invalid-argument', 'move-would-cycle', 'delete-not-permitted', 'purge-not-in-trash', 'purge-protected', 'change-value-not-literal', 'not-implemented'.

SDS_ChangeSet / SDS_EntryChangeSet

type SDS_EntryChangeSet = Set<string>
type SDS_ChangeSet = Record<string,SDS_EntryChangeSet>

Delivered to onChangeInvoke handlers after every mutation. The ChangeSet maps each affected entry Id to the set of property keys that changed ('Label', 'Value', 'outerItem', 'innerEntryList', 'Info.<key>', …).

SDS_SyncCursor / SDS_PatchSeqNumber

type SDS_SyncCursor = Uint8Array // opaque; format is backend-specific
type SDS_PatchSeqNumber = number // maintained by SDS_SyncEngine

Provider interfaces

These interfaces are implemented by the infrastructure packages and consumed by SDS_SyncEngine.

SDS_PersistenceProvider

interface SDS_PersistenceProvider {
  loadSnapshot ():Promise<Uint8Array | undefined>
  saveSnapshot (Data:Uint8Array):Promise<void>

  loadPatchesSince (SeqNumber:SDS_PatchSeqNumber):Promise<Uint8Array[]>
  appendPatch (Patch:Uint8Array, SeqNumber:SDS_PatchSeqNumber):Promise<void>
  prunePatches (beforeSeqNumber:SDS_PatchSeqNumber):Promise<void>

  loadValue (ValueHash:string):Promise<Uint8Array | undefined>
  saveValue (ValueHash:string, Data:Uint8Array):Promise<void>
  releaseValue (ValueHash:string):Promise<void>

  close ():Promise<void>
}

Implemented by @rozek/sds-persistence-browser (IndexedDB) and @rozek/sds-persistence-node (SQLite).

SDS_NetworkProvider

type SDS_ConnectionState = 'disconnected' | 'connecting' | 'connected' | 'reconnecting'

interface SDS_ConnectionOptions {
  Token:string             // JWT
  reconnectDelayMs?:number // auto-reconnect backoff in ms (default 2000)
}

interface SDS_NetworkProvider {
  readonly StoreId:string
  readonly ConnectionState:SDS_ConnectionState

  connect (URL:string, Options:SDS_ConnectionOptions):Promise<void>
  disconnect ():void

  sendPatch (Patch:Uint8Array):void
  sendValue (ValueHash:string, Data:Uint8Array):void
  requestValue (ValueHash:string):void

  onPatch (Callback:(Patch:Uint8Array) => void):() => void
  onValue (Callback:(ValueHash:string, Value:Uint8Array) => void):() => void
  onConnectionChange (Callback:(State:SDS_ConnectionState) => void):() => void
}

Implemented by @rozek/sds-network-websocket and @rozek/sds-network-webrtc.

SDS_PresenceProvider

interface SDS_LocalPresenceState {
  PeerId?:string   // injected by the engine; not set by the user
  UserName?:string
  UserColor?:string
  UserFocus?: {
    EntryId:string
    Property:'Value' | 'Label' | 'Info'
    Cursor?:{ from:number; to:number }
  }
  custom?:unknown  // arbitrary JSON-serialisable application data
}

interface SDS_RemotePresenceState extends SDS_LocalPresenceState {
  PeerId:string   // always present for remote peers
  lastSeen:number // Date.now() timestamp of last received update
}

interface SDS_PresenceProvider {
  sendLocalState (State:SDS_LocalPresenceState):void
  onRemoteState (
    Callback:(PeerId:string, State:SDS_RemotePresenceState | undefined) => void
  ):() => void
  readonly PeerSet:ReadonlyMap<string,SDS_RemotePresenceState>
}

Usually implemented by the same class as SDS_NetworkProvider (@rozek/sds-network-websocket and @rozek/sds-network-webrtc both implement both interfaces).

Validation limits

All three backends enforce the same hard limits on user-supplied strings and metadata. Exceeding any limit throws SDS_Error('invalid-argument') with a descriptive message that includes the actual length and the limit.

| Field | Constant | Default | Checked in | |---|---|---|---| | Label | maxLabelLength | 1 024 chars | entry.Label = … | | MIMEType | maxMIMETypeLength | 256 chars | item.Type = …, newItemAt(…, mimeType) | | Info key | maxInfoKeyLength | 1 024 chars | entry.Info['key'] = … | | Info value | maxInfoValueSize | 262 144 bytes (UTF-8 JSON) | entry.Info['key'] = … |

Info values must also be JSON-serialisable; functions, symbols, and circular references are rejected.

The maxInfoValueSize limit is measured on the value's JSON.stringify output encoded as UTF-8, not on the raw value itself. JSON.stringify always adds overhead — strings gain two surrounding quote characters, objects pay for ", :, ,, {, and } per key, and non-ASCII characters may expand to up to four UTF-8 bytes or six characters via \uXXXX escaping. For a plain ASCII string the effective character limit is therefore 262 142, not 262 144.

Info is designed for compact metadata such as tag lists, content hashes, status flags, or short annotations — not for storing large payloads. Large binary or text data belongs in the entry's Value field, which has its own blob storage and chunked transfer mechanism.

All four constants are exported from @rozek/sds-core so that application code can display the limits to users without hard-coding them.

Order keys

Entries within a container are sorted by a string called OrderKey, generated by the fractional-indexing algorithm. These keys grow in length each time a new entry is inserted between two neighbours that already have adjacent keys. Under normal usage (appending, prepending, or occasional reordering) keys remain short. With repeated insertions at the same gap the keys can grow arbitrarily long.

Automatic rebalancing

All three backends automatically trigger rebalanceInnerEntriesOf inside the same transaction whenever a newly generated OrderKey would exceed 200 characters. Under normal usage this threshold is never reached, so the auto-trigger adds no overhead. In degenerate patterns (e.g. always inserting at position 0 in a large container), the rebalancing fires once and then the keys are short again for the next batch of operations.

Manual rebalancing

Call rebalanceInnerEntriesOf explicitly as a proactive measure before bulk operations where you know the same gap will be used many times in a row:

Store.rebalanceInnerEntriesOf(targetItem)
for (const record of largeImport) {
  Store.newItemAt('application/json', targetItem, 0)  // always inserts at position 0
}

CRDT safety

rebalanceInnerEntriesOf is a normal CRDT mutation — it updates the OrderKey field of every direct inner entry in a single transaction that syncs to all peers. There is one caveat: if a remote peer inserts an entry at a gap that is being rebalanced concurrently, the new entry may land in a slightly different relative position than intended. No data is lost and all peers converge to the same order; the only effect is a potential one-time positional surprise during the brief window of concurrent activity.

License

MIT License © Andreas Rozek