@edryslabs/genericprovider
v2.0.3
Published
A clean, minimal, backend-agnostic provider for Yjs that lets you implement any transport mechanism.
Readme
Generic Yjs Provider
A clean, minimal, backend-agnostic provider for Yjs that lets you implement any transport mechanism.
🚀 Live Demos
Try out all the different transport providers in action:
Explore 9 different transport implementations:
- Dummy - In-memory testing with network simulation
- SimplePeer - WebRTC peer-to-peer
- PeerJS - Simplified WebRTC with cloud signaling
- IndexedDB - Local browser persistence
- Gun - Decentralized graph database
- Trystero - Serverless WebRTC (Nostr, BitTorrent, MQTT, IPFS)
- PubNub - Cloud pub/sub with global CDN
- WebSocket - Standard client-server (y-websocket compatible)
- Matrix - Federated communication protocol
Design Philosophy
You implement: How data is transmitted (WebSocket, WebRTC, PubNub, IndexedDB, etc.)
Provider handles: All Yjs synchronization logic (updates, awareness, state vectors)
Quick Start
1. Implement the Transport Interface
import type { Transport, ConnectionConfig } from './transport'
class MyTransport implements Transport {
private connection: any
private messageCallback?: (data: Uint8Array) => void
get isConnected(): boolean {
return !!this.connection
}
async connect(config: ConnectionConfig): Promise<void> {
// Connect to your backend
this.connection = await connectToBackend(config.room)
// Listen for incoming data
this.connection.on('data', (data: Uint8Array) => {
if (this.messageCallback) {
this.messageCallback(data)
}
})
}
disconnect(): void {
this.connection?.close()
this.connection = null
}
send(data: Uint8Array): void {
this.connection?.send(data)
}
onMessage(callback: (data: Uint8Array) => void): () => void {
this.messageCallback = callback
return () => {
this.messageCallback = undefined
}
}
}2. Use the Provider
import * as Y from 'yjs'
import { GenericProvider } from './GenericProvider'
import { MyTransport } from './MyTransport'
// Create Yjs document
const doc = new Y.Doc()
// Create your transport
const transport = new MyTransport()
// Create provider
const provider = new GenericProvider(doc, transport)
// Connect
await provider.connect({
room: 'my-room-123',
url: 'wss://myserver.com'
})
// Use Yjs as normal - everything syncs automatically!
const ytext = doc.getText('content')
ytext.insert(0, 'Hello World')
// Access awareness for presence/cursors
provider.awareness.setLocalStateField('user', {
name: 'Alice',
color: '#ff0000'
})
// Listen to events
provider.on('synced', (synced) => {
console.log('Synced:', synced)
})
provider.on('status', (status) => {
console.log('Status:', status.state)
})
// Cleanup
provider.disconnect()
provider.destroy()API Reference
Transport Interface
The minimal interface your transport must implement:
interface Transport {
// Lifecycle
connect(config: ConnectionConfig): Promise<void>
disconnect(): void
// Data transmission
send(data: Uint8Array): void | Promise<void>
onMessage(callback: (data: Uint8Array) => void): () => void
// Status
readonly isConnected: boolean
}That's it! Just 4 methods + 1 property.
GenericProvider Class
class GenericProvider extends Observable<string> {
// Properties
readonly doc: Y.Doc
readonly transport: Transport
readonly awareness: Awareness
readonly pubsub: PubSubChannel
readonly status: ConnectionStatus
readonly connected: boolean
readonly synced: boolean
// Methods
constructor(doc: Y.Doc, transport: Transport, options?: {
awareness?: Awareness
syncInterval?: number // Auto-sync interval in ms (default: 5000, set 0 to disable)
verifyUpdates?: boolean // Send hash with each update for fast desync detection (default: true)
batchUpdates?: number // Batch/debounce updates in ms (default: 0 = disabled, recommended: 50-200)
})
connect(config: ConnectionConfig): Promise<void>
disconnect(): void
destroy(): void
syncNow(): void // Force immediate sync (useful after network interruptions)
// Events (inherited from Observable)
on(event: 'status', callback: (status: ConnectionStatus) => void): void
on(event: 'synced', callback: (synced: boolean) => void): void
off(event: string, callback: Function): void
emit(event: string, args: any[]): void
}
class PubSubChannel extends Observable<string> {
// Subscribe to a topic
subscribe(topic: string, callback: (message: any) => void): () => void
// Publish to a topic
publish(topic: string, message: any): void
}Events
status: Emitted when connection status changes{ state: 'disconnected' }{ state: 'connecting' }{ state: 'connected' }{ state: 'error', error: Error }
synced: Emitted when initial sync completestruewhen syncedfalsewhen connection lost
Built-in Features
✅ Automatic Document Sync
All changes to the Yjs document are automatically sent through your transport.
✅ Awareness Protocol
Presence information (cursors, users online, etc.) is handled automatically.
✅ State Vector Sync
Efficient synchronization using Yjs state vectors - only missing data is transmitted.
✅ Loop Prevention
Transaction origins prevent infinite update loops.
✅ Reconnection Ready
Can disconnect and reconnect without recreating the provider.
✅ Pub/Sub Channel
Built-in topic-based messaging for real-time communication that doesn't need CRDT properties (chat, notifications, RPC calls, etc).
Pub/Sub Channel
In addition to CRDT synchronization, the provider includes a pub/sub channel for real-time messaging:
// Subscribe to a topic
const unsubscribe = provider.pubsub.subscribe('chat', (message: any) => {
console.log('Chat message:', message)
})
// Publish to a topic
provider.pubsub.publish('chat', {
user: 'Alice',
text: 'Hello!',
timestamp: Date.now()
})
// Subscribe to all topics (wildcard)
provider.pubsub.subscribe('*', (message: any) => {
console.log('All messages:', message)
})
// Unsubscribe when done
unsubscribe()Use Cases for Pub/Sub
- Chat Messages: Ephemeral messages that don't need to be stored in CRDT
- Notifications: User joined, left, status changes
- RPC Calls: Request/response patterns between clients
- Events: Button clicks, interactions, temporary states
- Presence: Real-time updates that complement awareness
Pub/Sub vs Awareness
- Awareness: For user presence data (cursors, names, colors) - maintained by Yjs
- Pub/Sub: For custom real-time messaging - you control the schema
Example: Chat System
// Setup
const doc = new Y.Doc()
const transport = new WebSocketTransport()
const provider = new GenericProvider(doc, transport)
await provider.connect({ room: 'chat-room' })
// Subscribe to chat
provider.pubsub.subscribe('chat', (msg) => {
displayMessage(msg.user, msg.text)
})
// Send message
function sendMessage(text: string) {
provider.pubsub.publish('chat', {
user: getCurrentUser(),
text: text,
timestamp: Date.now()
})
}
// The Yjs doc can still be used for persistent chat history if needed
const yhistory = doc.getArray('chatHistory')Example Implementations
See examples.ts for complete implementations of:
- WebSocketTransport: Real-time WebSocket communication
- PubNubTransport: Pub/sub messaging
- IndexedDBTransport: Local persistence (acts as a "transport")
How It Works
┌─────────────────────────────────────────────┐
│ Your Application │
│ (Uses Y.Doc, Y.Text, Y.Array, etc.) │
└──────────────────┬──────────────────────────┘
│
│ Automatic sync
▼
┌─────────────────────────────────────────────┐
│ GenericProvider │
│ • Listens to doc changes │
│ • Encodes updates with y-protocols │
│ • Manages awareness │
│ • Handles sync protocol │
└──────────────────┬──────────────────────────┘
│
│ send(Uint8Array)
│ onMessage(callback)
▼
┌─────────────────────────────────────────────┐
│ Your Transport │
│ • WebSocket.send() │
│ • PubNub.publish() │
│ • WebRTC.sendData() │
│ • IndexedDB.put() │
│ • etc. │
└─────────────────────────────────────────────┘Advanced Usage
Custom Awareness
import { Awareness } from 'y-protocols/awareness'
const awareness = new Awareness(doc)
const provider = new GenericProvider(doc, transport, { awareness })Connection Configuration
Pass any config your transport needs:
await provider.connect({
room: 'my-room',
url: 'wss://server.com',
apiKey: 'secret-key',
timeout: 5000,
// ... any custom fields
})Async Transports
Your send() method can return a Promise:
async send(data: Uint8Array): Promise<void> {
await this.backend.publish(data)
}Error Handling
provider.on('status', (status) => {
if (status.state === 'error') {
console.error('Connection error:', status.error)
// Implement retry logic
}
})Force Re-Sync
After network interruptions or when you suspect clients are out of sync, trigger an immediate sync:
// Manual re-sync
provider.syncNow()
// Re-sync when network comes back online
window.addEventListener('online', () => {
console.log('Network restored')
provider.syncNow()
})
// Re-sync after connection issues
provider.on('status', (status) => {
if (status.state === 'connected') {
// Give transport a moment to stabilize
setTimeout(() => provider.syncNow(), 100)
}
})What syncNow() does:
- Sends the full local document state to all peers (ensures offline changes are transmitted)
- Sends a sync request (SyncStep1) to get missing updates from others
- Broadcasts the current awareness state
This is especially important after a client has been offline and made local changes, as it ensures those changes are sent to other clients when reconnecting.
Automatic Periodic Sync
By default, the provider automatically calls syncNow() every 5 seconds to recover from packet loss or intermittent network issues:
// Default behavior - auto-sync every 5 seconds
const provider = new GenericProvider(doc, transport)
// Faster recovery for unreliable networks
const provider = new GenericProvider(doc, transport, {
syncInterval: 2000, // Retry every 2 seconds
})
// Disable automatic sync (not recommended)
const provider = new GenericProvider(doc, transport, {
syncInterval: 0, // Manual sync only
})Why automatic sync is important:
- Yjs requires multi-step handshakes (SyncStep1 → SyncStep2 → Updates)
- If any message is lost due to packet loss, sync stalls
- Periodic retries ensure eventual consistency even on unreliable networks
- Performance impact is minimal (only sends if there are changes)
For most production scenarios, the default 5-second interval provides good resilience without excessive traffic. For testing with simulated packet loss, use a shorter interval (e.g., 2 seconds).
Update Batching (Debouncing)
By default, every document change triggers an immediate network transmission. For performance optimization, you can enable update batching (also called debouncing):
// Default behavior - send updates immediately
const provider = new GenericProvider(doc, transport)
// Batch updates: collect changes and send after 100ms of inactivity
const provider = new GenericProvider(doc, transport, {
batchUpdates: 100, // milliseconds
})How it works:
- User types "hello" (5 keystrokes)
- Instead of 5 separate network messages, wait 100ms after last keystroke
- Send one combined update containing all changes
- Result: 1 message instead of 5
Benefits:
- ✅ Reduces network traffic by ~70-90% during rapid typing
- ✅ Lower bandwidth usage - fewer messages, less overhead per message
- ✅ Better performance on slow or metered networks
- ✅ Still feels instant - 50-200ms delay is imperceptible to users
- ✅ Yjs automatically merges sequential updates efficiently
Recommended values:
0(default): Immediate transmission, no batching - best for slow networks or real-time cursors50-100ms: Good balance for most collaborative editing (typing, formatting)200ms: Maximum delay while still feeling responsive500ms+: Not recommended (users notice lag)
When to use:
- ✅ Collaborative text editing (typing)
- ✅ Drawing/whiteboard applications (mouse movements)
- ✅ Metered or slow networks
- ✅ Mobile/battery-conscious applications
When NOT to use:
- ❌ Real-time cursor tracking (needs immediate updates)
- ❌ Turn-based interactions (each change is discrete)
- ❌ Ultra-low latency requirements (<50ms)
Example - Optimized collaborative editor:
const provider = new GenericProvider(doc, transport, {
batchUpdates: 100, // Batch updates for efficiency
verifyUpdates: true, // Hash verification for reliability
syncInterval: 5000, // Periodic backup sync
})Fast Desync Detection
By default, the provider includes a document hash with every update for immediate detection of sync problems:
// Default behavior - hash verification enabled
const provider = new GenericProvider(doc, transport, {
verifyUpdates: true, // Default: true
})How it works:
- When sending an update, include hash of document state after the update
- Receiver applies update, computes own hash, compares
- If hashes don't match → immediately trigger re-sync (no waiting!)
- Much faster than waiting for periodic sync interval
Exponential backoff:
- First mismatch: retry after 10ms
- Second mismatch: retry after 50ms
- Third mismatch: retry after 250ms
- Fourth mismatch: retry after 1.25s
- Further mismatches: retry after 6.25s, capped at 10s
- If stable for 10 seconds, counter resets
- Prevents "sync storm" if clients persistently disagree
Performance:
- Hash computation is fast (~1ms for typical documents)
- Only happens when updates occur (not constantly)
- Adds minimal bandwidth (~4 bytes per update message)
When to disable:
// Disable for lowest possible bandwidth usage
const provider = new GenericProvider(doc, transport, {
verifyUpdates: false, // Rely only on periodic sync
})Only disable if you have a very reliable transport (no packet loss) or bandwidth is extremely constrained.
Comparison:
- Without verification: Lost message → wait up to 5 seconds → retry
- With verification (default): Lost message → next update detects mismatch → immediate retry
This makes sync much more responsive on unreliable networks!
Sequence Numbers for Ordering
When verifyUpdates is enabled, the provider automatically includes sequence numbers with each update for total ordering guarantees:
// Automatic with verifyUpdates enabled (default)
const provider = new GenericProvider(doc, transport, {
verifyUpdates: true, // Includes sequence numbers
})How it works:
- Each client maintains its own sequence counter (increments with each update sent)
- Updates include:
[seqNum, clientID, update, hash] - Receiver tracks last seen sequence number for each remote client
- Detects and handles:
- Duplicates: seqNum ≤ last seen → skip (already processed)
- Out-of-order: seqNum arrives before earlier ones → log warning
- Gaps: seqNum jumps (e.g., 5 → 8) → log packet loss, Yjs handles recovery
Benefits:
- Duplicate detection: Prevents processing the same update twice
- Order verification: Confirms updates arrive in causal order
- Gap detection: Identifies packet loss immediately (diagnostic)
- Debugging: Easier to trace message flow and diagnose issues
Example console output:
[GenericProvider] Sequence gap detected from client 423891: expected 12, got 15 (gap of 3 messages)
[GenericProvider] Duplicate update detected from client 892301: seqNum 8 <= lastSeen 10Performance:
- Adds ~2-8 bytes per update (variable-length encoding)
- O(1) duplicate detection (hash map lookup)
- Negligible CPU overhead
Why this matters:
- In unreliable networks (high packet loss), updates can arrive multiple times via retries
- Without sequence numbers, duplicate updates would cause unnecessary reprocessing
- With sequence numbers, duplicates are instantly detected and skipped
- Provides diagnostic information about network quality
Built-in Rate Limiting
The provider includes automatic rate limiting on sync requests to prevent spam and protect against malicious or buggy clients:
// Default behavior - built-in rate limiting
const provider = new GenericProvider(doc, transport)
// Max 20 sync requests per 10-second windowHow it works:
- Tracks all sync requests (SyncStep1) sent within a 10-second sliding window
- If more than 20 requests are attempted in that window, additional requests are dropped
- Warning logged to console when rate limit is hit
- Prevents "sync storm" scenarios where buggy code spams sync requests
Protection against:
- Buggy code with infinite retry loops
- Malicious clients attempting to flood the network
- Accidental DoS from misconfigured exponential backoff
- Network issues causing excessive retry attempts
Why this is safe:
- 20 requests per 10 seconds = 2 per second average (very generous)
- Normal operation uses far fewer requests (typically < 5 per 10 seconds)
- Combined with exponential backoff, prevents most abuse scenarios
- Doesn't affect normal document updates (only sync requests are limited)
The rate limiting is transparent and automatic - no configuration needed!
Migration from Existing Providers
If you have existing providers (like your PubNub/P2PT implementations), wrap them:
class PubNubTransportAdapter implements Transport {
constructor(private oldProvider: OldPubNubSync) {}
async connect(config) {
return this.oldProvider.connect(config)
}
send(data: Uint8Array) {
this.oldProvider.broadcast(true, data)
}
onMessage(callback) {
this.oldProvider.onReceive = callback
return () => { this.oldProvider.onReceive = undefined }
}
// ... etc
}Optional Monitoring Utilities
The GenericProvider includes built-in hash verification (enabled by default with verifyUpdates: true). This is sufficient for most use cases.
However, for advanced diagnostics and monitoring, the SyncHealthMonitor utility is available:
import { SyncHealthMonitor } from './sync-monitor'
const monitor = new SyncHealthMonitor(provider, {
checkInterval: 10000, // Check every 10 seconds
onDesync: (details) => {
// Send to error tracking, alerting system, etc.
console.error('Desync detected across peers!', details)
}
})
monitor.start()When to use SyncHealthMonitor:
- ✅ You need to monitor ALL peers simultaneously via pub/sub broadcasts
- ✅ You want periodic health checks for diagnostics/alerting
- ✅ You need separate monitoring infrastructure from core sync
- ✅ Production monitoring and error tracking
When to use built-in verification (default):
- ✅ Fast desync detection (immediate, on every update)
- ✅ Zero configuration (automatic)
- ✅ Exponential backoff, rate limiting, sequence numbers
- ✅ Recommended for most applications
Key difference:
- Built-in: Point-to-point verification (sender ↔ receiver) on each update
- SyncHealthMonitor: Periodic broadcasts across all peers for diagnostics
Most applications should use the built-in verification. Use SyncHealthMonitor only if you need additional monitoring for alerting/diagnostics.
License
Same as LiaScript (BSD-3-Clause)
