ZenoDB

Zeno - Offline-First Sync System

A lightweight, offline-first synchronization system that enables seamless data synchronization between client-side IndexedDB and server-side PostgreSQL. Perfect for note-taking applications and other data-heavy web apps that need to work offline.

⚠️ BETA VERSION NOTICE

This project is currently in beta stage and under active development. While it's functional, you may encounter:

• Unexpected behaviors
• Breaking changes between versions
• Incomplete features
• Limited documentation

Use in production with caution. We recommend thorough testing in development/staging environments before deploying to production.

Please report any issues you encounter on our GitHub repository.

Table of Contents

• Features
• Installation
• Quick Start
• How It Works
• Configuration
• Sync Status & Events
• Best Practices
• Schema Migrations & Conflict Resolution
• Examples
• Troubleshooting
• License
• TODO List & Roadmap

Features

• 🔄 Real-time Sync: Instant synchronization between all connected clients
• 📱 Offline Support: Continue working without internet connection
• 🔌 Auto-Reconnect: Automatically reconnects when network is available
• 💾 Persistent Storage: IndexedDB (client) + PostgreSQL (server)
• ⚡ WebSocket Protocol: Fast, real-time updates across clients
• 🔒 Safe Transactions: Ensures data consistency
• 📊 Sync Status: Real-time sync status indicators and offline mode support
• 🗑️ Soft Delete: Recoverable deletions with automatic cleanup

Installation

# Install the package
npm install zeno-db

# Or with yarn
yarn add zeno-db

Quick Start

Server Setup

1. Install dependencies:

npm install zeno-db

2. Create a server.js file:

import { startSyncServer } from 'zeno-db/server';

startSyncServer({
  port: 3000, 
  pg: {
    connectionString: 'your-postgresql-connection-string',
    table: 'notes' // Your table name
  }
});

Client Setup

1. Initialize the database in your app:

import { ZenoDB } from 'zeno-db';

const db = new ZenoDB({
  storage: {
    type: 'indexeddb',
    name: 'my-app-db'
  },
  sync: {
    type: 'websocket',
    url: 'ws://localhost:3000'
  },
  clientId: `client-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`,
  softDelete: {
    enabled: true,
    fieldName: 'deletedAt',
    permanentDeleteAfter: 30 // Delete after 30 days
  }
});

await db.init();

Usage Example (Notes App)

// Create a new note
const noteId = `note-${Date.now()}`;
await db.set(noteId, {
  id: noteId,
  text: 'Your note text',
  createdAt: new Date().toISOString()
});

// Read a note
const note = await db.get(noteId);

// Delete a note
await db.delete(noteId);

// Subscribe to changes
db.subscribe((key, value) => {
  console.log('Note changed:', key, value);
});

Batch Operations Example

You can efficiently get or set multiple records at once using batch operations:

// Batch get
const notes = await db.getMany(['note-1', 'note-2', 'note-3']);

// Batch set
await db.setMany([
  { id: 'note-4', text: 'Fourth note', createdAt: new Date().toISOString() },
  { id: 'note-5', text: 'Fifth note', createdAt: new Date().toISOString() }
]);

Examples

React Example with Sync Status

import { useState, useEffect } from 'react';
import { ZenoDB } from 'zeno-db';

function NotesApp() {
  const [db, setDb] = useState(null);
  const [syncStatus, setSyncStatus] = useState({
    isOnline: false,
    isSyncing: false,
    syncProgress: 0,
    pendingChangesCount: 0
  });

  useEffect(() => {
    const initDB = async () => {
      const zenoDB = new ZenoDB({
        storage: { type: 'indexeddb', name: 'notes-db' },
        sync: { type: 'websocket', url: 'ws://localhost:3000' },
        softDelete: {
          enabled: true,
          fieldName: 'deletedAt',
          permanentDeleteAfter: 30 // Delete after 30 days
        }
      });

      await zenoDB.init();
      setDb(zenoDB);

      // Subscribe to sync events
      zenoDB.subscribe((event) => {
        setSyncStatus(event.status);
      });
    };

    initDB();
  }, []);

  return (
    <div>
      <div className={`status ${syncStatus.isOnline ? 'online' : 'offline'}`}>
        {syncStatus.isOnline ? 'Online' : 'Offline'}
      </div>
      {syncStatus.isSyncing && (
        <div className="progress">
          Syncing... {syncStatus.syncProgress}%
        </div>
      )}
    </div>
  );
}

Vue Example

<template>
  <div>
    <div :class="['status', { online: syncStatus.isOnline }]">
      {{ syncStatus.isOnline ? 'Online' : 'Offline' }}
    </div>
    <div v-if="syncStatus.isSyncing" class="progress">
      Syncing... {{ syncStatus.syncProgress }}%
    </div>
  </div>
</template>

<script>
import { ZenoDB } from 'zeno-db';

export default {
  data() {
    return {
      db: null,
      syncStatus: {
        isOnline: false,
        isSyncing: false,
        syncProgress: 0,
        pendingChangesCount: 0
      }
    };
  },
  async mounted() {
    this.db = new ZenoDB({
      storage: { type: 'indexeddb', name: 'notes-db' },
      sync: { type: 'websocket', url: 'ws://localhost:3000' },
      softDelete: {
        enabled: true,
        fieldName: 'deletedAt',
        permanentDeleteAfter: 30 // Delete after 30 days
      }
    });

    await this.db.init();
    
    this.db.subscribe((event) => {
      this.syncStatus = event.status;
    });
  }
};
</script>

How It Works

Client Side

• Uses IndexedDB for local storage
• Queues changes while offline
• Auto-syncs when connection restores
• Real-time updates via WebSocket
• Sync status indicators and event handling

Server Side

• PostgreSQL database for persistent storage
• WebSocket server for real-time communication
• Handles concurrent updates
• Broadcasts changes to all connected clients

Configuration

Server Config

interface ServerConfig {
  port: number;
  pg: {
    connectionString: string;  // PostgreSQL connection string
    table: string;            // Table name for your data
  };
  softDelete?: {
    enabled: boolean;         // Enable soft delete functionality
    fieldName?: string;       // Field to store deletion timestamp (default: 'deletedAt')
    permanentDeleteAfter?: number; // Days to keep soft-deleted items
  };
}

Client Config

interface ClientConfig {
  storage: {
    type: 'indexeddb';
    name: string;        // IndexedDB database name
  };
  sync: {
    type: 'websocket';
    url: string;        // WebSocket server URL
  };
  clientId: string;     // Unique client identifier
  softDelete?: {
    enabled: boolean;   // Enable soft delete functionality
    fieldName?: string; // Field to store deletion timestamp (default: 'deletedAt')
    permanentDeleteAfter?: number; // Days to keep soft-deleted items
  };
}

Usage Example with Soft Delete

// Initialize with soft delete enabled
const db = new ZenoDB({
  storage: {
    type: 'indexeddb',
    name: 'my-app-db'
  },
  sync: {
    type: 'websocket',
    url: 'ws://localhost:3000'
  },
  clientId: 'client-1',
  softDelete: {
    enabled: true,
    fieldName: 'deletedAt',
    permanentDeleteAfter: 30 // Delete after 30 days
  }
});

// Soft delete an item
await db.delete('note-1');

// Restore a soft-deleted item
await db.restore('note-1');

// Get all items including soft-deleted ones
const allItems = await db.getAll(true);

// Permanently delete an item
await db.delete('note-1', true);

// Purge old soft-deleted items
await db.purgeDeleted();

Sync Status & Events

ZenoDB provides real-time sync status updates and event handling to help you build responsive UIs that reflect the current sync state of your application.

import { ZenoDB, SyncEvent, SyncStatus } from 'zeno-db';

// Initialize your database
const db = new ZenoDB({
  storage: {
    type: 'indexeddb',
    name: 'my-app-db'
  },
  sync: {
    type: 'websocket',
    url: 'ws://localhost:3000'
  },
  softDelete: {
    enabled: true,
    fieldName: 'deletedAt',
    permanentDeleteAfter: 30 // Delete after 30 days
  }
});

// Subscribe to sync events
db.subscribe((event: SyncEvent) => {
  const { type, status } = event;
  
  // status contains:
  // - isOnline: boolean (connection status)
  // - isSyncing: boolean (whether sync is in progress)
  // - syncProgress: number (0-100 percentage)
  // - pendingChangesCount: number (changes waiting to sync)
  
  switch(type) {
    case 'connection_change':
      updateConnectionUI(status.isOnline);
      break;
    case 'sync_started':
      showSyncStarted();
      break;
    case 'sync_progress':
      updateProgressBar(status.syncProgress);
      break;
    case 'sync_completed':
      showSyncComplete();
      break;
  }
});

// Get current status at any time
const currentStatus: SyncStatus = db.getSyncStatus();

Sync Event Types

• connection_change: Fired when connection status changes (online/offline)
• sync_started: Fired when sync operation begins
• sync_progress: Fired during sync with progress updates
• sync_completed: Fired when sync operation completes

Sync Status Properties

• isOnline: Current connection status
• isSyncing: Whether a sync operation is in progress
• syncProgress: Progress percentage (0-100)
• pendingChangesCount: Number of changes waiting to sync

Best Practices

1. Always initialize before use:

await db.init();

2. Handle offline/online transitions gracefully:

window.addEventListener('online', () => {
  console.log('Back online, syncing...');
});

3. Use try-catch for error handling:

try {
  await db.set('key', value);
} catch (error) {
  console.error('Error saving data:', error);
}

4. Monitor sync status:

db.subscribe((event) => {
  const { type, status } = event;
  // Update UI based on sync status.
});

5. Use soft delete for important data:

// Instead of permanent deletion
await db.delete('important-data');

// Later, if needed
await db.restore('important-data');

// Clean up old deleted items periodically
await db.purgeDeleted();

Schema Migrations & Conflict Resolution

Client-side Migration Example

When evolving your data schema, you may need to migrate existing records. For example, if you add a new field to your notes, you can write a migration function that updates all records after initializing the database:

// Example: Migrating notes to add a "tags" field if missing
async function migrateNotes(db) {
  const allNotes = await db.getAll();
  for (const note of allNotes) {
    if (!note.tags) {
      note.tags = [];
      await db.set(note.id, note); // Save migrated note
    }
  }
}

// Run migration after db.init()
await db.init();
await migrateNotes(db);

Tip: Store a schema version in each record or in IndexedDB metadata, and only run migrations when needed.

Application-level Merging Example

For collaborative or complex data structures, you may need to merge changes from multiple sources. Here is a simple example for merging two versions of a collaborative list:

// Example: Merging two versions of a collaborative list
function mergeLists(localList, remoteList) {
  const merged = [...localList];
  for (const item of remoteList) {
    if (!merged.find(i => i.id === item.id)) {
      merged.push(item); // Add new items from remote
    }
    // Optionally, resolve conflicts for items with the same id
  }
  return merged;
}

// Usage in sync event handler
// (Assuming your sync system emits a 'sync_conflict' event)
db.subscribe((event) => {
  if (event.type === 'sync_conflict') {
    const merged = mergeLists(event.local, event.remote);
    db.set(event.key, merged);
  }
});

Tip: For more complex data, consider using libraries like Automerge or Yjs for CRDT-based merging.

PostgreSQL Table Structure

Your PostgreSQL table should have this structure:

CREATE TABLE notes (
  id TEXT PRIMARY KEY,
  data JSONB NOT NULL,
  deleted_at TIMESTAMP WITH TIME ZONE,
  created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

-- Index for soft delete queries
CREATE INDEX idx_notes_deleted_at ON notes(deleted_at);

License

MIT License - see LICENSE file for details.

TODO List & Roadmap

Authentication & Security 🔐

• [ ] User authentication system integration
• [ ] JWT-based authentication
• [ ] Role-based access control (RBAC)
• [ ] API rate limiting
• [ ] Data encryption at rest
• [ ] Session management
• [ ] OAuth2 provider integration (Google, GitHub)

Data Management 📊

• [ ] Data versioning
• [ ] Soft delete functionality
• [ ] Data validation middleware
• [ ] Advanced conflict resolution strategies
• [ ] Bulk import/export functionality

Performance Optimizations ⚡

• [ ] Data compression
• [ ] Caching layer
• [ ] Request batching
• [ ] Lazy loading support
• [ ] Connection pooling
• [ ] Query optimization

Developer Experience 🛠️

• [ ] CLI tool for database management
• [ ] Better error handling and logging
• [ ] Development environment tooling
• [ ] TypeScript type definitions
• [ ] API documentation with Swagger/OpenAPI
• [ ] Integration tests
• [ ] E2E testing suite

Monitoring & Debugging 📈

• [ ] Telemetry integration
• [ ] Performance metrics dashboard
• [ ] Debug logging
• [ ] Error tracking integration
• [ ] Health check endpoints
• [ ] Audit logging

Infrastructure 🏗️

• [ ] Docker containerization
• [ ] CI/CD pipeline setup
• [ ] Automated deployment scripts
• [ ] Database migration tools
• [ ] Backup and restore functionality
• [ ] High availability setup

Troubleshooting

Common Issues

1. Connection Issues

   // Check if WebSocket server is running
   const ws = new WebSocket('ws://localhost:3000');
   ws.onerror = (error) => {
     console.error('WebSocket connection error:', error);
   };

2. Database Initialization Failures

   try {
     await db.init();
   } catch (error) {
     console.error('Database initialization failed:', error);
     // Check if IndexedDB is supported
     if (!window.indexedDB) {
       console.error('IndexedDB is not supported in this browser');
     }
   }

3. Sync Status Not Updating

   • Ensure you're properly subscribing to sync events
   • Check network connectivity
   • Verify WebSocket connection is active

Debugging Tips

1. Enable verbose logging:

   const db = new ZenoDB({
     // ... config
     debug: true
   });

2. Monitor sync events:

   db.subscribe((event) => {
     console.log('Sync event:', event);
   });