LacertaDB Documentation

Table of Contents

1. Introduction
2. Architecture Overview
3. Installation
4. Core Concepts
5. Getting Started
6. API Reference
7. Advanced Features
8. Performance Optimization
9. Security Considerations
10. Best Practices
11. Troubleshooting
12. Migration Guide

Introduction

LacertaDB represents a sophisticated abstraction layer over IndexedDB, architecting a comprehensive document-oriented database system directly within the browser environment. This library synthesizes multiple advanced capabilities including cryptographic operations, data compression, attachment management through the Origin Private File System (OPFS), and intelligent storage lifecycle management.

Key Differentiators

• Cryptographic Security: Military-grade AES-GCM encryption with PBKDF2 key derivation
• Compression Architecture: Native browser compression streams for optimal storage utilization
• Attachment Management: Seamless binary data handling via OPFS
• Automatic Space Management: Intelligent garbage collection with configurable retention policies
• Metadata Synchronization: Dual-layer metadata persistence for rapid access patterns
• Transaction Atomicity: ACID-compliant operations with rollback capabilities

Architectural Philosophy

LacertaDB embodies a multi-tiered storage strategy, leveraging IndexedDB for structured document storage while utilizing localStorage for metadata persistence and OPFS for binary attachment management. This tripartite architecture ensures optimal performance characteristics across diverse usage patterns.

Architecture Overview

Storage Layer Topology

┌─────────────────────────────────────────┐
│         Application Layer               │
├─────────────────────────────────────────┤
│            LacertaDB API                │
├─────────────────────────────────────────┤
│   ┌─────────────┬──────────┬──────────┐ │
│   │  Database   │Collection│ Document │ │
│   │  Manager    │ Handler  │ Processor│ │
│   └─────────────┴──────────┴──────────┘ │
├─────────────────────────────────────────┤
│   ┌─────────────┬──────────┬──────────┐ │
│   │  IndexedDB  │LocalStore│   OPFS   │ │
│   │  (Documents)│(Metadata)│(Attachm.)│ │
│   └─────────────┴──────────┴──────────┘ │
└─────────────────────────────────────────┘

Component Interactions

The system orchestrates through several interconnected subsystems:

1. Database Layer: Manages database lifecycle and collection orchestration
2. Collection Layer: Handles document CRUD operations and indexing
3. Document Layer: Processes packing, encryption, and compression
4. Metadata Layer: Maintains synchronized metadata across storage boundaries
5. Utility Layer: Provides cryptographic, compression, and file system operations

Installation

Module Import

// ES6 Module Import
import { Database, Document, Collection } from './lacertadb.js';
import JOYSON from 'joyson';  // Required dependency

Prerequisites

• Modern browser with IndexedDB support
• OPFS API availability (Chrome 86+, Edge 86+, Safari 15.2+)
• Native Compression Streams API
• Web Crypto API

Browser Compatibility Matrix

| Feature | Chrome | Firefox | Safari | Edge | |---------|--------|---------|--------|------| | IndexedDB | ✓ | ✓ | ✓ | ✓ | | OPFS | 86+ | 111+ | 15.2+ | 86+ | | Compression Streams | 80+ | 113+ | 16.4+ | 80+ | | Web Crypto | 37+ | 34+ | 7+ | 12+ |

Core Concepts

Document Model

Documents represent the atomic unit of storage, encapsulating both structured data and metadata:

{
  _id: "uuid-string",           // Unique identifier
  _created: 1234567890,          // Creation timestamp
  _modified: 1234567890,         // Modification timestamp
  _permanent: false,             // Deletion protection flag
  _encrypted: false,             // Encryption status
  _compressed: false,            // Compression status
  attachments: [],               // Binary attachment references
  data: {                        // User-defined payload
    // Application data
  }
}

Collection Paradigm

Collections function as logical containers for documents, providing:

• Document isolation and namespacing
• Independent size limits and retention policies
• Transaction boundaries
• Index management capabilities

Metadata Architecture

The metadata subsystem maintains dual-layer persistence:

• Database Metadata: Global statistics and collection registry
• Collection Metadata: Document inventory and size tracking

Getting Started

Basic Database Operations

// Initialize database with configuration
const db = new Database('myApplication', {
  sizeLimitKB: 50000,        // 50MB limit
  bufferLimitKB: -10000,     // 10MB buffer before cleanup
  freeSpaceEvery: 60000      // Cleanup check every 60 seconds
});

// Initialize database connection
await db.init();

// Create a collection
const users = await db.createCollection('users');

// Add a document
const isNew = await users.addDocument({
  data: {
    name: 'Alice Johnson',
    email: '[email protected]',
    preferences: {
      theme: 'dark',
      notifications: true
    }
  },
  _permanent: true  // Protect from automatic cleanup
});

// Retrieve document
const doc = await users.getDocument('document-id');
console.log(doc.data);

// Update document
await users.addDocument({
  _id: doc._id,
  data: {
    ...doc.data,
    lastLogin: Date.now()
  }
});

// Delete document
await users.deleteDocument(doc._id, true); // Force delete even if permanent

Working with Attachments

// Create document with attachments
const fileInput = document.getElementById('fileInput');
const files = Array.from(fileInput.files);

await users.addDocument({
  data: {
    title: 'Project Documentation',
    description: 'Q4 Analysis Report'
  },
  attachments: files.map(file => ({ data: file }))
});

// Retrieve document with attachments
const docWithFiles = await users.getDocument(
  'doc-id', 
  null,     // No encryption key
  true      // Include attachment data
);

// Access attachment blobs
for (const attachment of docWithFiles.attachments) {
  const blob = attachment.data;
  const url = URL.createObjectURL(blob);
  // Use the blob URL
}

Encryption Implementation

// Store encrypted document
const secretKey = 'user-provided-password-123';

await users.addDocument({
  data: {
    ssn: '123-45-6789',
    creditCard: '4111-1111-1111-1111',
    medicalRecords: { /* sensitive data */ }
  },
  _encrypted: true
}, secretKey);

// Retrieve and decrypt
const encryptedDoc = await users.getDocument('doc-id', secretKey);
// Returns false if wrong key provided

API Reference

Database Class

Constructor

new Database(dbName: string, settings?: object)

Parameters:

• dbName: Unique database identifier
• settings: Configuration object
  • sizeLimitKB: Maximum size in kilobytes (default: Infinity)
  • bufferLimitKB: Buffer before cleanup triggers (default: -20% of sizeLimitKB)
  • freeSpaceEvery: Cleanup interval in milliseconds (default: 10000)

Methods

async init()

Initializes database connection and loads existing collections.

const db = new Database('myApp');
await db.init();

async createCollection(collectionName: string)

Creates a new collection within the database.

Returns: Collection instance

const products = await db.createCollection('products');

async deleteCollection(collectionName: string)

Removes a collection and all associated documents.

await db.deleteCollection('deprecated_data');

async getCollection(collectionName: string)

Retrieves existing collection instance.

Returns: Collection instance

const orders = await db.getCollection('orders');

async close()

Properly closes database connections and cleans up resources.

await db.close();

async deleteDatabase()

Completely removes database and all associated data.

await db.deleteDatabase();

Properties

• name: Database identifier
• totalSizeKB: Total storage consumption in KB
• totalLength: Total document count across collections
• modifiedAt: Last modification timestamp
• collections: Map of active collection instances

Collection Class

Methods

async addDocument(documentData: object, encryptionKey?: string)

Inserts or updates a document in the collection.

Parameters:

• documentData: Document object with data and metadata
• encryptionKey: Optional encryption password

Returns: boolean - true if newly created, false if updated

const isNew = await collection.addDocument({
  data: { name: 'Item 1' },
  _compressed: true,
  _permanent: false
});

async getDocument(docId: string, encryptionKey?: string, includeAttachments?: boolean)

Retrieves a single document by ID.

Parameters:

• docId: Document identifier
• encryptionKey: Decryption key if encrypted
• includeAttachments: Load attachment data

Returns: Document object or false if not found

const doc = await collection.getDocument('abc-123', 'password', true);

async getDocuments(ids: string[], encryptionKey?: string, withAttachments?: boolean)

Batch retrieval of multiple documents.

Returns: Array of document objects

const docs = await collection.getDocuments(['id1', 'id2', 'id3']);

async deleteDocument(docId: string, force?: boolean)

Removes a document from the collection.

Parameters:

• docId: Document identifier
• force: Override permanent flag

Returns: boolean - success status

await collection.deleteDocument('obsolete-doc', true);

async query(filter?: object, options?: object)

Performs filtered queries on the collection.

Parameters:

• filter: Key-value pairs for matching
• options: Query configuration
  • limit: Maximum results
  • offset: Skip count
  • orderBy: Sort direction ('asc' or 'desc')
  • index: Index name to use
  • encryptionKey: Decryption key

Returns: Array of matching documents

const results = await collection.query(
  { 'status': 'active' },
  { 
    limit: 50, 
    offset: 100,
    orderBy: 'desc'
  }
);

async freeSpace(size: number)

Manually triggers space reclamation.

Parameters:

• size: Positive for target size, negative for amount to free

Returns: Amount of space freed in KB

// Keep collection under 10MB
const freed = await collection.freeSpace(10000);

// Free 5MB of space
const freed = await collection.freeSpace(-5000);

async createIndex(fieldPath: string, options?: object)

Creates an index for optimized queries.

Parameters:

• fieldPath: Dot-notation path to field
• options: Index configuration
  • unique: Enforce uniqueness
  • multiEntry: Index array values

await collection.createIndex('data.email', { unique: true });
await collection.createIndex('data.tags', { multiEntry: true });

Properties

• name: Collection identifier
• sizeKB: Total size in kilobytes
• length: Document count
• keys: Array of document IDs
• documentsMetadata: Metadata for all documents
• observer: Event observer instance

Document Class

Constructor

new Document(data: object, encryptionKey?: string)

Static Methods

static hasAttachments(documentData: object)

Checks for attachment presence.

if (Document.hasAttachments(doc)) {
  // Process attachments
}

static isEncrypted(documentData: object)

Verifies encryption status.

if (Document.isEncrypted(doc)) {
  // Request decryption key
}

static async decryptDocument(documentData: object, encryptionKey: string)

Decrypts an encrypted document.

const decrypted = await Document.decryptDocument(encDoc, 'password');

QuickStore Class

Provides synchronous localStorage-based storage for small documents.

const quickStore = db.quickStore;

// Synchronous operations
quickStore.setDocumentSync({ 
  _id: 'quick-1',
  data: { temp: true } 
});

const doc = quickStore.getDocumentSync('quick-1');
quickStore.deleteDocumentSync('quick-1');

const allKeys = quickStore.getAllKeys();

Advanced Features

Event System

The observer pattern enables reactive programming paradigms:

const collection = await db.getCollection('events');

// Register event handlers
collection.observer.on('beforeAdd', (doc) => {
  console.log('Document being added:', doc._id);
  // Validation logic
});

collection.observer.on('afterAdd', (doc) => {
  console.log('Document added successfully:', doc._id);
  // Trigger side effects
});

collection.observer.on('beforeDelete', (docId) => {
  console.log('Preparing to delete:', docId);
  // Cleanup logic
});

// Remove handler
const handler = (doc) => console.log(doc);
collection.observer.on('afterGet', handler);
collection.observer.off('afterGet', handler);

Transaction Management

LacertaDB ensures atomicity through transaction management:

// Batch operations execute atomically
const documents = [
  { data: { id: 1, value: 'A' }},
  { data: { id: 2, value: 'B' }},
  { data: { id: 3, value: 'C' }}
];

try {
  for (const doc of documents) {
    await collection.addDocument(doc);
  }
  // All succeed or all fail
} catch (error) {
  console.error('Transaction failed:', error);
  // Automatic rollback
}

Compression Strategies

// Enable compression for large documents
await collection.addDocument({
  data: {
    largeText: 'Lorem ipsum...'.repeat(10000),
    matrix: new Array(1000).fill(new Array(1000).fill(0))
  },
  _compressed: true  // Reduces storage footprint
});

Metadata Analysis

// Access collection metadata
const metadata = collection.documentsMetadata;
console.table(metadata);

// Analyze storage patterns
const analysis = metadata.reduce((acc, doc) => {
  acc.totalSize += doc.size;
  acc.avgSize = acc.totalSize / metadata.length;
  acc.permanent += doc.permanent ? 1 : 0;
  acc.withAttachments += doc.attachment > 0 ? 1 : 0;
  return acc;
}, { totalSize: 0, avgSize: 0, permanent: 0, withAttachments: 0 });

console.log('Storage Analysis:', analysis);

Custom Indexing Strategies

// Create compound indexes
await collection.createIndex('data.category');
await collection.createIndex('data.price');

// Query using indexes
const results = await collection.query(
  { 'data.category': 'electronics' },
  { 
    index: 'data_category',
    orderBy: 'asc',
    limit: 100 
  }
);

Performance Optimization

Storage Strategy Guidelines

1. Document Size Optimization

   • Keep documents under 1MB for optimal performance
   • Use compression for documents > 100KB
   • Store large binaries as attachments

2. Indexing Best Practices

   • Create indexes before bulk inserts
   • Index frequently queried fields
   • Avoid over-indexing (max 5-7 per collection)

3. Batch Operations

   // Efficient batch insert
   const batch = Array.from({ length: 1000 }, (_, i) => ({
     data: { index: i, value: Math.random() }
   }));
      
   for (const doc of batch) {
     await collection.addDocument(doc);
   }

4. Memory Management

   // Configure aggressive cleanup
   const db = new Database('app', {
     sizeLimitKB: 20000,      // 20MB limit
     bufferLimitKB: -2000,    // 2MB buffer
     freeSpaceEvery: 30000    // Check every 30s
   });

Query Optimization

// Use indexes for complex queries
await collection.createIndex('data.timestamp');

// Efficient pagination
async function* paginate(collection, pageSize = 100) {
  let offset = 0;
  let hasMore = true;
  
  while (hasMore) {
    const results = await collection.query({}, {
      limit: pageSize,
      offset: offset,
      orderBy: 'desc'
    });
    
    if (results.length < pageSize) {
      hasMore = false;
    }
    
    yield results;
    offset += pageSize;
  }
}

// Usage
for await (const page of paginate(collection)) {
  processDocuments(page);
}

Security Considerations

Encryption Architecture

LacertaDB implements a multi-layered security model:

1. Key Derivation: PBKDF2 with 600,000 iterations
2. Encryption: AES-GCM with 256-bit keys
3. Integrity: SHA-256 checksums
4. Salt Generation: Cryptographically secure random values

Security Best Practices

// Generate strong encryption keys
function generateSecureKey() {
  const array = new Uint8Array(32);
  crypto.getRandomValues(array);
  return btoa(String.fromCharCode(...array));
}

// Implement key rotation
async function rotateEncryption(collection, oldKey, newKey) {
  const docs = await collection.query({});
  
  for (const doc of docs) {
    if (doc._encrypted) {
      // Decrypt with old key
      const decrypted = await collection.getDocument(doc._id, oldKey);
      
      // Re-encrypt with new key
      await collection.addDocument({
        ...decrypted,
        _encrypted: true
      }, newKey);
    }
  }
}

Data Sanitization

// Sanitize before storage
function sanitizeDocument(doc) {
  // Remove sensitive fields
  delete doc.data.password;
  delete doc.data.creditCard;
  
  // Mask personal information
  if (doc.data.ssn) {
    doc.data.ssn = '***-**-' + doc.data.ssn.slice(-4);
  }
  
  return doc;
}

// Apply sanitization
await collection.addDocument(sanitizeDocument(userDoc));

Best Practices

1. Schema Design

// Define clear document structures
const UserSchema = {
  _id: null,  // Auto-generated
  data: {
    profile: {
      name: String,
      email: String,
      avatar: String
    },
    settings: {
      theme: String,
      notifications: Boolean
    },
    metadata: {
      createdAt: Number,
      updatedAt: Number,
      version: Number
    }
  },
  _permanent: false,
  _encrypted: false
};

2. Error Handling

class DatabaseService {
  async safeOperation(operation) {
    try {
      return await operation();
    } catch (error) {
      if (error.code === 'QUOTA_EXCEEDED') {
        await this.handleQuotaExceeded();
      } else if (error.code === 'TRANSACTION_FAILED') {
        await this.retryOperation(operation);
      } else {
        this.logError(error);
        throw error;
      }
    }
  }
  
  async handleQuotaExceeded() {
    const collection = await this.db.getCollection('cache');
    await collection.freeSpace(5000); // Free 5MB
  }
}

3. Migration Strategies

async function migrateDatabase(db, fromVersion, toVersion) {
  const migrations = {
    '1.0': async () => {
      // Version 1.0 -> 1.1 migration
      const users = await db.getCollection('users');
      const docs = await users.query({});
      
      for (const doc of docs) {
        if (!doc.data.version) {
          doc.data.version = '1.1';
          await users.addDocument(doc);
        }
      }
    },
    '1.1': async () => {
      // Version 1.1 -> 1.2 migration
      await db.createCollection('analytics');
    }
  };
  
  // Execute migrations sequentially
  const versions = Object.keys(migrations).sort();
  for (const version of versions) {
    if (version > fromVersion && version <= toVersion) {
      await migrations[version]();
    }
  }
}

4. Testing Patterns

// Unit testing example
describe('LacertaDB Operations', () => {
  let db, collection;
  
  beforeEach(async () => {
    db = new Database('test-db');
    await db.init();
    collection = await db.createCollection('test');
  });
  
  afterEach(async () => {
    await db.deleteDatabase();
  });
  
  test('Document CRUD operations', async () => {
    const doc = {
      data: { test: true },
      _permanent: false
    };
    
    // Create
    const isNew = await collection.addDocument(doc);
    expect(isNew).toBe(true);
    
    // Read
    const retrieved = await collection.getDocument(doc._id);
    expect(retrieved.data.test).toBe(true);
    
    // Update
    doc.data.test = false;
    const updated = await collection.addDocument(doc);
    expect(updated).toBe(false);
    
    // Delete
    const deleted = await collection.deleteDocument(doc._id);
    expect(deleted).toBe(true);
  });
});

Troubleshooting

Common Issues and Solutions

1. Quota Exceeded Errors

Symptom: DOMException: Quota exceeded

Solution:

// Implement automatic cleanup
db.settings.set('sizeLimitKB', 10000);
db.settings.set('freeSpaceEvery', 5000);

// Manual cleanup
const collection = await db.getCollection('cache');
await collection.freeSpace(2000); // Keep under 2MB

2. Encryption Key Loss

Symptom: Documents return false when retrieved

Solution:

// Implement key recovery mechanism
const recoveryQuestions = {
  q1: 'First pet name?',
  q2: 'Birth city?'
};

function deriveKeyFromAnswers(answers) {
  const combined = Object.values(answers).join('|');
  return crypto.subtle.digest('SHA-256', 
    new TextEncoder().encode(combined)
  );
}

3. Performance Degradation

Symptom: Slow query responses

Solution:

// Optimize with indexes
await collection.createIndex('data.timestamp');
await collection.createIndex('data.status');

// Use pagination
const results = await collection.query(
  { 'data.status': 'active' },
  { limit: 50, offset: 0 }
);

4. Transaction Failures

Symptom: Intermittent save failures

Solution:

// Implement retry logic
async function reliableSave(collection, doc, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await collection.addDocument(doc);
    } catch (error) {
      if (i === maxRetries - 1) throw error;
      await new Promise(r => setTimeout(r, 100 * Math.pow(2, i)));
    }
  }
}

Migration Guide

From LocalStorage

// Migrate localStorage data to LacertaDB
async function migrateFromLocalStorage() {
  const db = new Database('migrated');
  await db.init();
  const collection = await db.createCollection('legacy');
  
  for (let i = 0; i < localStorage.length; i++) {
    const key = localStorage.key(i);
    const value = localStorage.getItem(key);
    
    try {
      const data = JSON.parse(value);
      await collection.addDocument({
        _id: key,
        data: data
      });
    } catch (e) {
      // Handle non-JSON values
      await collection.addDocument({
        _id: key,
        data: { value: value }
      });
    }
  }
  
  // Optionally clear localStorage
  localStorage.clear();
}

From IndexedDB (Raw)

// Migrate from raw IndexedDB to LacertaDB
async function migrateFromIndexedDB(oldDbName, storeName) {
  // Open old database
  const oldDb = await new Promise((resolve, reject) => {
    const request = indexedDB.open(oldDbName);
    request.onsuccess = () => resolve(request.result);
    request.onerror = () => reject(request.error);
  });
  
  // Initialize LacertaDB
  const newDb = new Database('migrated');
  await newDb.init();
  const collection = await newDb.createCollection(storeName);
  
  // Transfer data
  const tx = oldDb.transaction([storeName], 'readonly');
  const store = tx.objectStore(storeName);
  const request = store.getAll();
  
  request.onsuccess = async () => {
    const records = request.result;
    for (const record of records) {
      await collection.addDocument({
        data: record
      });
    }
    
    oldDb.close();
    // Optionally delete old database
    indexedDB.deleteDatabase(oldDbName);
  };
}

License

MIT License - Copyright (c) 2024 Matias Affolter

Support and Contribution

For issues, feature requests, or contributions, please refer to the project repository. The LacertaDB ecosystem welcomes community involvement in advancing browser-based data persistence paradigms.

Development Roadmap

• v2.0: WebAssembly-accelerated cryptographic operations
• v2.1: Distributed synchronization protocols
• v2.2: Machine learning-driven compression algorithms
• v2.3: GraphQL query interface implementation

Documentation Version: 1.0.0 | Last Updated: 2024