🚀 BMDB - High-Performance Embedded Database

BMDB is a high-performance, ACID-compliant embedded database for Node.js and Bun, featuring optimized Write-Ahead Logging (WAL), MVCC transactions, and vector search capabilities.

✨ Features

• 🔥 High Performance: Optimized WAL with 100-200x improvement on write-heavy workloads
• 💾 Multiple Storage Engines: JSON, Binary (MessagePack), Memory, WAL-optimized
• 🔄 ACID Transactions: Full MVCC support with snapshot isolation
• 🔍 Vector Search: Built-in vector similarity search with LSH indexing
• 📊 Schema Validation: Zod-powered schema validation with unique constraints
• 🎯 TypeScript First: Full TypeScript support with comprehensive type definitions
• 🚀 Zero Dependencies: Lightweight with minimal external dependencies
• 📱 Universal: Works in Node.js, Bun, and edge environments

📦 Installation

npm install bmdb

yarn add bmdb

pnpm add bmdb

bun add bmdb

🚀 Quick Start

Basic Usage

import { TinyDB, JSONStorage } from 'bmdb';

// Create database with JSON storage
const db = new TinyDB('db.json', JSONStorage);

// Insert documents
const users = db.table('users');
const userId = users.insert({ name: 'Alice', age: 30, email: '[email protected]' });

// Query documents
const user = users.get(userId);
const adults = users.search({ age: { $gte: 18 } });

High-Performance WAL Storage

import { TinyDB, WALJSONStorage } from 'bmdb';

// Use optimized WAL storage for high-throughput applications
const db = new TinyDB('db.json', WALJSONStorage, {
  batchSize: 1000,        // Batch up to 1000 operations
  maxBatchWaitMs: 20      // Maximum 20ms wait for batching
});

// Perform high-throughput writes
const table = db.table('events');
for (let i = 0; i < 10000; i++) {
  table.insert({ 
    timestamp: Date.now(), 
    event: `event_${i}`, 
    data: { value: i } 
  });
}

Schema Validation

import { TinyDB, createSchema, field, unique } from 'bmdb';
import { z } from 'zod';

// Define schema with validation
const userSchema = createSchema({
  name: field(z.string().min(1).max(100)),
  email: field(z.string().email()).unique(),
  age: field(z.number().int().min(0).max(150))
});

const db = new TinyDB('users.json');
const users = db.schemaTable('users', userSchema);

// Type-safe operations with validation
const user = users.insert({
  name: 'Bob',
  email: '[email protected]',
  age: 25
}); // TypeScript knows the shape and validates uniqueness

Vector Search

import { TinyDB, MemoryStorage } from 'bmdb';

const db = new TinyDB(MemoryStorage);
const embeddings = db.table('embeddings');

// Insert vectors
embeddings.insert({ 
  text: 'Hello world', 
  vector: [0.1, 0.2, 0.3, 0.4] 
});
embeddings.insert({ 
  text: 'Machine learning', 
  vector: [0.2, 0.3, 0.4, 0.5] 
});

// Search similar vectors
const query = [0.15, 0.25, 0.35, 0.45];
const similar = embeddings.vectorSearch('vector', query, { limit: 5 });

Transactions

import { TinyDB, WALStorage } from 'bmdb';

const db = new TinyDB('transactional.db', WALStorage);

// Use transactions for atomic operations
const txid = db.storage.beginTransaction();
try {
  db.storage.writeInTransaction(txid, { 
    accounts: { 
      alice: { balance: 950 },
      bob: { balance: 1050 }
    }
  });
  db.storage.commitTransaction(txid);
} catch (error) {
  db.storage.abortTransaction(txid);
  throw error;
}

📚 Storage Engines

JSONStorage

import { TinyDB, JSONStorage } from 'bmdb';
const db = new TinyDB('data.json', JSONStorage);

• Use case: Development, small datasets, human-readable storage
• Format: JSON files
• Performance: Good for small to medium datasets

WALJSONStorage (Recommended)

import { TinyDB, WALJSONStorage } from 'bmdb';
const db = new TinyDB('data.json', WALJSONStorage, {
  batchSize: 1000,
  maxBatchWaitMs: 20
});

• Use case: High-throughput applications, production workloads
• Features: ACID transactions, MVCC, optimized batching
• Performance: 100-200x faster on write-heavy workloads

BinaryStorage

import { TinyDB, BinaryStorage } from 'bmdb';
const db = new TinyDB('data.msgpack', BinaryStorage);

• Use case: Large datasets, storage efficiency
• Format: MessagePack binary format
• Performance: Compact storage, fast serialization

MemoryStorage

import { TinyDB, MemoryStorage } from 'bmdb';
const db = new TinyDB(MemoryStorage);

• Use case: Caching, testing, temporary data
• Features: In-memory only, no persistence
• Performance: Fastest access, no I/O overhead

🔍 Querying

Basic Queries

const users = db.table('users');

// Find by field value
users.search({ name: 'Alice' });

// Complex conditions
users.search({ 
  age: { $gte: 18, $lt: 65 },
  status: 'active'
});

// Using query builder
import { where } from 'bmdb';
users.search(where('age').gte(18).and(where('status').equals('active')));

Advanced Queries

// Regular expressions
users.search({ email: { $regex: /@company\.com$/ } });

// Array operations
users.search({ tags: { $contains: 'premium' } });

// Nested objects
users.search({ 'profile.settings.notifications': true });

🎯 Performance Optimizations

BMDB includes several performance optimizations:

WAL Optimizations

• Intelligent Batching: Groups operations to reduce fsync storms
• Optimistic Locking: Microsecond-level lock acquisition
• Incremental Compaction: Non-blocking 4MB slice processing
• MVCC Snapshots: Consistent reads without blocking writes

Memory Optimizations

• Object Pooling: Reuses objects to reduce GC pressure
• Copy-on-Write: Efficient data structure copying
• LRU Caching: Intelligent caching for frequently accessed data
• B-Tree Indexing: Fast lookups and range queries

🔧 Configuration

WAL Storage Options

const db = new TinyDB('data.json', WALJSONStorage, {
  batchSize: 1000,              // Operations per batch
  maxBatchWaitMs: 20,           // Maximum batch wait time
  compactThreshold: 5000,       // WAL size trigger for compaction
  autoFlushMs: 100,             // Auto-flush interval
  backgroundCompaction: true,   // Enable background compaction
  useMsgPack: false            // Use MessagePack for WAL entries
});

Schema Configuration

import { field, unique, primaryKey, compoundIndex } from 'bmdb';

const schema = createSchema({
  id: field(z.string()).primaryKey(),
  email: field(z.string().email()).unique(),
  name: field(z.string()),
  createdAt: field(z.date())
}, {
  // Compound indexes for efficient queries
  compoundIndexes: [
    compoundIndex(['name', 'createdAt'])
  ]
});

📊 Benchmarks

Performance comparison on write-heavy workloads:

| Storage Engine | Throughput (ops/sec) | Latency (ms) | Memory Usage | |----------------|---------------------|--------------|--------------| | WALJSONStorage | 50,000+ | <1ms | Low | | JSONStorage | 500 | 20ms | Medium | | BinaryStorage | 15,000 | 2ms | Low | | MemoryStorage | 100,000+ | <0.1ms | High |

🧪 Testing

# Run tests
bun test

# Run performance benchmarks
bun run test/performance-comparison.ts

🤝 Contributing

Contributions are welcome! Please read our contributing guidelines and submit pull requests.

📄 License

MIT License - see LICENSE file for details.

🔗 Related Projects

• TinyDB - Original Python implementation
• LokiJS - JavaScript document database
• NeDB - Embedded persistent database

Made with ❤️ for high-performance applications