SmolDB

A file-based NoSQL database for Bun.js with slab allocation, secondary indexes, and background garbage collection.

Installation

bun install

Quick Start

import { SmolDB } from 'smoldb';

const db = new SmolDB('./data');
await db.init();

// Get a collection
const users = db.collection('users');

// CRUD operations
await users.insert('user_1', { name: 'Alice', role: 'admin' });
const user = await users.get('user_1');
await users.update('user_1', { name: 'Alice', role: 'superadmin' });
await users.delete('user_1');

// Secondary indexes for fast queries
await users.createIndex('role');
const admins = await users.find({ role: 'admin' });

// Cleanup
await db.close();

Features

• Slab-based storage: Documents stored in size-bucketed slots (1KB, 8KB, 64KB)
• O(1) document access: Direct offset reads via persisted index
• Secondary indexes: Query by field values without full scans
• In-place updates: Small updates don't relocate documents
• Batch writes & bulk insert: Run many mutations under one write lock (batch) or append many small docs in one write (insertMany)
• Optional in-memory cache: LRU-ish per-collection cache for many-small-doc workloads (cacheSize)
• Background GC: Worker thread compaction reclaims deleted space
• Blob storage: Large documents (>1MB) automatically stored separately
• Nested field queries: Index and query user.profile.country
• Index-only queries: findIds and count(filter) can avoid reading documents when fully covered by indexes

When to Use SmolDB

SmolDB is designed for specific use cases. Choose it when your application matches these patterns:

Good Use Cases

| Use Case | Why SmolDB Works | |----------|------------------| | Embedded databases | No external server needed, just files | | Prototyping & MVPs | Simple API, no schema setup required | | Desktop/Electron apps | File-based storage, works offline | | CLI tools | Lightweight, no daemon processes | | Configuration stores | Persistent key-value with querying | | Session storage | Fast lookups by session ID | | Caching layer | Persist cache to disk between restarts | | Log aggregation | Append-heavy workloads with periodic compaction | | Single-user apps | No concurrent write conflicts | | Development/testing | Easy to inspect, reset, and version control |

When NOT to Use SmolDB

| Scenario | Better Alternative | |----------|-------------------| | Concurrent writers | Use a proper database with MVCC/locking | | Large datasets (>1M docs) | Use SQLite, MongoDB, or PostgreSQL | | Complex queries | Use SQL databases with proper query planners | | ACID transactions | Use SQLite or PostgreSQL | | Real-time sync | Use Firebase, Supabase, or CRDTs | | Multi-process access | Use client-server databases | | Production web APIs | Use PostgreSQL, MySQL, or MongoDB |

Benchmarks & Comparisons

Benchmarks live under benchmarks/ and write results to benchmarks/results/*.json.

The numbers below are from the latest committed runs on Linux x64 with Bun 1.3.4 (timestamp: 2025-12-17). Treat them as directional; they vary by SSD, filesystem, CPU governor, and dataset shape.

Versus LowDB (write-through workloads)

SmolDB is faster on write-through operations (each op persists to disk) and on indexed counting. LowDB can still win on “bulk insert then flush once” patterns.

| Operation (10k-doc dataset where applicable) | SmolDB vs LowDB | |---|---:| | Write-through insert | ~4.81× faster | | Write-through update | ~6.30× faster | | Write-through delete | ~6.02× faster | | Query: count() fully covered by indexes | ~1.69× faster | | Cold start: load + single read (10,000 docs) | ~1.39× faster | | Persist indexes / DB state | ~509× faster |

Versus vanilla JSON file storage

The JSON baseline is “read whole file → JSON.parse → modify → JSON.stringify → write whole file”. SmolDB avoids whole-file rewrites by writing fixed-size slots and maintaining a binary index.

| Operation (10k-doc dataset where applicable) | SmolDB vs JSON | |---|---:| | Write-through insert | ~1.39× faster | | Write-through update | ~1.88× faster | | Query: indexed count() vs JSON scan | ~2.04× faster | | Cold start: load + single read (10,000 docs) | ~1.45× faster |

Interpreting the “Batch Insert” numbers

Some baselines can bulk-load into memory and flush once, so their “batch insert” can be extremely fast.

SmolDB provides two bulk paths:

• collection.insertMany() for fast append of many small documents (single contiguous write)
• collection.batch() to group mixed inserts/updates/deletes under a single storage lock and flush metadata once

If your workload is “build everything in memory then write once”, a plain JSON write will often win.

The Sweet Spot

SmolDB excels when you have:

• Hundreds to tens of thousands of documents (not millions)
• Many small documents and a mix of reads/writes
• Need for secondary indexes without a full database
• Single-process access (CLI tools, desktop apps, scripts)
• Preference for simplicity over a full SQL engine

Honest Limitations

1. No multi-operation transactions: batch() reduces overhead, but it is not an ACID transaction API.
2. Single-process semantics: No cross-process locking; don’t point multiple processes at the same DB.
3. Index/query model: Secondary indexes are equality-only (no ranges/ordering/joins).
4. Memory usage: Primary + secondary indexes are in RAM (fine for typical embedded workloads; not ideal for huge cardinality).
5. Document reads are real I/O: find() must read matching documents; use findIds() / count(filter) when you only need IDs/counts.

API Reference

SmolDB

const db = new SmolDB(path: string, options?: SmolDBOptions);

interface SmolDBOptions {
  gcEnabled?: boolean;      // Enable background GC (default: true)
  gcTriggerRatio?: number;  // Trigger GC when fileSize/liveData > ratio (default: 2.0)
  blobThreshold?: number;   // Store docs larger than this as blobs (default: 1MB)
  cacheSize?: number;       // Optional per-collection document cache size (default: 0)
}

await db.init();                    // Initialize database
db.collection(name: string);        // Get or create collection
db.listCollections();               // List collection names
await db.dropCollection(name);      // Delete a collection
await db.compact();                 // Compact all collections
await db.persistAllIndexes();       // Save all indexes to disk
await db.getStats();                // Get database statistics
await db.close();                   // Close database

Collection

const coll = db.collection('users');

// CRUD
await coll.insert(id, document);    // Insert (throws if exists)
await coll.get(id);                 // Get by ID -> { id, data } | null
await coll.update(id, document);    // Update (throws if not found)
await coll.upsert(id, document);    // Insert or update
await coll.delete(id);              // Delete (returns boolean)
await coll.has(id);                 // Check existence

// Querying
await coll.find({ field: value });      // Find by field -> Array<{ id, data }>
await coll.findOne({ field: value });   // First match -> { id, data } | null
await coll.findIds({ field: value });   // Return matching IDs (index-only when fully indexed)
await coll.getAll();                    // Get all documents -> Array<{ id, data }>
await coll.keys();                  // Get all IDs
await coll.count();                 // Document count
await coll.count({ field: value }); // Count with filter (index-only when fully indexed)

// Indexes
await coll.createIndex('field');    // Create secondary index
await coll.createIndex('nested.field');
await coll.getIndexes();            // List indexed fields

// Maintenance
await coll.compact();               // Reclaim deleted space
await coll.clear();                 // Delete all documents
await coll.reset();                 // Fast truncate: clear storage + delete blobs
await coll.getStats();              // Collection statistics

// Bulk operations
await coll.insertMany([[id, doc], ...]); // Fast bulk insert for many small documents
await coll.batch(async (ops) => {
  await ops.insert('id1', { ... });
  await ops.update('id2', { ... });
  await ops.delete('id3');
});

// Iteration
for await (const { id, data } of coll) {
  console.log(id, data);
}

Examples

See the examples/ folder:

bun examples/basic-usage.ts      # CRUD operations
bun examples/secondary-indexes.ts # Querying with indexes
bun examples/nested-fields.ts    # Nested field indexing
bun examples/compaction.ts       # Space reclamation
bun examples/persistence.ts      # Data persistence
bun examples/http-api.ts         # REST API example

Benchmarks

bun benchmarks/run.ts            # Run all benchmarks
bun benchmarks/bench-json.ts     # vs Vanilla JSON
bun benchmarks/bench-lowdb.ts    # vs LowDB
bun benchmarks/bench-scale.ts    # Scale tests

Results are saved to benchmarks/results/*.json.

Architecture

Data File (.data)           Index File (.idx)
┌─────────────────┐         ┌─────────────────┐
│ Header (64B)    │         │ Header (64B)    │
├─────────────────┤         ├─────────────────┤
│ Slot 0 [1KB]    │◄────────│ id → offset     │
├─────────────────┤         │ id → offset     │
│ Slot 1 [8KB]    │◄────────│ ...             │
├─────────────────┤         ├─────────────────┤
│ Slot 2 [1KB]    │         │ Secondary Index │
│ (deleted)       │         │ field → values  │
├─────────────────┤         │ value → ids     │
│ Slot 3 [64KB]   │◄────────│ ...             │
└─────────────────┘         └─────────────────┘

• Slab sizes: 1KB → 8KB → 64KB → 4KB-aligned (huge)
• Slot header: 16 bytes (flags, length, slab size, CRC32)
• Index: Binary format for fast load/save

License

MIT