MPackDB

A fast, local, append-only JSON database with MessagePack serialization for Node.js.

Features

• 🚀 High Performance - Append-only writes with MessagePack binary serialization
• 📇 Flexible Indexing - Optional numeric and lexical indexes for fast queries
• 🔒 Concurrent Access - File-based locking for safe multi-process access
• 💾 Auto-Persistence - Configurable automatic index persistence
• 🗜️ Auto-Compaction - Removes deleted records on startup
• 🎯 Simple API - Intuitive CRUD operations with async/await
• 📦 Zero Dependencies - Only requires msgpackr

Installation

npm install mpackdb

Quick Start

import MPackDB from 'mpackdb';

// Create a database with auto-increment numeric primary key
const db = new MPackDB('data/users', {
  primaryKey: '*id',  // * prefix = numeric auto-increment
  indexes: ['email', '*age']  // Index email (lexical) and age (numeric)
});

// Insert records
await db.insert({ name: 'Alice', email: '[email protected]', age: 30 });
await db.insert({ name: 'Bob', email: '[email protected]', age: 25 });

// Find all records
for await (const user of db.find()) {
  console.log(user);
}

// Find by primary key
const users = await db.find(0);  // Returns array with Alice

// Query with function
for await (const user of db.find(r => r.age > 28)) {
  console.log(user.name);  // Alice
}

// Update records
await db.update(0, { age: 31 });
await db.update(r => r.age < 26, { status: 'junior' });

// Delete records
await db.delete(r => r.age < 18);

// Close database (persists all changes)
await db.close();

API Reference

Constructor

new MPackDB(dbFile, options)

Parameters:

• dbFile (string): Path to database file (without extension)
• options (object):
  • primaryKey (string): Primary key field name with optional prefix:
    • *field - Numeric auto-increment (e.g., *id)
    • @field - UUID (e.g., @uuid)
    • field - String (e.g., username)
  • primaryKeyType (PrimaryKeyType): Explicit type override
  • indexes (string[]): Fields to index. Prefix options:
    • *field - Numeric index
    • @field - UUID index (lexical)
    • !field - Unique index (rejects duplicates)
    • !*field - Unique numeric index
    • field - Lexical index (default)
  • debug (boolean): Enable debug logging (default: false)
  • compact (boolean): Run compaction on init (default: true, set false for read-only / secondary instances)
  • indexPersistInterval (number): Auto-persist interval in ms (default: 60000, 0 to disable)
  • indexPersistThreshold (number): Auto-persist after N changes (default: 1000)

Examples: functional style:

import MPackDB from '@worldapi.org/mpackdb';

const db = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: ['category', '*price', '@sku'],
  indexPersistThreshold: 100
});

OOP style:

import { MPackDB, Model, PrimaryKeyType } from 'mpackdb';

export class Product extends Model {
  id = 0;
  name = '';
  price = 0;
  category = '';
  sku = '';
}

export class Products extends MPackDB {
  _classToUse = Product;
  _primaryKey = 'id';
  _primaryKeyType = PrimaryKeyType.UUID;
  _indexes = ['category', '*price', '@sku'];
}

export default products = new Products('data/products');

insert(record, options)

Insert a new record into the database.

await db.insert({ name: 'Alice', age: 30 });
// Returns: { id: 0, name: 'Alice', age: 30 }

Parameters:

• record (object): The record to insert
• options (object):
  • skipPrimaryKey (boolean): Don't auto-generate primary key

Returns: Promise - The inserted record with primary key

find(query, options)

Find records in the database. Returns an async iterable Cursor.

The Cursor is both an async iterable (for streaming with for await) and a thenable (awaiting it returns an array).

// Find all (streaming)
for await (const record of db.find()) {
  console.log(record);
}

// Find all (array)
const all = await db.find();

// Find by primary key
const users = await db.find(0);

// Find with filter function (stream)
for await (const user of db.find(r => r.age > 30)) {
  console.log(user.name);
}

// Find with single index — only reads records from the index range
for await (const user of db.find(r => r.age <= 50, {
  index: { field: 'age', from: 30, to: 50 }
})) {
  console.log(user.name);
}

// Find with index (descending)
for await (const user of db.find(null, {
  index: { field: 'age', from: 50, direction: 'desc' }
})) {
  if (user.age < 18) break;
  console.log(user.name);
}

// Find with index intersection — intersects offset sets, then streams only matches
const results = await db.find(null, {
  index: [
    { field: 'x', from: 0, to: 100 },
    { field: 'status', value: 'active' }
  ]
});

Parameters:

• query (undefined|string|number|Function):
  • undefined/null - Returns all records
  • string/number - Primary key value
  • Function - Filter function (record) => boolean
• options (object):
  • mode (string): Return mode - 'record', 'raw', or 'mixed'
  • index (object|array): Index hint(s) to narrow disk reads:
    • field (string): Indexed field name
    • value (any): Exact match lookup
    • from (any): Start key (inclusive) for range scan
    • to (any): End key (inclusive) for range scan
    • direction ('asc'|'desc'): Scan direction (default: 'asc')

When index is an array, offset sets from each index are intersected before streaming records from disk.

Returns: Cursor (async iterable + thenable)

update(query, data, options)

Update records matching a query.

// Update by primary key
await db.update(0, { age: 31 });

// Update with query function
await db.update(r => r.age > 30, { status: 'senior' });

// Update with callback
await db.update(r => r.age > 30, r => ({ ...r, age: r.age + 1 }));

Parameters:

• query (string|Function): Primary key or query function
• data (object|Function): Data to update or callback function
• options (object):
  • upsert (boolean): Insert if no records match
  • index (object|array): Index hint(s), same as find()

Returns: Promise<Object[]> - Array of updated records

upsert(query, data, options)

Update records or insert if not found.

await db.upsert(r => r.email === '[email protected]', {
  name: 'Alice',
  email: '[email protected]',
  age: 30
});

Parameters:

• query (string|Function): Primary key or query function
• data (object|Function): Data to update/insert or callback function
• options (object):
  • index (object|array): Index hint(s), same as find()

delete(query, options)

Delete records matching a query.

// Delete by primary key
await db.delete(0);

// Delete with query function
await db.delete(r => r.age < 18);

// Delete with index hint (avoids full scan)
await db.delete(r => r.status === 'inactive', {
  index: { field: 'status', value: 'inactive' }
});

Parameters:

• query (string|Function): Primary key or query function
• options (object):
  • index (object|array): Index hint(s), same as find()

Returns: Promise<Object[]> - Array of deleted records

compact()

Compact the database by removing deleted records. This rewrites the data file without tombstones.

await db.compact();

Note: Compaction happens automatically on database initialization.

boundingBox(corners, filter)

Find records within a bounding box defined by 4 corner coordinates. Requires numeric indexes on x and y fields. Corners can be in any order — min/max are extracted automatically.

const db = new MPackDB('data/sectors', {
  primaryKey: '*id',
  indexes: ['*x', '*y']
});

// Find all sectors in a bounding box
const sectors = await db.boundingBox([
  { x: -5, y: 5 },
  { x: 5, y: 5 },
  { x: 5, y: -5 },
  { x: -5, y: -5 }
]);

// Streaming
for await (const sector of db.boundingBox([
  { x: 0, y: 0 }, { x: 100, y: 0 },
  { x: 100, y: 100 }, { x: 0, y: 100 }
])) {
  console.log(sector);
}

// With additional filter
const active = await db.boundingBox([
  { x: -5, y: 5 }, { x: 5, y: 5 },
  { x: 5, y: -5 }, { x: -5, y: -5 }
], s => s.status === 'active');

Parameters:

• corners (Array<{x: number, y: number}>): 4 corner coordinates
• filter (Function): Optional additional filter function

Returns: Cursor (async iterable + thenable)

withLock(callback)

Execute a callback while holding the database lock. The lock is re-entrant: find, insert, delete, update called inside the callback reuse the same lock instead of deadlocking.

Use this for compound operations that must be atomic, e.g. find-then-insert.

const user = await db.withLock(async () => {
  const [existing] = await db.find(u => u.email === email);
  if (existing) return existing;
  return db.insert({ email, name });
});

Parameters:

• callback (Function): Async function to execute under lock

Returns: Promise - The return value of the callback

close()

Close the database and persist all pending changes. Should be called before process exit.

await db.close();

Primary Key Types

MPackDB supports three primary key types:

Numeric (Auto-increment)

const db = new MPackDB('data/users', {
  primaryKey: '*id'  // * prefix
});

await db.insert({ name: 'Alice' });
// { id: 0, name: 'Alice' }

UUID

const db = new MPackDB('data/sessions', {
  primaryKey: '@sessionId'  // @ prefix
});

await db.insert({ data: 'session data' });
// { sessionId: 'lz7gdcwh9x4r', data: 'session data' }

String

const db = new MPackDB('data/users', {
  primaryKey: 'username'  // No prefix
});

await db.insert({ username: 'alice', name: 'Alice' });
// { username: 'alice', name: 'Alice' }

Indexes

Indexes dramatically improve query performance for large datasets.

Index Types

• Lexical (default): String sorting, good for text fields
• Numeric: Number sorting, good for integers/floats
• UUID: Treated as lexical (string)
• Unique: Rejects duplicate values on insert (any type)

Creating Indexes

const db = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: [
    'category',      // Lexical index
    '*price',        // Numeric index
    '*stock',        // Numeric index
    '@sku',          // UUID index (lexical)
    '!email'         // Unique lexical index
  ]
});

Unique Indexes

Unique indexes prevent duplicate values. Primary keys are always unique by default. Use the ! prefix to make secondary indexes unique:

const db = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['!email', '!username', '*age']
});

await db.insert({ email: '[email protected]', username: 'alice', age: 30 }); // ok
await db.insert({ email: '[email protected]', username: 'bob', age: 25 });
// throws: Error { code: 'DUPLICATE_KEY', field: 'email', value: '[email protected]' }

Combine ! with type prefixes: !*field for unique numeric, !@field for unique UUID.

Uniqueness is enforced atomically under the write lock, so concurrent inserts cannot create duplicates.

Index Persistence

Indexes are automatically persisted based on:

• Threshold: After N changes (default: 1000)
• Interval: Every N milliseconds (default: 60000)
• On close: When db.close() is called

const db = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['email'],
  indexPersistThreshold: 100,  // Persist after 100 changes
  indexPersistInterval: 30000  // Persist every 30 seconds
});

File Structure

MPackDB creates the following files:

data/
  users.mpack          # Main data file (MessagePack binary)
  users.meta.json       # Metadata (nextId, deleted offsets)
  users.id.txt          # Primary key index
  users.email.txt       # Email field index
  users.age.txt         # Age field index
  users.lock            # Lock file (temporary)

Concurrency

MPackDB uses file-based locking to ensure safe concurrent access:

// Process 1
const db1 = new MPackDB('data/users', { primaryKey: '*id' });
await db1.insert({ name: 'Alice' });

// Process 2 (waits for lock)
const db2 = new MPackDB('data/users', { primaryKey: '*id' });
await db2.insert({ name: 'Bob' });

Multi-process reads

MPackDB automatically re-reads meta.json from disk before every read operation (refresh()). This means a second process (e.g. an admin UI) opening the same database files will always see the latest state, including records deleted by the main application. Without this, the in-memory _meta.deleted array would go stale and deleted records would reappear in query results.

You can also call db.refresh() manually if needed.

Important: Secondary/read-only instances should disable compaction with compact: false. Compaction replaces the data file via rename(), which invalidates the write stream of any other instance pointing at the old file inode.

// Main app — owns the data, compacts on startup (default)
const db = new MPackDB('data/users', { primaryKey: '*id' });

// Admin UI or secondary reader — no compaction
const admin = new MPackDB('data/users', { primaryKey: '*id', compact: false });

Performance Tips

1. Use indexes for frequently queried fields
2. Adjust persist thresholds based on your write patterns
3. Call compact() periodically if you have many deletes
4. Use numeric indexes for number fields
5. Batch operations when possible

Examples

User Management System

import MPackDB from '@worldapi.org/mpackdb';

const users = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['email', '*age', 'role']
});

// Register user
await users.insert({
  email: '[email protected]',
  name: 'Alice',
  age: 30,
  role: 'admin'
});

// Find by email
for await (const user of users.find(u => u.email === '[email protected]')) {
  console.log('Found user:', user.name);
}

// Get all admins
for await (const admin of users.find(u => u.role === 'admin')) {
  console.log('Admin:', admin.name);
}

// Update age
await users.update(u => u.email === '[email protected]', { age: 31 });

// Delete inactive users
await users.delete(u => u.lastLogin < Date.now() - 90 * 24 * 60 * 60 * 1000);

await users.close();

Product Catalog

const products = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: ['category', '*price', '@sku']
});

// Add products
await products.insert({ sku: 'ABC-123', name: 'Laptop', category: 'Electronics', price: 999 });
await products.insert({ sku: 'DEF-456', name: 'Mouse', category: 'Electronics', price: 29 });

// Find by category
for await (const product of products.find(p => p.category === 'Electronics')) {
  console.log(product.name, product.price);
}

// Find products under $50
for await (const product of products.find(p => p.price < 50)) {
  console.log('Affordable:', product.name);
}

// Update price
await products.update(p => p.sku === 'ABC-123', { price: 899 });

await products.close();

License

MIT

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

Related Projects

• BsonDB - Similar database using BSON serialization