MPackDB

A fast, local, append-only JSON database with MessagePack serialization for Node.js.

Features

• 🚀 High Performance - Append-only writes with MessagePack binary serialization
• 📇 Flexible Indexing - Optional numeric and lexical indexes for fast queries
• 🔒 Concurrent Access - File-based locking for safe multi-process access
• 💾 Auto-Persistence - Configurable automatic index persistence
• 🗜️ Auto-Compaction - Removes deleted records on startup
• 🎯 Simple API - Intuitive CRUD operations with async/await
• 📦 Zero Dependencies - Only requires msgpackr

Installation

npm install @worldapi.org/mpackdb

Quick Start

import MPackDB from '@worldapi.org/mpackdb';

// Create a database with auto-increment numeric primary key
const db = new MPackDB('data/users', {
  primaryKey: '*id',  // * prefix = numeric auto-increment
  indexes: ['email', '*age']  // Index email (lexical) and age (numeric)
});

// Insert records
await db.insert({ name: 'Alice', email: '[email protected]', age: 30 });
await db.insert({ name: 'Bob', email: '[email protected]', age: 25 });

// Find all records
for await (const user of db.find()) {
  console.log(user);
}

// Find by primary key
const users = await db.find(0);  // Returns array with Alice

// Query with function
for await (const user of db.find(r => r.age > 28)) {
  console.log(user.name);  // Alice
}

// Update records
await db.update(0, { age: 31 });
await db.update(r => r.age < 26, { status: 'junior' });

// Delete records
await db.delete(r => r.age < 18);

// Close database (persists all changes)
await db.close();

API Reference

Constructor

new MPackDB(dbFile, options)

Parameters:

• dbFile (string): Path to database file (without extension)
• options (object):
  • primaryKey (string): Primary key field name with optional prefix:
    • *field - Numeric auto-increment (e.g., *id)
    • @field - UUID (e.g., @uuid)
    • field - String (e.g., username)
  • primaryKeyType (PrimaryKeyType): Explicit type override
  • indexes (string[]): Fields to index (use prefixes like primary key)
  • debug (boolean): Enable debug logging (default: false)
  • indexPersistInterval (number): Auto-persist interval in ms (default: 60000, 0 to disable)
  • indexPersistThreshold (number): Auto-persist after N changes (default: 1000)

Examples: functional style:

import MPackDB from '@worldapi.org/mpackdb';

const db = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: ['category', '*price', '@sku'],
  indexPersistThreshold: 100
});

OOP style:

import { MPackDB, Model, PrimaryKeyType } from '@worldapi.org/mpackdb';

export class Product extends Model {
  id = 0;
  name = '';
  price = 0;
  category = '';
  sku = '';
}

export class Products extends MPackDB {
  _classToUse = Product;
  _primaryKey = 'id';
  _primaryKeyType = PrimaryKeyType.UUID;
  _indexes = ['category', '*price', '@sku'];
}

export default products = new Products('data/products');

insert(record, options)

Insert a new record into the database.

await db.insert({ name: 'Alice', age: 30 });
// Returns: { id: 0, name: 'Alice', age: 30 }

Parameters:

• record (object): The record to insert
• options (object):
  • skipPrimaryKey (boolean): Don't auto-generate primary key

Returns: Promise - The inserted record with primary key

find(query, options)

Find records in the database. Returns an async iterable Cursor.

// Find all
for await (const record of db.find()) {
  console.log(record);
}

// Find by primary key
const users = await db.find(0);

// Find with query function
for await (const user of db.find(r => r.age > 30)) {
  console.log(user.name);
}

Parameters:

• query (undefined|string|Function):
  • undefined/null - Returns all records
  • string - Primary key value
  • Function - Query function (record) => boolean
• options (object):
  • mode (string): Return mode - 'record', 'raw', or 'mixed'

Returns: Cursor (async iterable)

update(query, data, options)

Update records matching a query.

// Update by primary key
await db.update(0, { age: 31 });

// Update with query function
await db.update(r => r.age > 30, { status: 'senior' });

// Update with callback
await db.update(r => r.age > 30, r => ({ ...r, age: r.age + 1 }));

Parameters:

• query (string|Function): Primary key or query function
• data (object|Function): Data to update or callback function
• options (object):
  • upsert (boolean): Insert if no records match

Returns: Promise<Object[]> - Array of updated records

upsert(query, data)

Update records or insert if not found.

await db.upsert(r => r.email === '[email protected]', {
  name: 'Alice',
  email: '[email protected]',
  age: 30
});

delete(query, callback)

Delete records matching a query.

// Delete by primary key
await db.delete(0);

// Delete with query function
await db.delete(r => r.age < 18);

// Delete with callback
await db.delete(r => r.age < 18, async (record) => {
  console.log('Deleting:', record.name);
  return record;
});

Parameters:

• query (string|Function): Primary key or query function
• callback (Function): Optional callback for each deleted record

Returns: Promise<Object[]> - Array of deleted records

compact()

Compact the database by removing deleted records. This rewrites the data file without tombstones.

await db.compact();

Note: Compaction happens automatically on database initialization.

close()

Close the database and persist all pending changes. Should be called before process exit.

await db.close();

Primary Key Types

MPackDB supports three primary key types:

Numeric (Auto-increment)

const db = new MPackDB('data/users', {
  primaryKey: '*id'  // * prefix
});

await db.insert({ name: 'Alice' });
// { id: 0, name: 'Alice' }

UUID

const db = new MPackDB('data/sessions', {
  primaryKey: '@sessionId'  // @ prefix
});

await db.insert({ data: 'session data' });
// { sessionId: 'a1b2c3d4-...', data: 'session data' }

String

const db = new MPackDB('data/users', {
  primaryKey: 'username'  // No prefix
});

await db.insert({ username: 'alice', name: 'Alice' });
// { username: 'alice', name: 'Alice' }

Indexes

Indexes dramatically improve query performance for large datasets.

Index Types

• Lexical (default): String sorting, good for text fields
• Numeric: Number sorting, good for integers/floats
• UUID: Treated as lexical (string)

Creating Indexes

const db = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: [
    'category',      // Lexical index
    '*price',        // Numeric index
    '*stock',        // Numeric index
    '@sku'           // UUID index (lexical)
  ]
});

Index Persistence

Indexes are automatically persisted based on:

• Threshold: After N changes (default: 1000)
• Interval: Every N milliseconds (default: 60000)
• On close: When db.close() is called

const db = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['email'],
  indexPersistThreshold: 100,  // Persist after 100 changes
  indexPersistInterval: 30000  // Persist every 30 seconds
});

File Structure

MPackDB creates the following files:

data/
  users.mpack          # Main data file (MessagePack binary)
  users.meta.json       # Metadata (nextId, deleted offsets)
  users.id.txt          # Primary key index
  users.email.txt       # Email field index
  users.age.txt         # Age field index
  users.lock            # Lock file (temporary)

Concurrency

MPackDB uses file-based locking to ensure safe concurrent access:

// Process 1
const db1 = new MPackDB('data/users', { primaryKey: '*id' });
await db1.insert({ name: 'Alice' });

// Process 2 (waits for lock)
const db2 = new MPackDB('data/users', { primaryKey: '*id' });
await db2.insert({ name: 'Bob' });

Performance Tips

1. Use indexes for frequently queried fields
2. Adjust persist thresholds based on your write patterns
3. Call compact() periodically if you have many deletes
4. Use numeric indexes for number fields
5. Batch operations when possible

Examples

User Management System

import MPackDB from '@worldapi.org/mpackdb';

const users = new MPackDB('data/users', {
  primaryKey: '*id',
  indexes: ['email', '*age', 'role']
});

// Register user
await users.insert({
  email: '[email protected]',
  name: 'Alice',
  age: 30,
  role: 'admin'
});

// Find by email
for await (const user of users.find(u => u.email === '[email protected]')) {
  console.log('Found user:', user.name);
}

// Get all admins
for await (const admin of users.find(u => u.role === 'admin')) {
  console.log('Admin:', admin.name);
}

// Update age
await users.update(u => u.email === '[email protected]', { age: 31 });

// Delete inactive users
await users.delete(u => u.lastLogin < Date.now() - 90 * 24 * 60 * 60 * 1000);

await users.close();

Product Catalog

const products = new MPackDB('data/products', {
  primaryKey: '*id',
  indexes: ['category', '*price', '@sku']
});

// Add products
await products.insert({ sku: 'ABC-123', name: 'Laptop', category: 'Electronics', price: 999 });
await products.insert({ sku: 'DEF-456', name: 'Mouse', category: 'Electronics', price: 29 });

// Find by category
for await (const product of products.find(p => p.category === 'Electronics')) {
  console.log(product.name, product.price);
}

// Find products under $50
for await (const product of products.find(p => p.price < 50)) {
  console.log('Affordable:', product.name);
}

// Update price
await products.update(p => p.sku === 'ABC-123', { price: 899 });

await products.close();

License

MIT

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

Related Projects

• BsonDB - Similar database using BSON serialization