o876-json-db

An embeddable JSON document database for Node.js, written in TypeScript.

o876-json-db stores JSON documents in collections, either on the file system (one .json file per document) or in memory. It keeps lightweight in-memory indexes on top-level properties and offers a small MongoDB-like query language. It has no runtime dependencies.

Think of it as a smarter file system rather than a database server: your data stays plain JSON files, and the library adds indexes, queries and safe writes on top of them. See Intended use.

Table of Contents

• Intended use
• Installation
• Quick start
• Concepts
• Indexes
• Queries
• Updating documents
• Shutting down
• TypeScript
• Storage adapters
• API reference
• Limitations
• Development

Intended use

o876-json-db gives an application simple persistence for its entities, one step above reading and writing JSON files by hand. It suits:

• registered users and their accounts: lookups by email or username, with unique indexes,
• user-related data: profiles, settings, inventories, histories…, keyed or indexed by user id,
• snapshots of the internal state of parts of the application, saved and reloaded by key.

It is designed for one application process handling up to a few tens of thousands of documents per collection. In that range, indexed queries take a few milliseconds. init() takes about 2 seconds per 50,000 documents.

It is not designed for a complex business data model. There are no relations or joins, no transactions spanning several documents, no sorting, no $or/$in, and no access from several processes. If you need those, use a real database (SQLite through node:sqlite is a good next step).

Installation

npm install o876-json-db

Requires Node.js 20 or later. The package is published as an ES module with TypeScript type declarations (TypeScript 5.4 or later). CommonJS projects can require() it on Node.js ≥ 20.19 / 22.12.

Quick start

import { Collection, DiskStorage, INDEX_TYPES } from 'o876-json-db';

interface User {
    name: string;
    email: string | null;
    age: number;
    active: boolean;
}

// 1. Declare the collection and its indexes
const users = new Collection<User>('./data/users', {
    name: { type: INDEX_TYPES.PARTIAL, size: 3, caseInsensitive: true },
    email: { type: INDEX_TYPES.HASH, caseInsensitive: true, nullable: true, unique: true },
    age: { type: INDEX_TYPES.NUMERIC, precision: 10 },
    active: { type: INDEX_TYPES.BOOLEAN },
});

// 2. Choose a storage adapter (files only readable by the process owner)
users.storage = new DiskStorage({ fileMode: 0o600, dirMode: 0o700 });

// 3. Initialize: creates ./data/users if needed, reads all documents and builds the indexes
await users.init();

// 4. Use it
await users.save('alice', { name: 'Alice', email: null, age: 31, active: true });
await users.save('bob', { name: 'Bob', email: '[email protected]', age: 45, active: false });

const alice = await users.load('alice');

const cursor = await users.find({ age: { $gte: 30, $lt: 50 }, active: true });
const results = await cursor.fetchAll(); // [{ name: 'Alice', ... }]

// read-modify-write, safe against concurrent updates of the same document
await users.update('alice', (user) => user && { ...user, active: false });

await users.delete('bob');

// 5. On shutdown: wait for pending writes
await users.close();

Concepts

• Collection: a set of documents sharing the same location (a folder for DiskStorage).
• Document: any JSON object. It is identified by a key that you choose.
• Key: may only contain a-z A-Z 0-9 _ - .. With DiskStorage the key is the file name (<key>.json). On case-insensitive file systems (macOS, Windows by default), Alice and alice are the same file: avoid keys that only differ by case.
• Index: an in-memory structure associating a top-level property value with document keys. Indexes are declared when the collection is built and rebuilt by init().
• Cursor: the result of find(). It holds the matching keys and loads documents lazily.

Indexes

Indexes are declared as the second constructor parameter. They cannot be added after init(). Only top-level properties can be indexed.

new Collection(path, {
    propertyName: { type: INDEX_TYPES.XXX, ...options },
});

| Type | Value type | Purpose | Options | | --------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | HASH | string | Equality searches on strings with many distinct values (emails, ids…). Stores a CRC of the value. | size (16 or 32, default 32), caseInsensitive, nullable, unique | | PARTIAL | string | Equality and range searches on strings. Only the first size characters are indexed (0 = whole string). | size (default 0), caseInsensitive, nullable, unique | | NUMERIC | number | Equality and range searches on numbers. Values are grouped in buckets of width precision. | precision (> 0, default 1), nullable, unique | | BOOLEAN | boolean | Equality searches on booleans. | nullable | | TRUTHY | anything | Indexes whether a value is truthy, and keeps null values apart. Speeds up { ban: null } or { ban: { $neq: null } } queries. | none (always accepts null and missing values) |

Options

• nullable (default false): allows null or missing values. A missing property is indexed as null, so { email: null } finds both. Without nullable, saving a document whose indexed property is null or missing throws a TypeError. Declare new indexes as nullable when existing documents don't have the property yet. Otherwise init() fails on those documents.
• unique (default false, HASH, PARTIAL and NUMERIC only): two documents cannot have the same value. See Unique indexes.
• caseInsensitive (default false, string indexes only): 'jack' matches 'Jack' and 'JACK'. Accents remain significant: 'elodie' does not match 'Élodie'.
• size:
  • PARTIAL: number of leading characters stored in the index. A small size uses less memory but makes queries load more documents to check them. 0 indexes the whole string.
  • HASH: CRC size, 16 or 32 bits.
• precision (NUMERIC): bucket width. With precision: 10, values 40 to 49.99 share a bucket. 1 (the default) stores exact values. It must be greater than 0.

Exact and non-exact indexes

Some indexes store the exact value (NUMERIC with precision 1, BOOLEAN, PARTIAL with size 0 and case-sensitive). For these, an equality query is answered by the index alone. The other indexes only narrow the search: the candidate documents are then loaded and checked. Results are always exact. Only the number of loaded documents differs, which collection.stats reports.

Unique indexes

With unique: true, save() rejects with a UniqueConstraintError when another document already has the same value. Nothing is written in that case.

import { UniqueConstraintError } from 'o876-json-db';

try {
    await users.save('bob', { name: 'Bob', email: '[email protected]', age: 40, active: true });
} catch (e) {
    if (e instanceof UniqueConstraintError) {
        // e.property: 'email', e.conflictingKey: 'alice', e.key: 'bob'
    }
}

• "Same value" follows the index settings: with caseInsensitive, [email protected] and [email protected] are duplicates. With a PARTIAL index, whole values are compared, not just the indexed prefix.
• null and missing values are not considered duplicates: several documents may have no email.
• The check is safe against concurrent saves: if several save() calls use the same value at the same time, only one succeeds.
• init() rejects with a UniqueConstraintError if stored documents already contain duplicates.
• The error message names the property and the documents, but not the value, which may be personal data.
• Uniqueness is guaranteed within one process only (see Limitations).

Document keys are unique too. When a value never changes (a username, for example), using it as the key is the simplest way to make it unique.

Type checking

Saving a document is all-or-nothing: if an indexed property has the wrong type, save() throws a TypeError before anything is written, and the previous version of the document stays untouched.

Queries

find(query) returns a Cursor. A query is an object whose properties are conditions on the document's top-level properties. All conditions must be satisfied (logical AND).

await users.find({ name: 'Alice' }); // equality
await users.find({ email: null }); // property is null or missing
await users.find({ name: /^al/i }); // regular expression (string properties only)
await users.find({ age: { $gte: 18, $lt: 65 } }); // operators (all must be satisfied)
await users.find({ active: true, age: { $gt: 40 } }); // several properties
await users.find({}); // all documents

| Operator | Matches documents whose property… | Operand type | | -------- | ----------------------------------- | ----------------------------- | | $eq | is equal to operand | string, number, boolean, null | | $neq | is not equal to operand | string, number, boolean, null | | $gt | is greater than operand | string, number | | $gte | is greater than or equal to operand | string, number | | $lt | is less than operand | string, number | | $lte | is less than or equal to operand | string, number |

Semantics:

• A missing property is considered null: { email: null } matches documents without an email, { email: { $neq: null } } matches documents with a non-null email, and { email: { $neq: 'x' } } includes documents without an email.
• Range operators ($gt, $gte, $lt, $lte) only match values of the same type as the operand: { age: { $lt: 10 } } never matches null, missing values or strings.
• Strings are compared with String.prototype.localeCompare (locale-aware order).
• On an indexed property, the value must match the index type: { age: 'abc' } on a NUMERIC index throws a TypeError.
• Range operators use the index on NUMERIC and PARTIAL properties. On other properties (HASH, non-indexed…) they scan the documents.
• An unknown operator (e.g. a typo such as $gtee) throws a TypeError. Nested objects, arrays and dates are not supported as query values.
• Indexed conditions are evaluated first. Non-indexed conditions then only scan the documents selected so far. Put at least one indexed property in queries on large collections.
• Results are returned in collection key order (insertion order).

Updating documents

To change a document, use update() rather than load() followed by save():

// ❌ two concurrent requests both read logins = 3, both write 4: one login is lost
const user = await users.load('alice');
await users.save('alice', { ...user!, logins: user!.logins + 1 });

// ✅ the read, the change and the write happen as a single operation
await users.update('alice', (user) => user && { ...user, logins: user.logins + 1 });

update(key, updater) waits for pending operations on that document, then calls updater with a copy of the current document (undefined if it doesn't exist). The new version is then saved, and no other save(), update() or delete() on the same document can happen in between.

• updater returns the new document, or undefined to leave the document unchanged. It may be async.
• Returning a document for a missing key creates it: update(key, (doc) => doc ?? defaultDoc).
• update() resolves with the document as stored after the call.
• The new document goes through the same checks as save() (index types, unique indexes). If a check fails, or updater throws, nothing is written and update() rejects.
• updater must not call save(), update() or delete() on the same key: it would wait for itself forever. Other keys and other collections are fine.

Shutting down

close() waits for every pending save(), update() and delete() to complete. After it, these methods reject, while load() and find() keep working. Call it before the process exits, so that a deploy or a restart never interrupts a write the application has already acknowledged:

process.on('SIGTERM', async () => {
    await server.close(); // stop accepting requests first
    await Promise.all([users.close(), sessions.close()]);
    process.exit(0);
});

TypeScript

Give the document type as a type parameter. Interfaces and type aliases both work, including nested objects, arrays and optional properties. Properties that are not JSON values (Date, Map, functions…) are compile errors, since they would not survive being saved as JSON.

interface User {
    name: string;
    email?: string | null;
    logins: number;
    profile: { bio: string; tags: string[] };
}

const users = new Collection<User>('./data/users', {
    email: { type: INDEX_TYPES.HASH, nullable: true }, // index names are checked too
});

await users.find({ email: '[email protected]', logins: { $gte: 1 } }); // ✅
await users.find({ emial: '[email protected]' }); // ❌ compile error: unknown property
await users.find({ logins: '3' }); // ❌ compile error: logins is a number
await users.find({ logins: /3/ }); // ❌ compile error: RegExp only on string properties

Without a type parameter, a collection holds any JSON object (Collection<JsonObject>) and queries are not checked.

To write a generic class around a collection, constrain its type parameter with JsonCompatible:

import { Collection, type JsonCompatible } from 'o876-json-db';

class Repository<T extends JsonCompatible<T>> {
    constructor(protected readonly collection: Collection<T>) {}
}

Exported types include Query<T> (a query on documents of type T), IndexDeclarations<T>, DocumentUpdater<T>, JsonCompatible<T>, JsonObject and UniqueConstraintError.

Storage adapters

A storage adapter must be assigned to collection.storage before init().

DiskStorage

Stores each document as <collection path>/<key>.json (pretty-printed JSON). Writes go to a temporary file which is then renamed, so a crash never leaves a truncated document. Files that do not end with .json (such as .DS_Store) are ignored. I/O errors and invalid JSON files are reported as rejected promises.

new DiskStorage(); // Node.js default permissions
new DiskStorage({ fileMode: 0o600, dirMode: 0o700 }); // recommended for personal data

| Option | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------- | | fileMode | Permissions of document files. Applied every time a document is written, so existing files get them on their next save. | | dirMode | Permissions of folders created by the storage. Existing folders are not modified. |

By default, Node.js creates files readable by every user of the machine (0o644 with the usual umask). Use these options when documents contain personal data such as emails or password hashes. Modes are subject to the process umask, which can only remove permissions.

MemoryStorage

Keeps documents in memory. Data is lost when the process exits. Documents are deep-copied (structuredClone) on write and on read, so mutating a loaded document never alters stored data. Several collections may share the same MemoryStorage instance as long as their paths differ.

Custom storage

Implement the IStorage interface to store documents elsewhere (a key-value store, S3…):

import type { IStorage, JsonObject } from 'o876-json-db';

class MyStorage implements IStorage {
    async createLocation(location: string): Promise<void> {}
    async getList(location: string): Promise<string[]> {} // keys of all documents in location
    async read(location: string, key: string): Promise<JsonObject | undefined> {} // undefined if missing
    async write(location: string, key: string, data: JsonObject): Promise<void> {}
    async remove(location: string, key: string): Promise<void> {}
}

read() must return a fresh object on each call (never a reference to stored data), otherwise modifying a loaded document before saving it corrupts the indexes.

API reference

Collection<T>

| Member | Description | | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | new Collection<T>(path, indexes?) | Creates a collection. indexes maps property names to index options (see Indexes). | | storage | Storage adapter (get/set). Must be set before init(). | | init(): Promise<void> | Creates the location, lists documents and builds indexes. Must be called once, before anything else. | | save(key, doc): Promise<void> | Creates or replaces a document. Rejects with a TypeError (invalid indexed value) or a UniqueConstraintError. | | update(key, updater): Promise<T \| undefined> | Reads, modifies and saves a document as a single operation (see Updating documents). | | load(key): Promise<T \| undefined> | Loads a document, or undefined if it does not exist. | | delete(key): Promise<void> | Deletes a document. Does nothing if it does not exist. | | has(key): boolean | Returns true if a document exists with this key. Instant, no storage access. | | find(query): Promise<Cursor<T>> | Searches documents (see Queries). | | filter(predicate, keys?): Promise<string[]> | Returns the keys of documents for which predicate(doc, key, index) is true. Scans all documents, or only keys. | | keys: string[] | Keys of all documents (a new array on each access: prefer has() and size). | | size: number | Number of documents. | | path: string | Collection path. | | name: string | Last segment of the path. | | stats: CollectionStat | Statistics of the last find(): documents loaded, indexed/non-indexed properties, duration (µs). | | close(): Promise<void> | Waits for pending writes, then makes save(), update() and delete() reject (see Shutting down). |

Concurrent save()/update()/delete() calls on the same key are queued, so Promise.all is safe. So are concurrent saves of the same unique value.

Cursor<T>

| Member | Description | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | count: number | Number of documents. | | keys: string[] | Keys of the documents. | | index: number | Current position (get/set). | | currentKey: string \| undefined | Key at the current position. | | current(): Promise<T \| undefined> | Document at the current position. | | first(), last(), next(), previous() | Moves the position and returns the document there (undefined past the ends). | | forEach(fn): Promise<void> | Calls fn(doc, key) for each document, in order. Awaits fn if it returns a promise. | | fetchAll(start?, end?): Promise<T[]> | Loads documents from start (inclusive) to end (exclusive) at once. Beware of memory on large cursors. | | merge(cursor) | Adds the keys of another cursor (union, no duplicates). |

Limitations

• Single process. Indexes live in memory and are not shared. Do not open the same DiskStorage collection from two processes, and do not modify the .json files while the collection is open. Unique indexes are only enforced for the process that holds the collection.
• Startup cost. init() reads every document to build the indexes.
• Memory. Index size grows with the number of documents and distinct values.
• No transactions. Each save()/update()/delete() is atomic for one document only.
• Query language. No $in, $or, nested properties or sorting.

Development

| Script | Description | | ----------------------- | --------------------------------------------------------- | | npm test | Runs the test suite once (Vitest). | | npm run test:watch | Runs tests in watch mode. | | npm run test:coverage | Runs tests with a coverage report in coverage/. | | npm run typecheck | Type-checks the sources. | | npm run lint | Lints the sources (ESLint + Prettier). | | npm run build | Compiles to dist/ (ES modules + type declarations). | | npm run benchmark | Runs the benchmark. |

npm publish runs typecheck, lint, tests and build first (prepublishOnly).

The examples/mail-system folder contains a small mail system built on top of the library.

Contributions are welcome: please open an issue or a pull request on GitHub.