The database fast, easy and json format

In-memory JSON storage with a disk file behind it, with a schema/model API in the style of mongoose. Zero dependencies, encrypted at rest, and safe to use across cluster workers.

Installation

Requires Node.js 18 or newer. TypeScript is optional (this runs in plain JavaScript too) — if you do use it, 5.0+ gets you compile-time type checking on Schema/model from your field definitions; older versions still run, just without that checking.

npm i @webkn/storage

or

yarn add @webkn/storage

Features

• Fast — 100k inserts in ~0.25s, indexed lookups in O(1)
• Mongoose-style Schema/model API, with the document's TypeScript type inferred from the schema (TS 5.0+)
• Cluster-safe — many workers, one consistent copy of the data
• Encrypted at rest by default (AES-256-GCM, also detects tampering) — or plain JSON with encrypt: false
• Crash-safe atomic writes, with a backup file
• Auto id
• Zero runtime dependencies

Quick start

import { Schema, model } from '@webkn/storage';

const userSchema = new Schema(
  {
    userName: { type: String, required: true, index: true },
    email: String,
    role: { type: String, default: 'member' },
  },
  { path: './db', encrypt: false }, // storage options — see Schema options below
);

const User = model('User', userSchema);

// create() — build and save in one step
const admin = await User.create({ userName: 'admin', email: '[email protected]' });

// or new + save()
const guest = new User({ userName: 'guest', email: '[email protected]' });
await guest.save();

// queries
const all = await User.find();
const found = await User.findOne({ userName: 'admin' });
const byId = await User.findById(admin._id);

// mutate + save()
found.email = '[email protected]';
await found.save();

// or update in one call
await User.updateOne({ userName: 'guest' }, { role: 'admin' });

await found.remove();

Every static (find, findOne, create, updateOne, ...) and every instance method (save, remove) returns a Promise — await all of them.

Unlike mongoose, find/findOne resolve directly to documents (or null/[]), not to a chainable Query. There is no .sort()/.limit()/.select() to build a query further — just await it. Filters are plain objects and only support equality ({ userName: 'admin' }), ANDed across fields; no $gt/$in/etc.

Schema

new Schema(definition, options?)

definition describes fields, mongoose-style, and the document's TypeScript type is inferred from it — no separate interface User { ... } to write or keep in sync by hand:

const userSchema = new Schema({
  userName: { type: String, required: true, index: true }, // enforced + indexed
  email: String,                                            // optional, no metadata
  role: { type: String, default: 'member' },                // filled in if missing
  createdAt: { type: Number, default: () => Date.now() },   // a function runs per document
});

const User = model('User', userSchema);

await User.create({ email: '[email protected]' });
//                 ~~~~~~~~~~~~~~~~~~~
// Property 'userName' is missing — it's `required`.

await User.create({ userName: 123 });
//                             ~~~
// Type 'number' is not assignable to type 'string'.

Both errors happen at compile time, in your editor, before the code ever runs. This needs TypeScript 5.0+; on an older TypeScript the same code still runs correctly, it just won't be checked.

Only required, default, and index change runtime behavior — type alone drives the inferred TypeScript type, there is no runtime casting or validation beyond those two checks. One thing this does not catch: a field that isn't in the schema at all (User.create({ userName: 'x', typo: true })) is still accepted, because documents are intentionally loose (DocumentObject allows any extra key) — the checking is "is every known field the right type and present," not "are there stray fields."

options (the second argument) is where storage behavior lives — same shape as NovinStorageConfig below, minus name (that's model's first argument) and indexes (derived from index: true on fields instead).

Model statics

| Method | Returns | |--|--| | new Model(data?) | an unsaved instance — call .save() to persist it. data is optional, but if given must satisfy every required field | | Model.create(data) | build and save in one step, returns the saved instance. data must satisfy every required field | | Model.find(filter?) | every match, {} or omitted for everything | | Model.findOne(filter?) | the first match, or null | | Model.findById(id) | the document, or null | | Model.updateOne(filter, update) | true/false — merges update into the first match | | Model.updateMany(filter, update) | count of documents updated | | Model.findByIdAndUpdate(id, update) | the updated document, or null | | Model.deleteOne(filter) | true/false | | Model.deleteMany(filter) | count of documents removed | | Model.findByIdAndDelete(id) | true/false | | Model.countDocuments(filter?) | a number | | Model.storage | the underlying NovinStorage, for forceSave()/close()/cluster status |

updateOne/updateMany/findByIdAndUpdate do not run required checks — same as mongoose's default (validators run on save(), not on updates, unless you opt in).

Document instances

| Method | Effect | |--|--| | save() | insert if new, replace if already saved. Checks required fields first | | remove() | delete this document | | toObject() | a plain-object copy of its own fields, no methods |

const user = await User.findOne({ userName: 'admin' });
user.role = 'owner';
await user.save(); // update, because it was already loaded from storage

Calling model('User', schema) twice throws overwrite_model — same as mongoose, compile each model once and reuse the export rather than calling model() per request.

Advanced: the storage layer directly

Model is a thin wrapper over NovinStorage, exported too for anyone who wants to skip the schema and work with plain records:

import { NovinStorage } from '@webkn/storage';
import type { DocumentObject } from '@webkn/storage';

interface User extends DocumentObject {
  userName: string;
  email: string;
}

const users = new NovinStorage<User>({
  name: 'user',
  path: './db',
  encrypt: false,
  indexes: ['userName'],
});

await users.ready;

const user = await users.set({ userName: 'admin', email: '[email protected]' });
users.findById(user._id); // sync, no await — reads hit local memory

Reads are synchronous, writes return a promise. Writes have to be committed by one owner (see Cluster) before they are real, so they are awaited. await storage.ready resolves once the initial data is loaded; reads before that see an empty storage, writes wait for it on their own.

NovinStorageConfig

| Key | Description | Default | |--|--|--| | name | Storage name, used as the file name | (required) | | path | Directory holding the storage files | './db' | | encrypt | Encrypt the file with AES-256-GCM | true | | encryptionKey | Passphrase. See Encryption | (generated keyfile) | | indexes | Fields to index for O(1) lookups | [] | | autoId | Assign a random _id on insert | true | | saveDebounce | Delay in ms before the disk write that follows a change | 1000 | | saveBeautiful | Write pretty formatted JSON | false | | logInConsole | Log every operation | false |

Reads (sync)

| Method | Returns | |--|--| | findById(id) | the record, or null | | findByIds(ids) | the records that exist | | findByKey(key, value) | the first match, or null | | findAllByKey(key, value) | every match on one field | | match(filter) | every match across several fields (what Model.find uses) | | find() | every record, as a new array | | length | number of records |

NovinStorage is iterable, so for (const user of users) walks the records without building an array.

Treat records you read as read-only. Reads hand back the stored object itself, so changing it in place would skip the indexes and the save, leaving memory, indexes and file disagreeing. Call update instead:

const user = users.findByKey('userName', 'admin');
await users.update({ ...user, email: '[email protected]' }); // not user.email = ...

Writes take a copy of what you pass, so mutating your own object afterwards is harmless — same rule the Model layer follows.

Writes (async)

| Method | Returns | |--|--| | set(doc) | the record, with _id filled in. Throws on a duplicate _id | | update(doc) | true, or false if no record has that _id | | updateByKey(key, value, doc) | true, or false if nothing matched | | removeItem(id) | true, or false if it was not there | | removeItemByKey(key, value) | true, or false if nothing matched | | clear() | true, or false if already empty | | forceSave() | resolves once the file is written | | close() | flushes and releases the storage |

forAll walks every record, awaiting the callback. Return false to stop:

await users.forAll(async (user) => {
  await sendMessage(user._id, 'Happy new year!');
  await users.update({ ...user, sent: true });
});

Indexes

Lookups scan by default. Naming a field in indexes (or index: true on a schema field) turns lookups on it into a hash lookup, at the cost of some memory and slightly slower writes. Index what you actually query.

Cluster

What cluster.fork() is, for context: Node's built-in way to run several copies of your process on one machine, so a multi-core server can use more than one core (a single Node process is one thread). cluster.fork() spawns one worker — a full copy of your process — and the process that called fork() is the primary. A typical setup forks once per CPU core:

import cluster from 'node:cluster';
import os from 'node:os';

if (cluster.isPrimary) {
  for (let i = 0; i < os.cpus().length; i++) cluster.fork();
}
else {
  // this is a worker — your actual app/server code runs here
}

Why storage needs to know about this at all: every worker is a separate OS process with its own memory. If each one opened ./db/user.dbn independently and wrote to it, they would overwrite each other's changes — worker 2's write has no idea worker 1 just wrote a different record a millisecond earlier.

So exactly one process is put in charge of the file:

• the primary (the one that called fork()) owns the records and is the only process that ever touches disk;
• each worker keeps an in-memory copy (a replica), which is why reads stay synchronous — no round trip needed to read;
• a worker's writes are sent to the primary over Node's built-in IPC channel (the same channel cluster.fork() sets up automatically); the primary applies the write, saves it, and sends the result back to every worker so their copies stay in sync.

        ┌─────────┐
        │ primary │  ← only process that reads/writes the file
        └────┬────┘
    write requests in, results broadcast out
   ┌──────────┼──────────┐
   ▼          ▼          ▼
worker 1   worker 2   worker 3
(memory copy, always in sync)

You do not choose which worker is primary — it is simply whichever process called cluster.fork() (cluster.isPrimary is true there, false in every process it forked). That process must import this package, even if all it does is fork workers and nothing else:

import cluster from 'node:cluster';
import '@webkn/storage'; // required in the primary, even if unused otherwise

if (cluster.isPrimary) {
  for (let i = 0; i < 4; i++) cluster.fork();
}
else {
  // workers use NovinStorage / Schema / model normally
}

Without that import, nobody is listening for the workers' requests, and their ready rejects with storage_primary_timeout after 10 seconds.

Cluster gives every process one shared IPC channel, so if your own app also sends messages between primary and workers, your handlers will see this library's frames too. They all carry ch: '@webkn/storage/v1'; ignore anything with a ch field (this library already ignores messages that are not its own):

worker.on('message', (msg) => {
  if (msg?.ch) return; // not ours
  // ...
});

Consistency

Reading back a record you just awaited always works — the primary broadcasts the change before it replies, so your own replica is current when your await resolves.

Records written by other workers arrive a tick later. A worker can briefly serve a record another worker just deleted. If you need a read that cannot be stale, do it on the primary.

Encryption

By default, files are encrypted with AES-256-GCM, which also makes tampering an error rather than silent garbage:

new NovinStorage({ name: 'user', encryptionKey: process.env.STORAGE_KEY });

Without encryptionKey, a random key is generated and written to <path>/<name>.key. A key sitting next to the data protects nothing against someone who can read the directory — it is a convenience default, and the library warns when it is used.

Without encryption

Pass encrypt: false to store plain JSON instead — no key, no .key file:

const users = new NovinStorage<User>({ name: 'user', path: './db', encrypt: false });

or on a schema:

new Schema(definition, { path: './db', encrypt: false });

user.dbn is then a readable JSON array, nothing else:

[{"userName":"admin","email":"[email protected]","_id":"72690acc-d680-46ba-adb1-5e1ed8d5b76f"}]

Add saveBeautiful: true for indented, human-readable output:

[
  {
    "userName": "admin",
    "email": "[email protected]",
    "_id": "72690acc-d680-46ba-adb1-5e1ed8d5b76f"
  }
]

Opening a file that was encrypted with encrypt: false fails loudly (file_is_encrypted_but_encrypt_is_disabled) instead of returning garbage — the encrypted format starts with a fixed 4-byte header (NVN1) the reader checks for. The wrong key on an encrypted file fails the same way, with decrypt_failed_wrong_key_or_corrupt_file.

Durability

Writes are debounced by saveDebounce and coalesced, so a burst of changes costs one disk write. The write itself goes to a temp file, is fsynced, then renamed over the target — a reader sees either the old file or the new one, never a half-written one. The previous version is kept as <name>.dbn.bak.

Pending changes are flushed on process exit. Call await forceSave() (or storage.forceSave() off Model.storage) when you need a change on disk at a specific moment.

Data lives in memory, so a storage has to fit in it. This is a fast local store, not a substitute for a database that outgrows RAM.