Offload DB

A lightweight file-based key-value storage engine for Node.js that stores fixed-size records on disk using configurable storage classes. It is designed for predictable disk layouts, fast random access, heavy cache usage, minimal metadata overhead, and crash recovery through catalog reconstruction.

Features

• Fixed-size storage records
• Configurable storage classes ([keySize, valueSize])
• Automatic storage class selection
• Persistent on-disk catalog
• Crash recovery by rebuilding the catalog
• FIFO read cache
• In-memory transient cache
• Efficient record reuse through free-slot tracking
• Asynchronous API
• No external database required

Why? And for Who?

This is designed for small local-first user applications.

I got annoyed at my options for data storage when building silly apps for personal use. You either get JSON file storage - that is really fragile upon read/write failures - or have to pack an entire engine, like SQLite - with its binary and command-line overhead.

So, this aims to be a quick-and-dirty replacement for simple database and cache needs. Just copy the files, put it on whatever lib dir on your project, and voilá, it's working! you've just got an auto-managed LRU/MRU/FIFO cache and a local database system. As a plus, you get: crash recovery, low disk access, low memory usage, and optimized KV storage that you can customize to your specific needs.

Installation

npm install offload-db

const OffloadDB = require('offload-db');

Storage Classes

Storage is organized into storage classes, represented as:

[keySize, valueSize]

Example:

[
    [32, 128],
    [32, 256],
    [32, 512],
];

This means:

| Storage Class | Maximum Key | Maximum Value | | ------------- | ----------: | ------------: | | [32,128] | 32 bytes | 128 bytes | | [32,256] | 32 bytes | 256 bytes | | [32,512] | 32 bytes | 512 bytes |

Whenever a record is created, the engine automatically selects the smallest storage class capable of holding both the key and value.

This minimizes wasted space while keeping records fixed-size.

Example Storage Directory Layout

For classes [[32, 128], [32, 256], [32, 512]] the disk storage would look like:

{your root dir}/
└──{the name of your DB}/
    ├── .shutdown
    ├── catalog.json
    ├── gap.json
    │
    ├── 32-128/
    │     ├── 0.dat
    │     └── 1.dat
    │
    ├── 32-256/
    │     └── 0.dat
    │
    └── 32-512/
          └── 0.dat

Each storage class owns one or more data files.

When a data file reaches the configured maximum size, a new file is created automatically.

Initialization

const OffloadDB = require('offload-db');

const db = new OffloadDB('repl', storagePath);

const stat = await db.initialize([
    [32, 128],
    [32, 256],
    [32, 512],
]);

if (stat.error) {
    await db.shutdown();
    return;
}

Parameters:

| Parameter | Description | | ----------- | ---------------------------------------- | | namespace | Name of the database | | storagePath | Base directory where data will be stored |

initialize() creates missing directories, opens file handles and loads the catalog (or rebuilds it if the previous shutdown was unclean).

CRUD API

CREATE

await db.CREATE('MyKey', 'MyValue');

Optional size hints, that help the DB handle space, if that value grows later:

await db.CREATE('MyKey', 'MyValue', {
    estimatedKeySize: 32,
    estimatedValueSize: 128,
});

Returns

{
    error: null,
    storageClass: [32,128],
    fileNumber: 0,
    offset: 12
}

READ

const result = await db.READ('username');

console.log(result.data.value);

Returns

{
    error: null,
    data: {
        key: "username",
        value: "Tom",
        timestamp: ...,
        ...
    }
}

UPDATE

await db.UPDATE('username', 'Thomas');

Updates automatically copies the record into a larger storage class if necessary.

No cleanup tasks are needed due to the catalog structure.

DELETE

await db.DELETE('username');

Deletion marks the record inactive and returns the slot to the free-slot registry for reuse.

Listing Keys

Retrieve every key:

const list = db.LIST();

Filter using a string:

db.LIST('user');

Or a regular expression:

db.LIST(/^player_/);

Searching

FILTER

Reads every record and returns those matching a predicate.

const users = await db.FILTER(record => record.value.age >= 18);

FIND

Returns the first matching record.

const admin = await db.FIND(record => record.key === 'admin');

FIND_LAST

Returns the last matching record.

const latest = await db.FIND_LAST(record => record.value.active);

Temporary Cache

There is also a non-persistent (auto-managed) in-memory cache.

Create:

db.CREATE_TEMP('session', sessionData);

Read:

db.READ_TEMP('session');

Update:

db.UPDATE_TEMP('session', newData);

Delete:

db.DELETE_TEMP('session');

List keys:

db.LIST_TEMP();

Clear cache:

db.CLEAR_TEMP();

The transient cache automatically grows and shrinks based on utilization.

The main purpose of this feature is to allow you to create storage queues before making changes to the database. This optimizes disk access if you have worker threads, for example. Also, simplifies creation of queues, semaphore access control, and higher-level authorization implementations.

Shutdown

Always shut the database down cleanly:

await db.shutdown();

A successful shutdown:

• writes catalog.json
• writes gap.json
• creates the .shutdown marker
• closes every open file handle

If the process exits unexpectedly, the next initialization automatically scans data files to rebuild the catalog, but a clean shutdown speeds up the next initialization considerably.

Storage Format

Each record stored on disk contains:

[active]
[type]
[reserved]
[timestamp]
[valueLength]
[key]
[value]

Metadata is fixed-size, while the key and value sizes are determined by the storage class.

Inactive records remain on disk until reused.

Error Handling

Every public operation returns an object containing an error field.

Example:

const result = await db.READ('missing-key');

if (result.error) {
    console.error(result.error);
}

Common error codes include:

| Code | Description | | --------------- | ------------------------------------------- | | KEY_EXISTS | Attempted to create an existing key | | NO_VALID_STC | No storage class can accommodate the record | | UNDEFINED_KEY | Requested key does not exist |

API Reference

Constructor

new OffloadDB(namespace, storageDirectory)

Creates a new database manager instance.

Parameters

| Parameter | Type | Description | | ------------------ | -------- | ----------------------------------------------------------------------------------- | | namespace | string | Database namespace. A directory with this name is created under storageDirectory. | | storageDirectory | string | Base directory where all storage files will be located. |

Returns

instance of OffloadDB;

Initialization

initialize(storageClasses, options?)

Initializes the database, creates missing directories, opens storage files, loads or rebuilds the catalog, and prepares caches.

Parameters

storageClasses

Array<[number, number]>;

List of available storage classes.

Example:

[
    [32, 128],
    [32, 256],
    [32, 512],
];

options

{
    inherit?: boolean
}

| Property | Default | Description | | --------- | ------- | ------------------------------------------------------------- | | inherit | true | Automatically reuse storage classes already existing on disk. |

Returns

Promise<{
    error: string | null;
}>;

shutdown()

Safely closes the database.

Actions performed:

• Saves catalog
• Saves empty-slot registry
• Writes shutdown marker
• Closes all file handles

Returns

Promise<boolean>;

Returns true on success.

Persistent Storage API

CREATE(key, value, options?)

Creates a new record.

Parameters

key

string;

Unique key.

value

string | Buffer;

Data to store.

options

{
    estimatedKeySize?: number;
    estimatedValueSize?: number;
}

Optional storage size hints.

Returns

Promise<{
    error: string | Error | null;
    storageClass?: [number, number];
    fileNumber?: number;
    offset?: number;
}>;

Errors

• KEY_EXISTS
• NO_VALID_STC

READ(key)

Reads one record.

Parameters

| Name | Type | | ----- | -------- | | key | string |

Returns

Promise<{
    error: string | Error | null;
    data: {
        record_key_size: number;
        record_value_size: number;
        active: boolean;
        type: 'string' | 'buffer';
        reserved: number;
        timestamp: number;
        value_length: number;
        key: string;
        value: string | Buffer;
    } | null;
}>;

Errors

• UNDEFINED_KEY

UPDATE(key, value)

Updates an existing record.

If the new value no longer fits inside the original storage class, the record is transparently migrated to a larger one.

Parameters

| Name | Type | | ------- | ------------------ | | key | string | | value | string \| Buffer |

Returns

Promise<{
    error: string | Error | null;
}>;

Errors

• UNDEFINED_KEY

DELETE(key)

Deletes a record.

The storage slot becomes available for future insertions.

Parameters

| Name | Type | | ----- | -------- | | key | string |

Returns

Promise<{
    error: string | Error | null;
}>;

Errors

• UNDEFINED_KEY

Listing & Querying

LIST(regex?)

Lists stored keys.

Parameters

regex?: string | RegExp

Optional filter.

Returns

{
    error: null;
    data: string[];
}

FILTER(predicate)

Reads every stored record and returns those matching a predicate.

Parameters

predicate: (record, index) => boolean;

Returns

Promise<{
    error: null;
    data: Array<Record>;
}>;

Where Record is the object returned by READ().

FIND(predicate)

Returns the first matching record.

Parameters

predicate: record => boolean;

Returns

Promise<{
    error: string | Error | null;
    data: Record | null;
}>;

FIND_LAST(predicate)

Returns the last matching record.

Parameters

predicate: record => boolean;

Returns

Promise<{
    error: string | Error | null;
    data: Record | null;
}>;

Temporary Cache API

These methods operate only on the in-memory FIFO cache and are not persisted to disk.

CREATE_TEMP(key, value)

Creates a transient cache entry.

Parameters

| Name | Type | | ------- | -------- | | key | string | | value | any |

Returns

boolean;

READ_TEMP(key)

Reads a transient value.

Parameters

| Name | Type | | ----- | -------- | | key | string |

Returns

any;

UPDATE_TEMP(key, value)

Updates a transient entry.

Parameters

| Name | Type | | ------- | -------- | | key | string | | value | any |

Returns

boolean;

DELETE_TEMP(key)

Removes a transient entry.

Parameters

| Name | Type | | ----- | -------- | | key | string |

Returns

boolean;

LIST_TEMP()

Lists all transient cache keys.

Returns

string[]

CLEAR_TEMP()

Clears the transient cache.

Returns

boolean;

OPTIMIZE_TEMP()

Automatically resizes the transient cache according to utilization.

Normally there is no need to call this manually since it is invoked automatically during cache operations.

Returns

void

Returned Record Structure

Every successful READ(), FILTER(), FIND(), and FIND_LAST() returns records with the following structure:

interface StorageRecord {
    record_key_size: number;
    record_value_size: number;

    active: boolean;

    type: 'string' | 'buffer';

    reserved: number;

    timestamp: number;

    value_length: number;

    key: string;

    value: string | Buffer;
}

Error Codes

| Error | Description | | ------------------- | ------------------------------------------------------ | | KEY_EXISTS | Attempted to create an existing key. | | NO_VALID_STC | No storage class can accommodate the requested record. | | UNDEFINED_KEY | The requested key does not exist. | | INIT_FAIL_STDIR | Storage directory initialization failed. | | INIT_FAIL_SHUTCHK | Shutdown verification failed. | | INIT_FAIL_STCLASS | Storage class initialization failed. | | INIT_FAIL_FHANDLE | Failed to open storage files. | | INIT_FAIL_CATALOG | Catalog initialization or reconstruction failed. |