jopi-filedb

A lightweight, hierarchical, file-based document store for Node.js and Bun. It provides a NoSQL-like experience with collections, sub-collections, views, and events, using the filesystem as the persistence layer.

Features

• Document-Oriented: Store data as JSON documents.
• Hierarchical: Support for deep nested sub-collections (e.g., users/{id}/posts/{id}/comments).
• File-Based: Data is stored human-readable JSON files.
• Views: Map-reduce style views (similar to CouchDB) for querying and indexing.
• Events: Listen to CRUD events on items and children.
• Caching: Optional memory caching for views to boost performance.

Core Concepts

Understanding the hierarchy and components of jopi-filedb:

• Item: A single document (JSON) within a collection.
• Collection: A logical container for items. Is like an Array, but with data stored on disk.
• Sub-collection: Item can have their own collections.
• Events: A notification system that allows you to react to item creation, updates, or deletions. Events can also propagate from sub-collections to parent listeners.
• View: An array of small objects, derived from item data. The goal is efficient retrieval of data.

Hierarchical Structure Example

The storage hierarchy directly maps to your filesystem:

db-root/
└── users/                 <-- Collection
    ├── [email protected]/ <-- Item (folder name is the id)
    │   ├── v.json         <-- Item data (Alice's profile)
    │   └── posts/         <-- Sub-collection
    │       ├── post1/     <-- Item (inside posts)
    │       │   └── v.json
    │       └── post2/
    │           └── v.json
    └── [email protected]/
        ├── v.json
        └── posts/
            └── post1/
                └── v.json

If a collection doesn't need sub-collections, you can use isStructured: false in the collection definition. This is simpler and stores one file per item:

db-root/
└── simple-logs/     <-- Collection
    ├── log-1.json   <-- Item (filename is the id)
    ├── log-2.json
    └── log-3.json

Usage

1. Initialization

Initialize the engine with a storage factory. The JsonFileStoreFactory stores data in a specified directory.

import { DbEngine, JsonFileStoreFactory } from 'jopi-filedb';

// Initialize storage in the './db-data' directory
const storeFactory = new JsonFileStoreFactory('./db-data');
const db = new DbEngine(storeFactory);

2. Defining Collections

Define your schema using declareCollection. You can nest collections and attach views.

import { newView, newView_AlwaysInMemCache } from 'jopi-filedb';

db.declareCollection({
    name: 'users',
    isStructured: true, // Default is true if having sub-collections

    // Custom ID generation.
    // If not provided, the id will be generated using UUID v4
    // Id format allows letters, numbers, hyphens, underscores, dots and @.
    generateId: (user) => user.email,
    
    // Define Views
    views: [
        // Simple view
        newView('by_age', async (emit, coll, user) => {
            // Each time an item is created or updated, this function is called.
            // It will emit the now rows for this items.
            // Old one will be automatically removed.
            //
            await emit(user.age, user.name);
        }),

        // View with memory caching enabled.
        // Allows keeping the view itemps in memory, for small lists frequently accessed.
        // 
        newView_AlwaysInMemCache('by_role', async (emit, coll, user) => {
            await emit(user.role, user.name);
        })
    ],

    // Define Sub-collections.
    // It a collection owned by an item himself.
    //
    // For exemple: all the orders of a client can be stored in a sub-collection of the client item.
    // It allows very efficient queries for such cases.
    // 
    collections: {
        posts: {
            isStructured: false, // Simple flat collection of files
            views: [
                newView('by_title', async (emit, coll, post) => {
                    await emit(post.title, post._id); 
                })
            ]
        }
    }
});

3. CRUD Operations

Creating and Writing Items:

const users = db.getCollection('users');

// Create a new item (ID generated automatically)
const alice = await users.create({ 
    name: "Alice", 
    email: "[email protected]", 
    age: 30 
});

console.log(`Created user with ID: ${alice._id}`);

// Update item
await alice.write({ 
    ...await alice.read(),
    age: 31 
});

// Create or update (upsert)
// The only difference with create() is that the _id is created if missing.
//
await users.createOrUpdate({
    _id: "bob",
    name: "Bob",
    age: 25
});

Reading Items:

const user = users.item("[email protected]");

if (await user.exists()) {
    const data = await user.read();
    console.log(data);
}

Iterating:

await users.forAll(async (item) => {
    const data = await item.read();
    console.log("-", data.name);
});

4. Working with Sub-collections

Sub-collections are accessible via the parent item.

const alice = users.item("[email protected]");
const alicePosts = alice.getCollection('posts');

await alicePosts.create({
    title: "My first post",
    content: "Hello world"
});

5. Using Views

Views allow you to query data efficiently. They are updated automatically when data changes.

const ageView = users.getView('by_age');

// Iterate over the view, sorted by key (age)
await ageView.forEach({ limit: 10 }, (row) => {
    console.log(`Age: ${row.k}, Name: ${row.r}`);
});

6. Rebuilding Views

If you modify a view's logic or if you suspect data inconsistency, you can manually trigger a full rebuild of the view. This will clear the existing view data and re-process every item in the collection through the view's handler.

You can trigger a rebuild either from the collection or directly from the view instance:

// Option 1: From the collection (requires the view name)
await users.rebuildView('by_age');

// Option 2: Directly from the view instance
const ageView = users.getView('by_age');
await ageView.rebuild();

7. Events

You can attach listeners to collections to react to changes.
It allows creating something like custom views.

db.declareCollection({
    name: 'logs',
    eventListeners: [{
        onItemCreated: async (coll, item, value) => {
            console.log(`New log entry: ${item._id}`);
        }
    }]
});

8. Embedded Resources

You can store resources (like images, binaries, or larger text files) that can be read and written using Streams. These files represent the final application content.

Note: In future versions, these resources might be moved to optimized external storage (S3, Cloud Storage, etc.), while keeping the same API.

const item = users.item("[email protected]");

// Get a resource accessor (supports sub-directories)
// Note: always use / as separator, even on Windows.
//
const photo = item.getEmbeddedResource("photos/2024/profile.jpg");

// Write via Stream (Web Streams API)
// Parent directories are created automatically if they don't exist.
const response = await fetch("https://example.com/photo.jpg");
await photo.writeStream(response.body!);

// Read via Stream
const stream = photo.readStream();
// ... process stream

// You can also use text methods
await item.getEmbeddedResource("config.json").writeText('{"active": true}');
const config = await item.getEmbeddedResource("config.json").read();

// Other methods
await photo.exists();
await photo.delete();

// Iterate over all resources (including sub-directories)
for await (const name of item.iterateEmbeddedResources()) {
    console.log("Resource:", name);
}

Under the hood: The files are stored inside a #res/ sub-folder. For example: users/alice/#res/profile.jpg.

Note: This feature requires the collection to be structured (isStructured: true), which is the default for collections with sub-collections.

9. Storing Data Files (Internal)

You can store additional files (like logs, scripts, or small data sets) directly attached to a specific item.

This feature is generally reserved for internal database data or structural data tied to the schema, such as data managed by custom views. Unlike Embedded Resources, these files are closely linked to the database engine's storage.

Note: This feature requires the collection to be structured (isStructured: true), which is the default for collections with sub-collections.

const item = users.item("[email protected]");

// Get a file accessor
const file = item.getDataFile("notes.txt");

// Write text (utf8) - overwrites existing content
await file.writeText("These are my notes.");

// Append to existing content
await file.append("\nMore specifically, about jopi-filedb.");

// Check if file exists
if (await file.exists()) {
    // Read the content
    const content = await file.read();
    console.log("Content:", content);
}

// Delete the file
await file.delete();

Under the hood: The files are stored inside the item's folder with a #data- prefix. For example: users/alice/#data-notes.txt.

Configuration

Structured vs Unstructured

• Structured (isStructured: true): Creates a folder for each item (id/v.json). Essential if the item will have sub-collections (id/sub_collection/) or data files (id/#data-filename).
• Unstructured (isStructured: false): Stores items as simple JSON files (id.json). More compact for leaf collections, but does not support sub-collections or item-attached files.

Memory Caching

For read-heavy workloads where view persistence I/O is a bottleneck, you can enable memory caching for specific views.

newView_AlwaysInMemCache('my_view', handler);

or

newView('my_view', handler, { alwaysInMemCache: true });

This keeps a copy of the view rows in memory to speed up reads and updates.