soly-db v1.5.4

A modern, high-quality NoSQL database for Node.js and TypeScript.

soly-db is a simple, fast, and reliable file-based NoSQL database designed for developers who want an easy-to-use solution for storing structured data in JSON format. Built with TypeScript for type safety and modern development workflows.

Developed and maintained by PROLEAK Innovation and JMM Entertainment.

What's new in v1.5.4

• Per-collection directories: data is stored under data/{collection}/
• Sharded NDJSON storage + secondary index (by id)
• WAL (append-only) + compaction, snapshot rotation (gzipped)
• Single-writer async queue, atomic writes (tmp + rename)
• New config: configure({ maxItemsPerShard, maxBytesPerShard })
• New async APIs: readAllAsync, readByIdAsync, saveBulkAsync, deleteByIdAsync, compactCollection, snapshotCollection
• read/save integrate with the new storage automatically (when items have id)

Features

• Blazing fast and lightweight
• File-based JSON storage
• Safe and automatic file creation
• TypeScript support with type safety
• Simple API for reading and writing data
• Perfect for prototyping, small apps, and learning
• Now safer for production use

Installation

npm install soly-db

Usage

Basic Usage (No Schema Validation)

import { read, save } from 'soly-db';

interface User {
  id: number;
  name: string;
}

const users = read<User>('users.json');

const newUsers: User[] = [
  { id: 1, name: 'John' },
  { id: 2, name: 'Jane' }
];
save<User>('users.json', newUsers);

Notes in v1.5.4:

• If data items have an id field (string | number), save('users.json', items) writes via WAL and sharded NDJSON under data/users/ automatically. read('users.json') aggregates from shards + WAL seamlessly.
• If items do not have id, it falls back to a single JSON array file at data/users.json.

Advanced Usage (With Schema Validation)

import { readWithSchema, saveWithSchema, createSchema } from 'soly-db';

interface User {
  id: number;
  name: string;
  email?: string;
  age: number;
}

// Create schema (optional)
const userSchema = createSchema({
  id: { type: 'number', required: true },
  name: { type: 'string', required: true },
  email: { type: 'string', required: false },
  age: { 
    type: 'number', 
    required: true,
    validate: (value: number) => value >= 0 && value <= 150
  }
});

// Read with schema validation
const { data: users, validation } = readWithSchema<User>('users.json', userSchema);
if (!validation.isValid) {
  console.warn('Data validation errors:', validation.errors);
}

// Save with schema validation
const newUsers: User[] = [
  { id: 1, name: 'John', age: 25 },
  { id: 2, name: 'Jane', email: '[email protected]', age: 30 }
];

const saveResult = saveWithSchema('users.json', newUsers, userSchema, {
  strict: false,        // Don't throw error on validation failure
  skipInvalid: false    // Save all data even if some items are invalid
});

Advanced Usage (Async, Sharding, WAL)

import { configure, saveBulkAsync, readByIdAsync, readAllAsync, compactCollection, snapshotCollection } from 'soly-db';

// Configure shard thresholds (optional)
configure({ maxItemsPerShard: 100_000, maxBytesPerShard: 100 * 1024 * 1024 });

// Write many records (append to WAL)
await saveBulkAsync('users.json', [
  { id: 1, name: 'John', age: 25 },
  { id: 2, name: 'Jane', age: 30 }
]);

// Compact WAL into a new NDJSON shard
await compactCollection('users.json');

// Read by id (uses index)
const user = await readByIdAsync('users.json', 2);

// Read limited list (newest shards first)
const recent = await readAllAsync('users.json', { limit: 100 });

// Snapshot the collection (gzipped files kept under ./snapshots)
const dir = await snapshotCollection('users.json', 7);

API

Basic Functions

read<T>(filePath: string): T[]

Reads from the collection. If a sharded layout exists (data/{collection}/), it aggregates from shards + WAL; otherwise it reads data/{collection}.json (legacy single-file array).

• File name must match: /^[a-zA-Z0-9_-]+\.json$/
• Path traversal is blocked (only under data/)
• Returns: Array of T (may be large; prefer async + limit for big datasets)

save<T>(filePath: string, data: T[]): void

Writes the provided data array.

• If every item has an id (string | number): saved via WAL → compacted to NDJSON shards under data/{collection}/
• Otherwise: saved as a single JSON array file at data/{collection}.json

Schema Validation Functions

readWithSchema<T>(filePath: string, schema?: Schema): { data: T[]; validation: ValidationResult }

Enhanced read function with optional schema validation.

• Returns: Object containing data and validation result
• Schema is optional: Works like regular read() if no schema provided

saveWithSchema<T>(filePath: string, data: T[], schema?: Schema, options?: SaveOptions): ValidationResult

Enhanced save function with optional schema validation and flexible error handling.

• Options:
  • strict?: boolean - If true, throw error on validation failure (default: false)
  • skipInvalid?: boolean - If true, save only valid items (default: false)
• Returns: ValidationResult with details about validation

createSchema(fields: SchemaDefinition): Schema

Helper function to create type-safe schema definitions.

validateItem(item: any, schema: Schema): ValidationResult

Utility function to validate a single item against a schema.

Schema Types

Async & Maintenance APIs (v1.5.4)

• configure(options: { maxItemsPerShard?: number; maxBytesPerShard?: number }): set sharding thresholds
• readAllAsync<T>(filePath: string, options?: { schema?: Schema; limit?: number }): Promise<T[]>
• readByIdAsync<T>(filePath: string, id: string | number, schema?: Schema): Promise<T | undefined>
• saveBulkAsync<T extends { id: string | number }>(filePath: string, data: T[], schema?, options?)
• deleteByIdAsync(filePath: string, id: string | number)
• compactCollection(filePath: string): collapse WAL → new shard, rotate active shard
• snapshotCollection(filePath: string, retain = 7): create gzipped snapshot under ./snapshots

Storage Layout (v1.5.4)

For a collection users.json:

data/
  users/
    meta.json            # manifest (active shard, shard list)
    byId.idx.json        # secondary index (id -> {file, offset, deleted})
    wal                  # append-only log
    00001.ndjson         # shards (NDJSON), newest first in manifest
    00002.ndjson

Snapshots:

snapshots/
  users-snapshot-YYYYMMDD-HHMMSS/
    meta.json.gz
    byId.idx.json.gz
    wal.gz
    00001.ndjson.gz
    00002.ndjson.gz

Performance Tips

• Prefer readByIdAsync for point lookups (uses index)
• Use readAllAsync(..., { limit }) and pagination for large lists
• Run compactCollection periodically to keep WAL small
• Tune configure({ maxItemsPerShard, maxBytesPerShard }) per dataset size
• Consider caching at the application layer for hot keys

interface SchemaField {
  type: 'string' | 'number' | 'boolean' | 'object' | 'array' | 'any';
  required?: boolean;
  validate?: (value: any) => boolean;
}

interface Schema {
  [key: string]: SchemaField;
}

interface ValidationResult {
  isValid: boolean;
  errors: string[];
}

Error Handling

• In production (NODE_ENV=production), error logs are generic and do not expose file paths or sensitive details.
• In development, detailed error logs are shown for easier debugging.

Example: Handling Errors

try {
  const users = read<User>('users.json');
} catch (err) {
  console.error('Failed to read users:', err);
}

Security Notes

• Only .json files with safe names are allowed (no directory traversal)
• All files and directories are created with secure permissions (700/600)
• Do not use untrusted user input as file names
• Data is not encrypted by default; add encryption if storing sensitive data

Why soly-db?

• No server required, zero config
• Works out-of-the-box with Node.js and TypeScript
• Great for rapid prototyping, learning, and small projects
• Developed by experienced teams: PROLEAK Innovation & JMM Entertainment

License

ISC

soly-db is an open-source project by PROLEAK Innovation and JMM Entertainment. Contributions and feedback are welcome!