nywion.db

Advanced, robust local JSON database for Node.js with transactions, caching, encryption, indexes, and more.

Features

| Category | Features | |---|---| | Storage | In-memory, filesystem, hybrid (memory + disk) adapters | | CRUD | Get, set, delete, has, table management | | Querying | Filter operators, logical operators ($and/$or/$not), sorting, projection, pagination | | Query Builder | Fluent API with where, sort, limit, offset, select, exclude | | Transactions | ACID with Mutex, timeout, rollback, read-only mode | | Schema Validation | Type checking, regex patterns, required fields, custom validators, strict mode | | Caching | LRU and LFU eviction strategies with configurable TTL | | Encryption | AES-256-GCM encryption with key derivation | | Compression | gzip, brotli, deflate | | Indexes | B-Tree, Hash, Unique index types | | WAL | Write-Ahead Logging for crash recovery | | Snapshots | Backup and restore database state | | Migrations | Versioned schema migrations with up/down | | Observability | Structured logging, metrics (counters/gauges/histograms), distributed tracing | | Servers | HTTP REST API and WebSocket server | | Utilities | deepClone, Mutex, Semaphore, retry, debounce, throttle, path manipulation |

Install

npm install nywion.db

Requires Node.js >= 18.0.0.

Quick Start

import { Database } from 'nywion.db';

const db = new Database({
  storage: { adapter: 'memory' },
});

await db.init();

// Create a table and store data
await db.createTable('users');
await db.set('user1', { name: 'Alice', age: 30, email: '[email protected]' }, 'users');
await db.set('user2', { name: 'Bob', age: 25, email: '[email protected]' }, 'users');

// Get a single value
const user = await db.get('user1', 'users');
console.log(user); // { name: 'Alice', age: 30, email: '[email protected]' }

// Query with filters
const result = await db.query('users', { field: 'age', op: 'gte', value: 18 });
console.log(result.data); // [user1, user2]

// Transaction
await db.transaction(async (ctx) => {
  const user = await ctx.get('user1');
  await ctx.set('user1', { ...user, age: 31 });
});

await db.close();

Configuration

const db = new Database({
  // Storage adapter: 'memory' | 'file' | 'hybrid'
  storage: {
    adapter: 'hybrid',
    path: './data',           // Base directory for file storage
    pretty: true,             // Pretty-print JSON files
    wal: true,                // Enable Write-Ahead Logging
    compactInterval: 300_000, // Auto-compact interval in ms (5 min)
  },

  // Cache configuration
  cache: {
    maxSize: 10_000,      // Max items in cache
    defaultTtl: 60_000,   // Default TTL in ms (1 min)
    strategy: 'lru',      // 'lru' or 'lfu'
  },

  // Security (optional)
  security: {
    encryption: {
      enabled: false,
      key: '<base64-encoded-key>', // AES-256 key (auto-generated if omitted)
      algorithm: 'aes-256-gcm',   // Default: aes-256-gcm
    },
    compression: {
      enabled: false,
      algorithm: 'gzip', // 'gzip' | 'brotli' | 'deflate'
    },
  },

  // Transaction settings (optional)
  transactions: {
    timeout: 30_000,                         // Default timeout in ms
    isolation: 'snapshot',                   // 'snapshot' | 'read-uncommitted' | 'read-committed'
  },

  // Observability (optional)
  observability: {
    logging: true,    // Enable structured logging
    metrics: true,    // Enable metrics collection
    telemetry: false, // Send performance telemetry
  },
});

Default Values

| Setting | Default | |---|---| | storage.adapter | 'hybrid' | | storage.path | './data' | | storage.pretty | true | | storage.wal | true | | cache.maxSize | 10_000 | | cache.defaultTtl | 60_000 | | cache.strategy | 'lru' | | transactions.timeout | 30_000 | | transactions.isolation | 'snapshot' | | observability.logging | true |

Storage Adapters

Memory

Fully in-memory, no persistence. Best for testing and temporary data.

const db = new Database({ storage: { adapter: 'memory' } });

File

Filesystem-based storage. Each table is a JSON file under .nywion/.

const db = new Database({
  storage: { adapter: 'file', path: './data', pretty: true },
});

Hybrid

Memory-backed with filesystem persistence. Reads from memory, writes to disk. Best for production.

const db = new Database({
  storage: { adapter: 'hybrid', path: './data', wal: true },
});

CRUD Operations

await db.init();
await db.createTable('users');

// Set (with optional TTL for cache)
await db.set('key1', { name: 'Alice' }, 'users');
await db.set('key1', { name: 'Alice' }, 'users', 30_000); // 30s cache TTL

// Get
const value = await db.get('key1', 'users');

// Check existence
const exists = await db.has('key1', 'users');

// Delete
const deleted = await db.delete('key1', 'users'); // true

// List tables
const tables = await db.listTables();

// Drop table
await db.dropTable('users');

// Close
await db.close();

Querying

Direct Query

const result = await db.query('users', { field: 'age', op: 'gte', value: 18 });
console.log(result);
// { data: [...], count: 2, total: 2, limit: null, offset: 0 }

With Sorting, Projection, and Pagination

const result = await db.query('users',
  { field: 'age', op: 'gte', value: 18 },
  {
    sort: [{ field: 'age', direction: 'desc' }],
    projection: { include: ['name', 'email'] },
    limit: 10,
    offset: 0,
  }
);

Query Builder

const query = db.buildQuery()
  .where('age', 'gte', 18)
  .where('status', 'eq', 'active')
  .sort('name', 'asc')
  .select('name', 'email')
  .limit(10)
  .offset(5);

const result = await db.query('users', query.build());

Filter Operators

| Operator | Description | Example | |---|---|---| | eq | Equal | { field: 'name', op: 'eq', value: 'Alice' } | | neq | Not equal | { field: 'name', op: 'neq', value: 'Bob' } | | gt | Greater than | { field: 'age', op: 'gt', value: 18 } | | gte | Greater than or equal | { field: 'age', op: 'gte', value: 18 } | | lt | Less than | { field: 'age', op: 'lt', value: 65 } | | lte | Less than or equal | { field: 'age', op: 'lte', value: 65 } | | in | Value in array | { field: 'role', op: 'in', value: ['admin', 'editor'] } | | nin | Value not in array | { field: 'role', op: 'nin', value: ['banned'] } | | contains | String contains | { field: 'name', op: 'contains', value: 'lic' } | | regex | Regex match | { field: 'email', op: 'regex', value: '@example\\.com$' } | | exists | Field exists | { field: 'email', op: 'exists', value: true } | | type | Type check | { field: 'age', op: 'type', value: 'number' } | | all | Array contains all | { field: 'tags', op: 'all', value: ['a', 'b'] } | | size | Array length | { field: 'tags', op: 'size', value: 3 } |

Logical Operators

// $and
await db.query('users', {
  $and: [
    { field: 'age', op: 'gte', value: 18 },
    { field: 'status', op: 'eq', value: 'active' },
  ],
});

// $or
await db.query('users', {
  $or: [
    { field: 'role', op: 'eq', value: 'admin' },
    { field: 'age', op: 'gte', value: 65 },
  ],
});

// $not
await db.query('users', {
  $not: { field: 'status', op: 'eq', value: 'banned' },
});

Transactions

await db.transaction(async (ctx) => {
  const user = await ctx.get('user1');
  await ctx.set('user1', { ...user, age: user.age + 1 });
  // Auto-committed on success, auto-rolled-back on error
});

Transaction Options

await db.transaction(async (ctx) => {
  // ...
}, {
  timeout: 10_000,     // 10s timeout (default: 30s)
  readOnly: false,      // Read-only mode
  label: 'update-age', // Label for observability
});

Read-Only Transactions

await db.transaction(async (ctx) => {
  // ctx.set() and ctx.delete() will throw TransactionError
  const user = await ctx.get('user1');
}, { readOnly: true });

Schema Validation

import { Schema } from 'nywion.db';

const userSchema = new Schema('user', {
  name: { required: true, type: 'string' },
  age: { required: true, type: 'number' },
  email: { type: 'string', pattern: '^[\\w.-]+@[\\w.-]+\\.\\w+$' },
  role: { default: 'user' },
}, { strict: true });

db.setSchema('users', userSchema.definition);

// Now set() validates against the schema
await db.set('user1', { name: 'Alice', age: 30, email: '[email protected]' }, 'users');

// Invalid data throws ValidationError
await db.set('user2', { name: 'Bob' }, 'users');
// ValidationError: age is required

Schema Rules

| Rule | Type | Description | |---|---|---| | required | boolean | Field is required | | type | string | Type check: 'string', 'number', 'boolean', 'object', 'array', 'null' | | pattern | string \| RegExp | Regex pattern for string validation | | default | unknown \| Function | Default value (static or dynamic) | | validate | Function | Custom validation: returns true, false, or error message |

Strict Mode

When strict: true (default), unknown fields are rejected:

const schema = new Schema('item', {
  name: { required: true, type: 'string' },
}, { strict: true });

schema.validate({ name: 'test', unknown: 'value' });
// { valid: false, errors: [{ field: 'unknown', message: 'Unknown field "unknown" in strict mode' }] }

Caching

Cache is integrated automatically. Two eviction strategies:

// LRU (Least Recently Used) - default
const db = new Database({
  cache: { maxSize: 10_000, strategy: 'lru', defaultTtl: 60_000 },
});

// LFU (Least Frequently Used)
const db = new Database({
  cache: { maxSize: 10_000, strategy: 'lfu' },
});

Per-Key TTL

await db.set('key1', value, 'table', 30_000); // 30s TTL for this key

Encryption

AES-256-GCM encryption with authenticated encryption:

const db = new Database({
  security: {
    encryption: { enabled: true }, // Auto-generates a key
  },
});

// Or provide your own key
const db = new Database({
  security: {
    encryption: {
      enabled: true,
      key: Buffer.alloc(32).toString('base64'), // Your 256-bit key in base64
    },
  },
});

Data is transparently encrypted on write and decrypted on read.

Compression

Transparent compression for stored values:

const db = new Database({
  security: {
    compression: {
      enabled: true,
      algorithm: 'gzip', // 'gzip' | 'brotli' | 'deflate'
    },
  },
});

Indexes

// Create indexes
db.createIndex({ name: 'age-idx', field: 'age', type: 'btree' });
db.createIndex({ name: 'email-idx', field: 'email', type: 'hash' });
db.createIndex({ name: 'id-idx', field: 'id', type: 'unique' });

// List index stats
const stats = db.getIndexManager().stats();

// Drop an index
db.dropIndex('age-idx');

| Index Type | Use Case | |---|---| | btree | Range queries, sorting | | hash | Exact-match lookups | | unique | Enforce uniqueness |

Snapshots

// Create snapshot
const name = await db.createSnapshot('before-migration');

// Restore snapshot
await db.restoreSnapshot('before-migration');

Migrations

import { createMigration } from 'nywion.db';

const migration1 = createMigration(1, 'add-email-field', async (ctx) => {
  await ctx.addField('users', 'email', '');
});

const migration2 = createMigration(2, 'rename-users-to-members', async (ctx) => {
  await ctx.renameTable('users', 'members');
});

const runner = db.getMigrations();
runner.register(migration1, migration2);

// Run all pending migrations
const count = await db.runMigrations();

// Check status
runner.getPending();  // Unapplied migrations
runner.getApplied();  // Applied migrations

Migration Context

The MigrationContext provides:

| Method | Description | |---|---| | renameTable(from, to) | Rename a table | | dropTable(name) | Drop a table | | addField(table, field, default) | Add field to all records | | removeField(table, field) | Remove field from all records | | migrateData(table, transform) | Transform all records in a table |

Observability

Logger

const logger = db.getLogger();
logger.info('Operation started', { userId: '123' });
logger.warn('Slow query', { duration: 1500 });
logger.error('Failed', { error: 'timeout' });

// In-memory log entries
const entries = logger.getEntries();

Metrics

const metrics = db.getMetrics();

metrics.counter('queries', 1, { table: 'users' });
metrics.gauge('cacheSize', 42);
metrics.histogram('queryDuration', 150);

// Get stats
metrics.snapshot(); // { counters: {...}, gauges: {...}, histograms: {...} }

Tracer

const tracer = db.getTracer();

// Synchronous span
const result = tracer.span('process', () => {
  return doWork();
});

// Async span
const result = await tracer.spanAsync('db-query', async () => {
  return await db.query('users', filter);
});

// Get all spans
const spans = tracer.getSpans();

Servers

HTTP Server

import { HttpServer } from 'nywion.db';

const http = new HttpServer(db, { port: 3000 });
await http.start();

// Endpoints:
// GET  /health
// GET  /tables
// GET  /api/:table/:key
// GET  /api/:table/query?filter=...&sort=...&limit=...&offset=...
// POST /api/:table/:key
// POST /api/:table/query
// DELETE /api/:table/:key

WebSocket Server

import { WsServer } from 'nywion.db';

const ws = new WsServer(db);

// Add client
const client = ws.addClient((msg) => {
  console.log('Response:', msg);
});

// Send messages
ws.handleMessage(client.id, { type: 'get', table: 'users', key: 'user1', id: 'r1' });
ws.handleMessage(client.id, { type: 'set', table: 'users', key: 'user1', value: { name: 'Alice' }, id: 'r2' });
ws.handleMessage(client.id, { type: 'subscribe', table: 'users' });

Utility Functions

Object Utilities

import { deepClone, getPath, setPath, mergeDeep, pick, omit, equals } from 'nywion.db';

const clone = deepClone({ a: { b: 1 } });
const val = getPath({ a: { b: 1 } }, 'a.b');       // 1
setPath(obj, 'a.b', 2);                              // Sets nested value
const merged = mergeDeep({ a: 1 }, { b: 2 });       // { a: 1, b: 2 }
const subset = pick({ a: 1, b: 2, c: 3 }, ['a']);   // { a: 1 }
const rest = omit({ a: 1, b: 2, c: 3 }, ['a']);     // { b: 2, c: 3 }
equals({ a: 1 }, { a: 1 });                          // true

Async Utilities

import { sleep, retry, debounce, throttle, Mutex, Semaphore } from 'nywion.db';

await sleep(1000);

const result = await retry(async () => {
  return await fetchData();
}, 3, 1000); // maxAttempts, delayMs

const debouncedFn = debounce(fn, 300);
const throttledFn = throttle(fn, 100);

// Mutex - ensures mutual exclusion
const mutex = new Mutex();
await mutex.runExclusive(async () => {
  // Only one caller at a time
});

// Semaphore - limits concurrency
const sem = new Semaphore(3); // Max 3 concurrent
await sem.acquire();
try {
  // Do work
} finally {
  sem.release();
}

JSON Utilities

import { safeParse, safeStringify, stableStringify, serialize, deserialize, checksumJson } from 'nywion.db';

safeParse('invalid json');                    // undefined (no throw)
safeStringify({ a: 1 }, true);               // Pretty-printed JSON
stableStringify({ b: 2, a: 1 });             // Sorted keys: {"a":1,"b":2}
checksumJson({ a: 1 });                       // Numeric hash

// Serialization with Date/BigInt/RegExp support
const str = serialize({ date: new Date(), big: BigInt(123) });
const obj = deserialize(str);                 // Restores Date, BigInt, RegExp instances

Path Utilities

import { join, dirname, basename, extname, relative } from 'nywion.db';

join('a', 'b', 'c');           // 'a/b/c'
dirname('/path/to/file.txt');   // '/path/to'
basename('/path/to/file.txt');  // 'file.txt'
extname('file.txt');            // '.txt'
relative('/a/b', '/a/b/c/d');  // 'c/d'

Error Types

All errors extend from DatabaseError:

| Error | Description | |---|---| | DatabaseError | Base error class | | NotFoundError | Key or resource not found | | ValidationError | Schema validation failed | | TransactionError | Transaction failed or timed out | | SchemaError | Schema definition error | | QueryError | Query execution error | | StorageError | Storage adapter error | | EncryptionError | Encryption/decryption failed | | MigrationError | Migration failed | | ConflictError | Concurrency conflict |

API Reference

Exports

// Core
Database

// Storage
MemoryAdapter, FileAdapter, HybridAdapter, WalManager, SnapshotManager

// Query
QueryBuilder, executeQuery, QueryPipeline, planQuery, matchFilter

// Session
Session, ReadOnlyView, TransactionManager, IdentityMap

// Validation
Schema, Validator, rules

// Security
Encryption, Compression, AccessControl

// Observability
Logger, Metrics, Tracer

// Migration
MigrationRunner, createMigration, compareVersions, latestVersion

// Cache
Cache, LruCache, LfuCache

// Index
BTreeIndex, HashIndex, UniqueIndex, IndexManager

// Server
HttpServer, WsServer, createResponse, createError, parseMessage, ErrorCode

// Utilities
deepClone, getPath, setPath, mergeDeep, pick, omit, equals,
join, dirname, basename, extname, relative,
safeParse, safeStringify, stableStringify, serialize, deserialize, checksumJson,
sleep, retry, debounce, throttle, Mutex, Semaphore

// Errors
DatabaseError, NotFoundError, ValidationError, TransactionError,
SchemaError, QueryError, StorageError, EncryptionError,
MigrationError, ConflictError

License

MIT