💾 jsondatabase

Lightweight, Concurrency-Controlled JSON Persistence

Built by: Pease Ernest / Ernest Tech House Co-operation

📋 Table of Contents

1. Overview
2. Installation
3. Quick Start
4. Complete API Reference
5. Configuration Guide
6. Usage Examples
7. Performance & Limitations
8. Architecture
9. Best Practices
10. Troubleshooting

🚀 Overview

jsondatabase is a specialized, embedded JSON file store designed for small to medium-sized projects that require stability and predictable I/O throughput.

What Makes It Different?

Unlike heavyweight databases (SQLite, MongoDB) which prioritize complex querying and indexing, jsondatabase trades those features for:

• ✅ Simple implementation - Zero learning curve
• ✅ Absolute control over file I/O concurrency
• ✅ Zero dependencies - Pure Node.js
• ✅ Human-readable storage - Plain JSON files
• ✅ 50 Worker threads - Massive parallel processing capability
• ✅ Automatic backups - Never lose data

Perfect For:

• 🎯 Indie developers building MVPs
• 🎯 Small serverless functions
• 🎯 Utility scripts and automation
• 🎯 Config/settings storage
• 🎯 Cache systems
• 🎯 Data collection services
• 🎯 Applications where database complexity is overkill

Not Recommended For:

• ❌ Large datasets (>50MB)
• ❌ Complex queries with joins
• ❌ High-frequency trading systems
• ❌ Systems requiring ACID transactions
• ❌ Real-time collaborative editing

📦 Installation

NPM Installation

npm install jsondatabase

From Source

git clone https://github.com/ernest-tech-house-co-operation/jsondatabase.git
cd jsondatabase
npm install

Requirements

• Node.js >= 14.0.0
• ES Modules support

🛠️ Quick Start

Basic Example

import jsondatabase from 'jsondatabase';

// 1. Initialize database
const db = jsondatabase({
  name: 'mydata.json',      // Database filename
  dataid: 'MyApp',          // Root key in JSON
  databackup: true,         // Enable auto-backup
  writing: 2,               // Max 2 writes/sec
  reading: 6                // Max 6 reads/sec
});

// 2. Add data (async/await required!)
await db.add('user_ernest', {
  name: 'Ernest P.',
  email: '[email protected]',
  balance: 600000000000000
});

// 3. Read data
const user = await db.get('user_ernest');
console.log(user.name); // "Ernest P."

// 4. Update nested fields
await db.setPath('user_ernest', 'balance', 1000000000000000);

// 5. Check existence
if (await db.has('user_ernest')) {
  console.log('User exists!');
}

// 6. Delete data
await db.delete('old_record');

// 7. Cleanup (important!)
await db.shutdown();

⚠️ Critical: Always Use await

ALL database operations return Promises! You MUST use await:

// ❌ WRONG - Won't work!
const user = db.get('user_123');
console.log(user.name); // undefined (Promise not resolved)

// ✅ CORRECT
const user = await db.get('user_123');
console.log(user.name); // Works!

📚 Complete API Reference

Initialization

jsondatabase(config)

Creates a new database instance.

Parameters:

| Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | name | string | ✅ Yes | - | Database filename (e.g., 'data.json') | | dataid | string | ✅ Yes | - | Root key for data structure | | databackup | boolean | No | true | Enable automatic backups | | writing | number | No | 2 | Max write ops/sec (1-50) | | reading | number | No | 6 | Max read ops/sec (1-50) | | logLevel | string | No | 'info' | Logging: silent/error/warn/info/debug |

Example:

const db = jsondatabase({
  name: 'userdata.json',
  dataid: 'UserDatabase',
  databackup: true,
  writing: 5,
  reading: 20,
  logLevel: 'debug'
});

Core Operations

add(key, value) → Promise<boolean>

Add or update a key-value pair.

Parameters:

• key (string): Unique identifier
• value (any): JSON-serializable value

Returns: true on success

Example:

// Add simple value
await db.add('setting_theme', 'dark');

// Add object
await db.add('user_123', {
  name: 'Alice',
  age: 30,
  interests: ['coding', 'music']
});

// Add array
await db.add('scores', [100, 95, 88]);

// Add number
await db.add('counter', 42);

get(key) → Promise<any>

Retrieve a value by key.

Parameters:

• key (string): Key to retrieve

Returns: The stored value, or undefined if key doesn't exist

Example:

const user = await db.get('user_123');
console.log(user.name); // 'Alice'

const missing = await db.get('nonexistent');
console.log(missing); // undefined

setPath(key, path, value) → Promise<boolean>

Update a nested field using dot notation.

Parameters:

• key (string): Key containing the object
• path (string): Dot-separated path (e.g., 'user.profile.age')
• value (any): Value to set

Returns: true on success

Throws: Error if key doesn't exist

Example:

// Setup initial data
await db.add('user_123', {
  profile: {
    name: 'Alice',
    address: {
      city: 'New York'
    }
  },
  settings: {
    notifications: true
  }
});

// Update nested field
await db.setPath('user_123', 'profile.address.city', 'Los Angeles');

// Create new nested path
await db.setPath('user_123', 'profile.bio.description', 'Software engineer');

// Update deep nested value
await db.setPath('user_123', 'settings.notifications', false);

// Verify changes
const user = await db.get('user_123');
console.log(user.profile.address.city); // 'Los Angeles'

delete(key) → Promise<boolean>

Delete a key from the database.

Parameters:

• key (string): Key to delete

Returns: true if key existed, false if it didn't

Example:

// Delete existing key
const deleted = await db.delete('user_123');
console.log(deleted); // true

// Try to delete non-existent key
const notFound = await db.delete('nonexistent');
console.log(notFound); // false

has(key) → Promise<boolean>

Check if a key exists.

Parameters:

• key (string): Key to check

Returns: true if exists, false otherwise

Example:

if (await db.has('user_123')) {
  console.log('User exists');
  const user = await db.get('user_123');
} else {
  console.log('User not found');
}

getAllKeys() → Promise<string[]>

Get array of all keys in the database.

Returns: Array of key strings

Example:

const keys = await db.getAllKeys();
console.log(keys); // ['user_123', 'user_456', 'setting_theme']

// Process all keys
for (const key of keys) {
  const value = await db.get(key);
  console.log(`${key}:`, value);
}

getAll() → Promise<Object>

Get a copy of all data in the database.

Returns: Object containing all key-value pairs

Example:

const allData = await db.getAll();
console.log(allData);
// {
//   user_123: { name: 'Alice', age: 30 },
//   user_456: { name: 'Bob', age: 25 },
//   setting_theme: 'dark'
// }

// Safe to modify (it's a copy)
allData.user_123.name = 'Changed';
// Original database unchanged

clear() → Promise<boolean>

Remove all data from the database.

Returns: true on success

Example:

// Clear everything
await db.clear();

// Verify
const keys = await db.getAllKeys();
console.log(keys.length); // 0

Utility Methods

getStats() → Object

Get current database statistics (synchronous).

Returns: Statistics object

Example:

const stats = db.getStats();
console.log(stats);
// {
//   queueSize: { read: 2, write: 1, total: 3 },
//   activeWorkers: 5,
//   availableWorkers: 45,
//   totalOperations: {
//     totalReads: 150,
//     totalWrites: 75,
//     totalOperations: 225
//   }
// }

shutdown() → Promise<void>

Gracefully shutdown the database.

Important: Always call before exiting your application!

Example:

// At application exit
process.on('SIGINT', async () => {
  console.log('Shutting down...');
  await db.shutdown();
  process.exit(0);
});

⚙️ Configuration Guide

Performance Tuning

// Low traffic application
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  writing: 2,   // 2 writes per second
  reading: 6    // 6 reads per second
});

// Medium traffic application
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  writing: 10,  // 10 writes per second
  reading: 30   // 30 reads per second
});

// High traffic application (max)
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  writing: 50,  // 50 writes per second (maximum)
  reading: 50   // 50 reads per second (maximum)
});

Logging Levels

// Production: Silent (no logs)
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  logLevel: 'silent'
});

// Production: Errors only
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  logLevel: 'error'
});

// Development: Info logging
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  logLevel: 'info'
});

// Debugging: Verbose output
const db = jsondatabase({
  name: 'data.json',
  dataid: 'App',
  logLevel: 'debug'
});

💡 Usage Examples

Example 1: User Management System

import jsondatabase from 'jsondatabase';

const db = jsondatabase({
  name: 'users.json',
  dataid: 'UserSystem',
  writing: 5,
  reading: 20
});

// Register new user
async function registerUser(username, email, password) {
  // Check if user exists
  if (await db.has(`user_${username}`)) {
    throw new Error('Username already taken');
  }
  
  // Create user
  await db.add(`user_${username}`, {
    username,
    email,
    password, // In production, hash this!
    createdAt: new Date().toISOString(),
    profile: {
      bio: '',
      avatar: null
    },
    settings: {
      notifications: true,
      theme: 'light'
    }
  });
  
  console.log(`User ${username} registered`);
}

// Update user profile
async function updateProfile(username, bio, avatar) {
  await db.setPath(`user_${username}`, 'profile.bio', bio);
  await db.setPath(`user_${username}`, 'profile.avatar', avatar);
  console.log(`Profile updated for ${username}`);
}

// Get user settings
async function getUserSettings(username) {
  const user = await db.get(`user_${username}`);
  return user?.settings;
}

// Usage
await registerUser('alice', '[email protected]', 'pass123');
await updateProfile('alice', 'Software engineer', 'avatar.jpg');
const settings = await getUserSettings('alice');
console.log(settings); // { notifications: true, theme: 'light' }

await db.shutdown();

Example 2: Shopping Cart System

import jsondatabase from 'jsondatabase';

const db = jsondatabase({
  name: 'shop.json',
  dataid: 'ShopData'
});

// Add product to catalog
async function addProduct(id, name, price, stock) {
  await db.add(`product_${id}`, {
    name,
    price,
    stock,
    createdAt: new Date().toISOString()
  });
}

// Create shopping cart
async function createCart(userId) {
  await db.add(`cart_${userId}`, {
    items: [],
    total: 0,
    createdAt: new Date().toISOString()
  });
}

// Add item to cart
async function addToCart(userId, productId, quantity) {
  const cart = await db.get(`cart_${userId}`);
  const product = await db.get(`product_${productId}`);
  
  if (!product) {
    throw new Error('Product not found');
  }
  
  if (product.stock < quantity) {
    throw new Error('Insufficient stock');
  }
  
  // Update cart
  cart.items.push({
    productId,
    name: product.name,
    price: product.price,
    quantity
  });
  
  cart.total += product.price * quantity;
  
  await db.add(`cart_${userId}`, cart);
  
  // Update stock
  await db.setPath(`product_${productId}`, 'stock', product.stock - quantity);
}

// Get cart summary
async function getCartSummary(userId) {
  const cart = await db.get(`cart_${userId}`);
  return {
    itemCount: cart.items.length,
    total: cart.total,
    items: cart.items
  };
}

// Usage
await addProduct(1, 'Laptop', 999.99, 10);
await addProduct(2, 'Mouse', 29.99, 50);

await createCart('user_123');
await addToCart('user_123', 1, 1);
await addToCart('user_123', 2, 2);

const summary = await getCartSummary('user_123');
console.log(summary);
// {
//   itemCount: 2,
//   total: 1059.97,
//   items: [...]
// }

await db.shutdown();

Example 3: Configuration Manager

import jsondatabase from 'jsondatabase';

class ConfigManager {
  constructor() {
    this.db = jsondatabase({
      name: 'config.json',
      dataid: 'AppConfig',
      writing: 2,
      reading: 10
    });
  }
  
  async init() {
    // Set default config if not exists
    if (!(await this.db.has('app_settings'))) {
      await this.db.add('app_settings', {
        server: {
          port: 3000,
          host: 'localhost'
        },
        database: {
          connectionString: 'mongodb://localhost:27017',
          poolSize: 10
        },
        features: {
          enableCache: true,
          enableLogging: true
        }
      });
    }
  }
  
  async get(path) {
    const config = await this.db.get('app_settings');
    return this._getNestedValue(config, path);
  }
  
  async set(path, value) {
    await this.db.setPath('app_settings', path, value);
  }
  
  async getAll() {
    return await this.db.get('app_settings');
  }
  
  _getNestedValue(obj, path) {
    return path.split('.').reduce((current, key) => current?.[key], obj);
  }
  
  async close() {
    await this.db.shutdown();
  }
}

// Usage
const config = new ConfigManager();
await config.init();

// Get specific config
const port = await config.get('server.port');
console.log(port); // 3000

// Update config
await config.set('server.port', 8080);
await config.set('features.enableCache', false);

// Get all config
const allConfig = await config.getAll();
console.log(allConfig);

await config.close();

Example 4: Task Queue System

import jsondatabase from 'jsondatabase';

class TaskQueue {
  constructor() {
    this.db = jsondatabase({
      name: 'tasks.json',
      dataid: 'TaskQueue',
      writing: 10,
      reading: 20
    });
  }
  
  async addTask(taskId, data) {
    await this.db.add(`task_${taskId}`, {
      id: taskId,
      status: 'pending',
      data,
      createdAt: new Date().toISOString(),
      startedAt: null,
      completedAt: null,
      error: null
    });
  }
  
  async startTask(taskId) {
    await this.db.setPath(`task_${taskId}`, 'status', 'running');
    await this.db.setPath(`task_${taskId}`, 'startedAt', new Date().toISOString());
  }
  
  async completeTask(taskId, result) {
    await this.db.setPath(`task_${taskId}`, 'status', 'completed');
    await this.db.setPath(`task_${taskId}`, 'completedAt', new Date().toISOString());
    await this.db.setPath(`task_${taskId}`, 'result', result);
  }
  
  async failTask(taskId, error) {
    await this.db.setPath(`task_${taskId}`, 'status', 'failed');
    await this.db.setPath(`task_${taskId}`, 'error', error);
  }
  
  async getPendingTasks() {
    const keys = await this.db.getAllKeys();
    const tasks = [];
    
    for (const key of keys) {
      const task = await this.db.get(key);
      if (task.status === 'pending') {
        tasks.push(task);
      }
    }
    
    return tasks;
  }
  
  async getTaskStatus(taskId) {
    const task = await this.db.get(`task_${taskId}`);
    return task?.status;
  }
  
  async close() {
    await this.db.shutdown();
  }
}

// Usage
const queue = new TaskQueue();

// Add tasks
await queue.addTask('task_1', { action: 'send_email', to: '[email protected]' });
await queue.addTask('task_2', { action: 'process_image', file: 'photo.jpg' });

// Process task
await queue.startTask('task_1');
// ... do work ...
await queue.completeTask('task_1', { sent: true });

// Check status
const status = await queue.getTaskStatus('task_1');
console.log(status); // 'completed'

// Get pending tasks
const pending = await queue.getPendingTasks();
console.log(pending.length); // 1

await queue.close();

🚨 Performance & Limitations

File Size Guidelines

| File Size | Status | Performance | Recommendation | |-----------|--------|-------------|----------------| | < 1MB | ✅ Excellent | Fast | Ideal for most use cases | | 1-10MB | ✅ Good | Normal | Recommended maximum | | 10-50MB | ⚠️ Caution | Degraded | Consider alternatives | | > 50MB | ❌ Not Recommended | Severe slowdown | Use real database |

Performance Benchmarks

Typical performance on modern hardware:

Write Operations:    50-100 ops/sec
Read Operations:     200-400 ops/sec
Path Updates:        40-80 ops/sec
Mixed Operations:    100-200 ops/sec

Concurrency

• 50 Worker Threads: Massive parallel processing
• Rate Limiting: Prevents file system overload
• Round-Robin Scheduling: Fair allocation between reads/writes
• Queue Management: Automatic backpressure handling

Known Limitations

1. Not Atomic: No transactional guarantees
2. No Rollback: Manual backup restore required
3. No Querying: Direct key lookup only (no .find(), .filter())
4. Memory Usage: Entire file loaded for each operation
5. Large Numbers: Store numbers > 2^53 as strings
6. Node.js Only: Requires fs and worker_threads

🏗️ Architecture

Component Overview

┌─────────────────────────────────────────┐
│         jsondatabase Instance           │
├─────────────────────────────────────────┤
│  - Configuration                        │
│  - Public API Methods                   │
└─────────────────┬───────────────────────┘
                  │
         ┌────────┼────────┐
         │        │        │
         ▼        ▼        ▼
   ┌─────────┐ ┌──────────┐ ┌─────────────┐
   │  File   │ │  Task    │ │   Worker    │
   │ Manager │ │Scheduler │ │    Pool     │
   └─────────┘ └──────────┘ └─────────────┘
       │            │              │
       │            │              │
       ▼            ▼              ▼
   ┌─────────┐ ┌──────────┐ ┌─────────────┐
   │  JSON   │ │Read/Write│ │ 50 Worker   │
   │  File   │ │  Queues  │ │  Threads    │
   └─────────┘ └──────────┘ └─────────────┘

File Manager

• Handles all disk I/O operations
• Creates/validates JSON structure
• Manages backup files
• Provides file size info

Task Scheduler

• Round-robin scheduling (write → read → write)
• Rate limiting (ops per second)
• Queue management
• Statistics tracking

Worker Pool

• 50 independent worker threads
• Automatic worker replacement on failure
• Task distribution
• Timeout handling

🎯 Best Practices

1. Always Use Async/Await

// ❌ BAD - Promises not handled
db.add('key', 'value');
const value = db.get('key');

// ✅ GOOD - Proper async handling
await db.add('key', 'value');
const value = await db.get('key');

2. Always Shutdown Gracefully

// ❌ BAD - Potential data loss
process.exit(0);

// ✅ GOOD - Clean shutdown
await db.shutdown();
process.exit(0);

// ✅ BEST - Handle signals
process.on('SIGINT', async () => {
  await db.shutdown();
  process.exit(0);
});

3. Use Try-Catch for Error Handling

try {
  await db.setPath('user_123', 'profile.name', 'Alice');
} catch (error) {
  if (error.message.includes('does not exist')) {
    // Create user first
    await db.add('user_123', { profile: { name: 'Alice' } });
  } else {
    throw error;
  }
}

4. Batch Related Operations

// ❌ BAD - Many small operations
for (let i = 0; i < 100; i++) {
  await db.add(`item_${i}`, { value: i });
}

// ✅ BETTER - Batch preparation
const items = {};
for (let i = 0; i < 100; i++) {
  items[`item_${i}`] = { value: i };
}
// Then write once (if your use case allows)

5. Check Existence Before Updates

// ❌ BAD - Might throw error
await db.setPath('user_123', 'profile.name', 'Alice');

// ✅ GOOD - Check first
if (await db.has('user_123')) {
  await db.setPath('user_123', 'profile.name', 'Alice');
} else {
  await db.add('user_123', { profile: { name: 'Alice' } });
}

6. Use Meaningful Keys

// ❌ BAD - Unclear naming
await db.add('123', userData);

// ✅ GOOD - Descriptive keys
await db.add('user_123', userData);
await db.add('order_2024_001', orderData);
await db.add('config_app_settings', configData);

7. Store Large Numbers as Strings

// ❌ BAD - Precision loss for large numbers
await db.add('balance', 999999999999999999);

// ✅ GOOD - Store as string
await db.add('balance', '999999999999999999');

🔧 Troubleshooting

Common Issues

Issue: "Database validation failed"

// Cause: Corrupted JSON file
// Solution: Delete file and reinitialize, or restore from backup

import fs from 'fs';
if (fs.existsSync('data.json.backup')) {
  fs.copyFileSync('data.json.backup', 'data.json');
}

Issue: Operations returning undefined

// ❌ Problem: Missing await
const user = db.get('user_123');
console.log(user.name); // undefined - user is a Promise!

// ✅ Solution: Use await
const user = await db.get('user_123');
console.log(user.name); // Works!

Issue: "Key does not exist" error

// ❌ Problem: Trying to update non-existent key
await db.setPath('user_999', 'name', 'Alice'); // Error!

// ✅ Solution: Check existence first
if (await db.has('user_999')) {
  await db.setPath('user_999', 'name', 'Alice');
} else {
  await db.add('user_999', { name: 'Alice' });
}

Issue: Slow performance

// Possible causes:
// 1. File too large (>50MB)
// 2. Too many operations per second
// 3. Disk I/O bottleneck

// Solutions:
// - Split data into multiple databases
// - Increase rate limits
// - Use SSD instead of HDD
// - Consider a real database for large datasets

📝 Additional Resources

Running Tests

npm test

Running Examples

# Demo showcase
npm run demo

# Performance benchmark
npm run benchmark

Project Structure

jsondatabase/
├── src/
│   ├── index.js          # Main entry point
│   ├── worker-pool.js    # 50 Worker thread pool
│   ├── worker.js         # Worker implementation
│   ├── scheduler.js      # Round-robin scheduler
│   ├── file-manager.js   # File I/O manager
│   └── utils/
│       └── logger.js     # Logging utility
├── examples/
│   ├── demo.js           # Feature demo
│   └── benchmark.js      # Performance tests
├── test/
│   └── test.js           # Test suite
├── package.json
└── README.md

Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new features
4. Submit a pull request

License

MIT License - See LICENSE file

Support

• GitHub Issues: Report bugs and request features
• Email: [email protected]

🙏 Acknowledgments

Built with ❤️ for indie developers who need something simple, reliable, and just right for small projects.

Happy coding! 💻

Built by Pease Ernest / Ernest Tech House Co-operation