yq-store

A blazingly fast, zero-dependency key-value store that just works. Built for developers who need reliable data persistence without the complexity.

Table of Contents

• 🚀 Why yq-store? - Discover what makes yq-store special
• ⚡ Quick Start - Get up and running in 30 seconds
• ✨ Key Features - What you get out of the box
• 🏗️ How It Works - The architecture that powers performance
• 📦 Installation - Simple npm install
• 📚 Usage Guide - Learn by example
  • Basic Operations - Store, retrieve, and delete data
  • Time-To-Live (TTL) - Automatic data expiration
  • Event Monitoring - React to store changes
  • Batch Operations - Process multiple operations efficiently
  • Transactions - ACID-compliant data operations
  • Querying & Filtering - Find data with ease
  • Storage Modes - Memory vs persistent storage
  • File-Based Caching with YqCacher - High-performance file adapter
• 🔧 Configuration - Customize for your needs
• 📖 API Reference - Complete method documentation
• ⚡ Performance - Benchmarks and optimization tips
• 🎯 Use Cases - Real-world applications
• 🔍 Troubleshooting - Common issues and solutions
• ❓ FAQ - Frequently asked questions
• 🤝 Contributing - Join the community
• 🆘 Support - Get help when you need it
• 📄 License - MIT licensed

🚀 Why yq-store?

Building applications shouldn't mean wrestling with complex databases or sacrificing performance for simplicity. yq-store gives you the best of both worlds:

• Zero Setup Complexity: No database servers, no configuration files, no headaches
• Production Ready: Built on proven foundations with enterprise-grade reliability
• Developer Friendly: TypeScript-first with intuitive APIs that feel natural
• Performance Focused: Optimized for real-world workloads, not just benchmarks

⚡ Quick Start

Get started in less than a minute:

import { YqStore } from 'yq-store';

// Create your store
const store = await YqStore.create();

// Store some data
await store.set('user:123', {
  name: 'Sarah Chen',
  email: '[email protected]',
  preferences: { theme: 'dark', notifications: true }
});

// Retrieve it later
const user = await store.get('user:123');
console.log(`Welcome back, ${user.name}!`);

// Clean up when done
await store.close();

That's it! Your data is automatically persisted to disk and will survive application restarts.

✨ Key Features

🏎️ Blazing Fast Performance

• 20,000+ reads per second
• Optimized storage backend with concurrent access support
• Intelligent indexing and caching

🛡️ Rock Solid Reliability

• ACID transactions
• Automatic data recovery
• Battle-tested foundation

⏰ Smart Data Management

• Built-in TTL (Time-To-Live) support
• Automatic cleanup and compaction
• Configurable memory limits

🎯 Developer Experience

• Full TypeScript support
• Zero external dependencies
• Intuitive, promise-based API

🔍 Powerful Querying

• Prefix and suffix filtering
• Pagination support
• Batch operations

🏗️ How It Works

yq-store is built on a modern, performance-optimized architecture:

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Your App     │───▶│    yq-store      │───▶│  Storage Layer  │
│                │    │                  │    │                 │
│ TypeScript API │    │ • Event System   │    │ • Persistence   │
│ Promise-based  │    │ • TTL Management │    │ • Transactions  │
│ Type-safe      │    │ • Auto Cleanup   │    │ • Indexing      │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Key Architecture Benefits:

• Concurrent Access: Multiple readers and writers supported
• Automatic Optimization: Background cleanup and performance tuning
• Memory Efficiency: Configurable limits prevent resource bloat
• Event-Driven: Real-time notifications for all store operations

📦 Installation

npm install yq-store

Requirements:

• Node.js 22.15+ or Bun 1.2+
• TypeScript 5.8+ (recommended)

Platform Support:

• ✅ Linux (x64, ARM64)
• ✅ macOS (Intel, Apple Silicon)
• ✅ Windows (x64)

No additional setup required! yq-store works out of the box with zero configuration.

📚 Usage Guide

Basic Operations

The core operations you'll use every day:

import { YqStore } from 'yq-store';

// Create your store
const store = await YqStore.create();

// Store any JSON-serializable data
await store.set('product:laptop-123', {
  name: 'MacBook Pro',
  price: 3999,
  specs: {
    cpu: 'M3 Pro',
    memory: '48GB',
    storage: '1TB SSD'
  },
  inStock: true
});

// Retrieve with full type safety
interface Product {
  name: string;
  price: number;
  specs: Record<string, string>;
  inStock: boolean;
}

const laptop = await store.get<Product>('product:laptop-123');
if (laptop) {
  console.log(`${laptop.name} - $${laptop.price}`);
  console.log(`In stock: ${laptop.inStock ? 'Yes' : 'No'}`);
}

// Check existence without retrieving
const exists = await store.has('product:laptop-123');
console.log(`Product exists: ${exists}`);

// Remove when no longer needed
const wasDeleted = await store.delete('product:laptop-123');
console.log(`Deletion successful: ${wasDeleted}`);

// Get store statistics
const activeCount = await store.getSize('active');
const totalCount = await store.getSize('all');
console.log(`Active entries: ${activeCount}, Total: ${totalCount}`);

Time-To-Live (TTL)

Perfect for sessions, caches, and temporary data:

// Cache API responses for 5 minutes
await store.set('api:weather:london', {
  temperature: 18,
  condition: 'partly-cloudy',
  humidity: 65,
  timestamp: Date.now()
}, 300); // 300 seconds = 5 minutes

// Store user sessions that expire in 24 hours
await store.set('session:user-456', {
  userId: '456',
  loginTime: new Date().toISOString(),
  permissions: ['read', 'write'],
  csrfToken: 'abc123xyz'
}, 86400); // 24 hours

// The data automatically disappears when it expires
setTimeout(async () => {
  const weather = await store.get('api:weather:london');
  console.log(weather); // null - expired and cleaned up
}, 301000); // Check after expiration

Event Monitoring

Stay informed about what's happening in your store:

// Monitor store lifecycle
store.on('ready', () => {
  console.log('🟢 Store is ready for operations');
});

store.on('close', () => {
  console.log('🔴 Store has been closed');
});

// Track data operations
store.on('set', (key, value) => {
  console.log(`📝 Stored: ${key}`);
  // Log to analytics, update UI, etc.
});

store.on('delete', (key) => {
  console.log(`🗑️ Deleted: ${key}`);
});

store.on('expire', (key) => {
  console.log(`⏰ Expired: ${key}`);
});

// Monitor performance and maintenance
store.on('compact:start', () => {
  console.log('🔧 Starting store optimization...');
});

store.on('compact:end', (stats) => {
  console.log(`✅ Optimization complete. Processed ${stats.keysProcessed} keys`);
});

// Handle errors gracefully
store.on('error', (error) => {
  console.error('❌ Store error:', error.message);
  // Send to error tracking service
});

Batch Operations

Process multiple operations efficiently and atomically:

// Import user data in a single atomic operation
await store.batch([
  {
    type: 'put',
    key: 'user:alice',
    value: { name: 'Alice Johnson', role: 'admin' }
  },
  {
    type: 'put',
    key: 'user:bob',
    value: { name: 'Bob Smith', role: 'user' },
    ttlSeconds: 3600 // Bob's account expires in 1 hour
  },
  {
    type: 'put',
    key: 'user:charlie',
    value: { name: 'Charlie Brown', role: 'moderator' }
  },
  {
    type: 'del',
    key: 'user:old-account' // Remove old account
  }
]);

console.log('✅ All users imported successfully!');

Transactions

Ensure data consistency with ACID transactions:

// Transfer credits between user accounts
const tx = store.createTransaction();

try {
  // Get current balances
  const sender = await store.get<{credits: number}>('user:alice:credits');
  const receiver = await store.get<{credits: number}>('user:bob:credits');
  
  if (!sender || sender.credits < 100) {
    throw new Error('Insufficient credits');
  }
  
  // Prepare the transfer
  tx.put('user:alice:credits', { credits: sender.credits - 100 });
  tx.put('user:bob:credits', { credits: (receiver?.credits || 0) + 100 });
  tx.put('transaction:log', {
    from: 'alice',
    to: 'bob',
    amount: 100,
    timestamp: Date.now()
  });
  
  // Execute atomically
  await tx.commit();
  console.log('💰 Transfer completed successfully!');
  
} catch (error) {
  // Rollback on any error
  tx.rollback();
  console.error('❌ Transfer failed:', error.message);
}

Querying & Filtering

Find and process data efficiently:

// Find all user records
const userKeys = await store.listKeys({ prefix: 'user:' });
console.log(`Found ${userKeys.length} users`);

// Get paginated results
const firstPage = await store.list<User>({
  prefix: 'user:',
  limit: 10,
  page: 1
});

firstPage.forEach(({ key, value }) => {
  console.log(`${key}: ${value.name} (${value.role})`);
});

// Process all session data
await store.forEach<Session>((key, session) => {
  if (session.lastActivity < Date.now() - 86400000) {
    console.log(`Inactive session: ${key}`);
    // Could mark for cleanup
  }
}, {
  prefix: 'session:',
  limit: 1000 // Process in batches
});

// Find configuration files
const configKeys = await store.listKeys({ suffix: '.config' });
configKeys.forEach(key => {
  console.log(`Config file: ${key}`);
});

Storage Modes

Choose the right storage mode for your use case:

// In-memory store (perfect for testing or temporary data)
const memoryStore = await YqStore.create({
  storage: {
    type: 'memory',
    maxEntries: 50000,
    maxMemory: 100 * 1024 * 1024 // 100MB limit
  }
});

// Persistent store with custom location
const persistentStore = await YqStore.create({
  storage: {
    type: 'persistence',
    persistence: {
      dbDir: './data/stores',
      dbFileName: 'my-application-store'
    }
  },
  compactionInterval: 1800000, // Compact every 30 minutes
  softDelete: true // Enable soft deletes for data recovery
});

File-Based Caching with YqCacher

For applications that need high-performance file-based caching with automatic cleanup and namespace support, YqCacher provides a specialized file adapter:

import { YqCacher } from 'yq-store/file-adapter';

// Create and initialize a file-based cache
const cache = await YqCacher.create({
  cacheDir: './cache',
  ttl: 3600, // 1 hour default TTL
  evictionInterval: 300000, // Clean expired files every 5 minutes
  maxItems: 10000 // Maximum items before LRU eviction
});

// Store data with automatic file management
await cache.set('user:profile:123', {
  name: 'John Doe',
  email: '[email protected]',
  lastLogin: new Date().toISOString()
}, 7200); // 2 hours TTL

// Use namespaces for organized storage
await cache.set('api:weather', weatherData, 300, 'london');
await cache.set('api:weather', weatherData, 300, 'paris');

// Retrieve with type safety and namespace support
const profile = await cache.get<UserProfile>('user:profile:123');
const londonWeather = await cache.get('api:weather', 'london');

// Method overloading: access data with or without namespaces
const globalData = await cache.get('shared:config'); // Global scope
const namespacedData = await cache.get('shared:config', 'production'); // Namespaced scope

// Check existence with namespace overloading
const existsGlobal = await cache.has('user:profile:123'); // Global scope
const existsNamespaced = await cache.has('user:profile:123', 'tenant-a'); // Namespaced scope

// Delete operations support namespace overloading
await cache.delete('temp:data'); // Delete from global scope
await cache.delete('temp:data', 'session-123'); // Delete from specific namespace

const stats = await cache.getStats();
console.log(`Cache: ${stats.totalEntries} entries, Hot cache: ${stats.hotCache.size}/${stats.hotCache.maxSize}`);

// Clean up when done
await cache.close();

YqCacher Features

• 🗂️ Namespace Support: Organize cache files into logical groups with complete data isolation
• 🔄 Method Overloading: Access data with or without namespaces using the same intuitive API
• ⚡ Blazing Fast: Highly optimized for maximum throughput
• 🔥 Hot Cache: In-memory cache for frequently accessed entries (configurable size)
• 🧹 Automatic Cleanup: Expired files are automatically removed
• 📊 Rich Statistics: Track entry counts, hot cache stats, and access patterns
• 🔒 Data Integrity: Atomic operations prevent corruption
• 🎯 Type Safety: Full TypeScript support with generics and method overloads
• 🚀 Non-Blocking: Doesn't prevent process exit by default

YqCacher Configuration

const cache = await YqCacher.create({
  // Storage location
  cacheDir: './cache',

  // TTL settings
  ttl: 3600, // Default expiration in seconds

  // Maintenance
  evictionInterval: 300000, // Cleanup frequency in milliseconds
  maxItems: 1000000, // Maximum items before LRU eviction

  // Performance tuning
  hotCacheSize: 1000, // In-memory cache size (0 to disable)
  eviction: true, // Enable LRU eviction

  // Process behavior
  keepAlive: false, // Don't block process exit (default)

  // Security
  encryptNamespace: false, // Encrypt namespace directory names

  // File organization
  softDelete: true, // Enable soft delete mode
});

YqCacher Configuration Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | cacheDir | string | './cache' | Directory for cache files and metadata | | ttl | number | 0 | Default TTL in seconds (0 = no expiration) | | maxItems | number | 1000000 | Maximum items before LRU eviction | | hotCacheSize | number | 1000 | In-memory LRU cache size (0 to disable) | | eviction | boolean | false | Enable automatic LRU eviction | | evictionInterval | number | 60000 | Eviction check interval in ms | | keepAlive | boolean | false | Keep process alive while cache is open | | softDelete | boolean | true | Mark deleted instead of removing | | encryptNamespace | boolean | false | Hash namespace directory names | | debug | boolean | false | Enable debug logging |

YqCacher Performance

YqCacher is optimized for extreme performance:

Expected throughput (operations per second):

| Operation | Cold (disk) | Hot (in-memory) | Scale | |-----------|-------------|-----------------|-------| | get() | 20-50K | 200K-1M | 10K-1M keys | | set() | 10-30K | 10-30K | 10K-1M keys | | has() | 50-100K | 1M+ | 10K-1M keys | | batch() | 50-100K | 50-100K | 100 ops/batch |

Scaling recommendations:

| Total Keys | Hot Cache Size | Memory Usage | |------------|----------------|--------------| | 10K | 1000 | ~2-4MB | | 100K | 5000 | ~20-40MB | | 1M | 10000 | ~200-400MB |

When to Use YqCacher

Choose YqCacher when you need:

• File-based caching for better disk space management
• Namespace organization for complex applications
• High-performance caching without database overhead
• Automatic cleanup of expired cache files
• Direct file access for external tools

Choose YqStore when you need:

• Complex querying and filtering capabilities
• ACID transactions and data consistency
• Event monitoring and real-time updates
• Advanced indexing and search features

🔧 Configuration

Customize yq-store to fit your specific needs:

const store = await YqStore.create({
  // Automatic maintenance
  compactionInterval: 3600000, // Compact every hour (0 = disabled)
  ttl: 0, // Default TTL in seconds (0 = no expiration)
  softDelete: true, // Enable soft deletes for data recovery
  
  // Storage configuration
  storage: {
    type: 'persistence', // 'memory' or 'persistence'
    maxEntries: 1000000, // Maximum number of entries
    maxMemory: 500 * 1024 * 1024, // 500MB memory limit
    
    persistence: {
      dbDir: './data', // Where to store the database
      dbFileName: 'app-store', // Custom database name
    },
    
    // LRU eviction settings
    eviction: true, // Enable automatic eviction
    evictionInterval: 60000, // Check every minute
  }
});

Configuration Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | compactionInterval | number | 3600000 | Auto-compaction interval in milliseconds (0 = disabled) | | ttl | number | 0 | Default TTL for entries in seconds (0 = no expiration) | | softDelete | boolean | true | Enable soft delete mode for data recovery | | storage.type | 'memory' \| 'persistence' | 'persistence' | Storage mode | | storage.maxEntries | number | 1000000 | Maximum number of entries before eviction | | storage.maxMemory | number | undefined | Memory limit in bytes | | storage.persistence.dbDir | string | os.tmpdir() | Directory for database files | | storage.persistence.dbFileName | string | auto-generated | Custom database filename |

📖 API Reference

Core Methods

YqStore.create(options?: KvStoreOptions): Promise<YqStore>

Creates and initializes a new store instance.

has(key: string): Promise<boolean>

Checks if a key exists and hasn't expired.

get<T>(key: string): Promise<T | null>

Retrieves a value by key with full type safety.

set<T>(key: string, value: T, ttlSeconds?: number): Promise<void>

Stores a value with optional TTL expiration.

delete(key: string): Promise<boolean>

Deletes a key and returns whether it existed.

getSize(): Promise<number>

Gets the total count of active entries in the store.

getSize(type: 'active' | 'deleted' | 'expired' | 'all'): Promise<number>

Gets the count of entries by type.

getSize(options: {prefix: string} | {suffix: string}): Promise<number>

Gets the count of entries matching the filter.

Query Methods

listKeys(): Promise<string[]>

Lists all keys in the store.

listKeys(options: {prefix?: string, suffix?: string}): Promise<string[]>

Lists keys matching the specified filters.

list<T>(options?: ListOptions): Promise<Array<{key: string, value: T}>>

Retrieves key-value pairs with pagination support.

interface ListOptions {
  prefix?: string;
  suffix?: string;
  limit?: number;
  page?: number;
}

forEach<T>(callback: (key: string, value: T) => void, options?: ForEachOptions): Promise<void>

Iterates over key-value pairs matching criteria.

interface ForEachOptions {
  prefix?: string;
  suffix?: string;
  limit?: number;
}

Batch Operations

batch(operations: BatchOperation[]): Promise<void>

Executes multiple operations atomically.

interface BatchOperation {
  type: 'put' | 'del';
  key: string;
  value?: any;
  ttlSeconds?: number;
}

createTransaction(): Transaction

Creates a new transaction for ACID operations.

interface Transaction {
  put<T>(key: string, value: T, ttlSeconds?: number): void;
  del(key: string): void;
  commit(): Promise<void>;
  rollback(): void;
}

Maintenance

compact(): Promise<void>

Manually triggers store compaction and optimization.

clearSoftDeletedEntries(): Promise<number>

Permanently removes soft-deleted entries.

clearExpiredEntries(): Promise<number>

Removes expired entries and returns count.

close(): Promise<void>

Closes the store and releases all resources.

Events

on<E>(event: E, listener: KvStoreEvents[E]): this

Registers an event listener.

off<E>(event: E, listener: KvStoreEvents[E]): this

Removes an event listener.

Available Events

• ready - Store is initialized and ready
• db:ready - Database connection established
• set - Key-value pair was stored
• delete - Key was deleted
• expire - Key expired due to TTL
• evict - Entries evicted due to memory limits
• compact:start - Compaction started
• compact:end - Compaction completed
• error - Error occurred
• close - Store was closed

⚡ Performance

yq-store is optimized for real-world performance:

Benchmark Results

Tested on MacBook Pro M3 with 48GB RAM

| Operation | Throughput | Notes | |-----------|------------|-------| | Sequential Writes | ~4,400 ops/sec | Single key operations | | Batch Writes | ~8,600 ops/sec | 100 operations per batch | | Sequential Reads | ~30,000 ops/sec | Cache-friendly access | | Random Reads | ~32,000 ops/sec | Real-world mixed access |

Performance Tips

1. Use Batch Operations: Group multiple writes for better throughput
2. Configure Memory Limits: Set appropriate maxMemory for your use case
3. Choose Storage Mode: Use memory mode for temporary data
4. Monitor Events: Track performance with timing events
5. Tune Compaction: Adjust interval based on write patterns

Memory Usage

• Base overhead: ~2-5MB for the store instance
• Per entry: ~100-200 bytes depending on key/value size
• Configurable limits: Set maxMemory to prevent unbounded growth
• Automatic cleanup: LRU eviction prevents memory bloat

🎯 Use Cases

yq-store excels in these scenarios:

🌐 Web Applications

// Session management
await store.set(`session:${sessionId}`, userSession, 3600);

// Feature flags
await store.set('feature:new-ui', { enabled: true, rollout: 0.1 });

// User preferences
await store.set(`prefs:${userId}`, { theme: 'dark', lang: 'en' });

🚀 Microservices

// Service configuration
await store.set('config:api-keys', { stripe: 'sk_...', sendgrid: 'SG...' });

// Circuit breaker state
await store.set('circuit:payment-service', { state: 'closed', failures: 0 });

// Rate limiting
await store.set(`rate:${clientId}`, { requests: 1, window: Date.now() }, 60);

📱 Desktop Applications

// Application state
await store.set('app:window-state', { x: 100, y: 100, width: 800, height: 600 });

// User data
await store.set('user:recent-files', ['/path/to/file1.txt', '/path/to/file2.txt']);

// Plugin configuration
await store.set('plugins:enabled', ['spell-check', 'auto-save', 'git-integration']);

🧪 Development & Testing

// Test fixtures
const testStore = await YqStore.create({ storage: { type: 'memory' } });
await testStore.set('test:user', mockUser);

// Development cache
await store.set('dev:api-response', cachedResponse, 300);

🔍 Troubleshooting

Common Issues

Store fails to initialize

// ❌ Problem
const store = new YqStore(); // Don't use constructor directly

// ✅ Solution
const store = await YqStore.create(); // Always use create() method

Memory usage growing unbounded

// ✅ Solution: Set memory limits
const store = await YqStore.create({
  storage: {
    maxEntries: 100000,
    maxMemory: 100 * 1024 * 1024, // 100MB limit
    eviction: true
  }
});

Poor write performance

// ❌ Problem: Individual operations
for (const item of items) {
  await store.set(item.key, item.value);
}

// ✅ Solution: Use batch operations
const operations = items.map(item => ({
  type: 'put' as const,
  key: item.key,
  value: item.value
}));
await store.batch(operations);

Database locks or corruption

// ✅ Solution: Always close the store properly
process.on('SIGINT', async () => {
  await store.close();
  process.exit(0);
});

Error Handling

// Proper error handling
store.on('error', (error) => {
  console.error('Store error:', error);
  // Implement your error recovery strategy
});

try {
  await store.set('key', 'value');
} catch (error) {
  console.error('Failed to set value:', error);
}

Performance Issues

• Slow reads: Increase memory limits or use memory storage mode
• Slow writes: Use batch operations for bulk inserts
• High memory usage: Enable LRU eviction and set appropriate limits
• Storage growing: Adjust compaction interval or manually trigger compaction

❓ FAQ

General Questions

Q: Is yq-store production ready? A: Yes, yq-store is built on battle-tested foundations and includes comprehensive error handling, transactions, and recovery mechanisms.

Q: Can I use yq-store in serverless environments? A: Yes, yq-store works well in serverless environments. Use memory storage mode for better cold start performance.

Q: Does yq-store support clustering or replication? A: yq-store is designed as a single-instance embedded database. For distributed scenarios, consider using it as a local cache with external synchronization.

Technical Questions

Q: What happens if my application crashes? A: All committed data is safely persisted to disk. Soft-deleted entries are preserved for recovery.

Q: Can I access the data from multiple processes? A: No, yq-store is designed for single-process access. Multiple instances pointing to the same database file will cause conflicts.

Q: How do I migrate data between versions? A: yq-store handles schema migrations automatically. For major version upgrades, consult the migration guide.

Q: Is there a size limit for values? A: No hard limit, but performance is optimized for values under 1MB. For larger data, consider using YqCacher.

Performance Questions

Q: How does yq-store compare to Redis? A: yq-store offers simpler deployment (no server required) with comparable performance for single-instance scenarios.

Q: Can I tune performance for my specific use case? A: Yes, adjust maxMemory, compactionInterval, and storage mode based on your read/write patterns.

🤝 Contributing

We love contributions! Here's how you can help:

1. 🐛 Report Bugs: Open an issue with reproduction steps
2. 💡 Suggest Features: Share your ideas for improvements
3. 📝 Improve Docs: Help make our documentation even better
4. 🔧 Submit PRs: Fix bugs or add features

Development Setup

# Clone the repository
git clone https://github.com/yuniqsolutions/yq-store.git
cd yq-store

# Install dependencies
npm install

# Run tests
npm test

# Build the project
npm run build

Contributing Guidelines

• Follow TypeScript best practices
• Add tests for new features
• Update documentation for API changes
• Use conventional commit messages

🆘 Support

Need help? We're here for you:

• 📖 Documentation: You're reading it! Check the API Reference
• 🐛 Bug Reports: GitHub Issues
• 💬 Discussions: GitHub Discussions
• 📧 Email: [email protected]

Enterprise Support

Looking for enterprise features or professional support? Contact us at [email protected]

📄 License

MIT License - see the LICENSE file for details.

Built with ❤️ by Yuniq Solutions Tech

Star us on GitHub if yq-store helps you build amazing applications!