@fimbul-works/storage

A type-safe, abstract storage system for TypeScript with a unified interface for CRUD operations across multiple backends.

Features

• 🔷 Type-safe — Full TypeScript support with generics
• 🗄️ Multiple Backends — In-memory, file-based, Redis, and layered storage
• 🔌 Custom Serialization — Pluggable adapters for different data formats
• 📊 Efficient Data Access — Stream large datasets, batch retrieve entries, or list all keys
• 🔔 Real-time Events — Subscribe to document creation, updates, and deletions
• ⚠️ Error Handling — Specific error types for duplicate keys and missing entries

Installation

npm install @fimbul-works/storage

For Redis support:

npm install redis

For YAML serialization support:

npm install yaml

Quick Start

import { createMemoryStorage } from '@fimbul-works/storage';

interface User {
  id: string;
  name: string;
  email: string;
}

const storage = createMemoryStorage<User, 'id'>('id');

await storage.create({ id: '1', name: 'John Doe', email: '[email protected]' });
const user = await storage.get('1');
await storage.update({ id: '1', name: 'John Updated', email: '[email protected]' });

// For small datasets
const allUsers = await storage.getAll();

// For large datasets, use streaming
for await (const user of storage.streamAll()) {
  console.log(user.name);
}

await storage.delete('1');

Storage Backends

In-Memory Storage

Fast storage using JavaScript's Map. Perfect for testing or temporary data.

Basic Usage

import { createMemoryStorage } from '@fimbul-works/storage';

const storage = createMemoryStorage<User, 'id'>('id');

TTL (Time-To-Live) Cache

Configure in-memory storage as a temporary cache with automatic expiration:

// Cache with 60-second TTL
const cache = createMemoryStorage<User, 'id'>('id', { ttl: 60_000 });

await cache.create({ id: '1', name: 'John', email: '[email protected]' });

// Entry exists immediately
const user = await cache.get('1'); // Returns user

// After 60 seconds, entry automatically expires
const expiredUser = await cache.get('1'); // Returns null

TTL Features:

• ⏱️ Automatic Expiration: Entries expire after the configured duration
• 🔄 TTL Reset: Updating an entry resets its TTL
• 🧹 Lazy Cleanup: Expired entries are removed on access
• 🎯 All Operations: Works with exists, get, getAll, getKeys, streamAll, update, delete

// Update resets the TTL
await cache.update({ id: '1', name: 'John Updated', email: '[email protected]' });
// Entry now has a fresh 60-second TTL

File-Based Storage

Persistent storage using the filesystem. Each entity is stored as a separate file.

import { createFileStorage } from '@fimbul-works/storage';

const storage = createFileStorage<User, 'id'>('id', { path: './data/users' });
await storage.create({ id: '1', name: 'John', email: '[email protected]' });
// Creates: ./data/users/1.json

Redis Storage

Distributed storage with automatic connection management.

import { createRedisStorage } from '@fimbul-works/storage/redis';

const storage = await createRedisStorage<User, 'id'>('id', {
  url: 'redis://localhost:6379',
  keyPrefix: 'users:',
});

await storage.create({ id: '1', name: 'John', email: '[email protected]' });

// Close connection when done
storage.close();

Custom Serialization for Redis

Use custom serialization adapters with Redis:

import { createRedisStorage, createYamlSerializationAdapter } from '@fimbul-works/storage/redis';

const yamlRedisStorage = await createRedisStorage<User, 'id'>('id', {
  url: 'redis://localhost:6379',
  keyPrefix: 'users:',
  serializationAdapter: createYamlSerializationAdapter(),
});

await yamlRedisStorage.create({
  id: '1',
  name: 'John Doe',
  email: '[email protected]',
});

yamlRedisStorage.close();

Layered Storage (Caching)

Combine backends for cache-aside patterns. Layers are ordered top to bottom (fastest first).

import { createLayeredStorage, createMemoryStorage, createFileStorage } from '@fimbul-works/storage';

const cache = createMemoryStorage<User, 'id'>('id');
const persistent = createFileStorage<User, 'id'>('id', { path: './data/users' });
const storage = createLayeredStorage([cache, persistent]);

// Reads check layers top-down (cache first)
const user = await storage.get('1');

// Writes persist to all layers
await storage.create({ id: '2', name: 'Jane', email: '[email protected]' });

TTL Cache with Persistent Storage

Combine a time-limited cache with persistent storage for optimal performance:

import { createLayeredStorage, createMemoryStorage, createFileStorage } from '@fimbul-works/storage';

// 60-second in-memory cache
const cache = createMemoryStorage<User, 'id'>('id', { ttl: 60_000 });

// Persistent file storage
const persistent = createFileStorage<User, 'id'>('id', { path: './data/users' });

// Layered storage with cache on top
const storage = createLayeredStorage([cache, persistent]);

// First read loads from persistent storage and caches it
const user1 = await storage.get('1'); // Loads from file, caches in memory

// Subsequent reads within 60 seconds use cache
const user2 = await storage.get('1'); // Returns from cache (fast!)

// After 60 seconds, cache expires but data persists in files
const user3 = await storage.get('1'); // Reloads from file, recaches

All layers must share the same key field, which is automatically determined from the first layer.

Layer behavior:

• exists/get: Check layers top-down, return first match
• getMany: Batch retrieve multiple keys efficiently
• create/update: Write to all layers
• delete: Remove from all layers that have the key
• getAll/streamAll/getKeys: Merge all layers (top layer wins for duplicates)

Reactive Cache Sync (Event Bubbling)

Layered storage automatically keeps upper layers in sync with lower layers. If a lower-level storage (like persistent file storage) emits a change (e.g., from an external file edit), the upper layers (like an in-memory cache) automatically update their state:

const cache = createMemoryStorage<User, 'id'>('id');
const persistent = createFileStorage<User, 'id'>('id', { path: './data/users' });
const storage = createLayeredStorage([cache, persistent]);

// If a file is modified directly on disk, the 'cache' layer
// of the 'storage' instance will automatically update!

API Reference

All storage implementations implement the Storage<T, K> interface:

| Property/Method | Description | Type/Returns | |----------------|-------------|--------------| | keyField | Read-only field indicating which property is used as the key | K | | exists(key) | Check if entry exists | Promise<boolean> | | create(entry) | Create new entry | Promise<void> | | get(key) | Retrieve entry by key | Promise<T \| null> | | getMany(keys) | Retrieve multiple entries by keys | Promise<T[]> | | getAll() | Retrieve all entries | Promise<T[]> | | getKeys() | Retrieve all keys | Promise<T[K][]> | | streamAll() | Stream all entries | AsyncIterableIterator<T> | | update(entry) | Update existing entry | Promise<void> | | delete(key) | Delete entry | Promise<void> | | on(event, cb) | Subscribe to storage events | () => void (cleanup) |

Error Types

• DuplicateKeyError: Thrown when creating an entry with an existing key
• KeyNotFoundError: Thrown when updating/deleting a non-existent entry

Storage Events

All storage implementations support reactive events, allowing you to react to data changes in real-time.

const storage = createMemoryStorage<User, 'id'>('id');

const unsubscribe = storage.on('create', (entry) => {
  console.log('New user created:', entry.name);
});

storage.on('update', (entry) => {
  console.log('User updated:', entry.id);
});

storage.on('delete', (entry) => {
  console.log('User deleted:', entry.id);
});

// Stop listening
unsubscribe();

Event Support

• MemoryStorage: Immediate emission on local changes.
• FileStorage: Integrated with chokidar to detect external filesystem changes.
• LayeredStorage: Proxies top-layer events and bubbles lower-layer changes upwards.
• RedisStorage: ⚠️ Currently unimplemented (API provided for consistency).

[!NOTE] Event support for Redis (via Pub/Sub) is planned for a future release. Currently, attaching a listener to a Redis storage will not result in any callbacks being triggered.

Advanced Usage

Streaming Large Datasets

For large datasets, use streamAll() to process entries efficiently without loading everything into memory:

const storage = createFileStorage<User, 'id'>('id', { path: './data/users' });

// Process users one at a time
for await (const user of storage.streamAll()) {
  console.log(`Processing: ${user.name}`);
  // Send to API, perform calculations, etc.
}

// Early termination - stop after finding what you need
for await (const user of storage.streamAll()) {
  if (user.email === '[email protected]') {
    console.log('Found target user!');
    break; // Stops iteration, saves resources
  }
}

Working with Keys

Sometimes you only need the keys without loading the full entries:

const storage = createFileStorage<User, 'id'>('id', { path: './data/users' });

// Get all user IDs
const userIds = await storage.getKeys();
console.log(`Found ${userIds.length} users`);

Key Type Coercion

File and Redis storage store keys as strings, but your application might use numbers or other types. Use keyFromStorage to convert keys back to your application type:

interface User {
  id: number;  // Application uses numbers
  name: string;
}

// File storage with number keys
const storage = createFileStorage<User, 'id'>('id', {
  path: './data/users',
  keyFromStorage: (raw) => Number.parseInt(raw, 10),
});

await storage.create({ id: 123, name: 'John' });
const keys = await storage.getKeys();  // Returns [123] as number[]

// Redis storage with number keys
const redisStorage = await createRedisStorage<User, 'id'>('id', {
  url: 'redis://localhost:6379',
  keyFromStorage: (raw) => Number.parseInt(raw, 10),
});

await redisStorage.create({ id: 456, name: 'Jane' });
const redisKeys = await redisStorage.getKeys();  // Returns [456] as number[]
redisStorage.close();

Custom Serialization

Create custom serialization adapters for different data formats.

JSON Serialization (Default)

JSON is the default serialization format for both file and Redis storage:

import { createFileStorage, createJsonSerializationAdapter } from '@fimbul-works/storage';

// Default JSON adapter
const storage = createFileStorage<User, 'id'>('id', { path: './data/users' });

// Custom JSON adapter with pretty printing
const prettyJsonStorage = createFileStorage<User, 'id'>('id', {
  path: './data/users',
  adapter: {
    encoding: 'utf-8',
    fileName: (key) => `${key}.json`,
    ...createJsonSerializationAdapter({ space: 2 }),
  },
});

YAML Serialization

Use YAML for human-readable configuration files:

import { createFileStorage, createYamlSerializationAdapter } from '@fimbul-works/storage';

const yamlStorage = createFileStorage<User, 'id'>('id', {
  path: './data/users',
  adapter: {
    encoding: 'utf-8',
    fileName: (key) => `${key}.yaml`,
    ...createYamlSerializationAdapter({ indent: 2 }),
  },
});

await yamlStorage.create({
  id: '1',
  name: 'John Doe',
  email: '[email protected]',
});
// Creates: ./data/users/1.yaml

Custom CSV Serialization

import { createFileStorage, type FileAdapter } from '@fimbul-works/storage';

const csvAdapter: FileAdapter<User, 'id'> = {
  encoding: 'utf-8',
  fileName: (key) => `user_${key}.csv`,
  serialize: (user) => `${user.id},${user.name},${user.email}`,
  deserialize: (str) => {
    const [id, name, email] = str.split(',');
    return { id, name, email };
  },
};

const storage = createFileStorage<User, 'id'>('id', {
  path: './data/users',
  adapter: csvAdapter,
});

Different Key Fields

Use any field as the unique key:

interface Product {
  sku: string;
  name: string;
  price: number;
}

const storage = createMemoryStorage<Product, 'sku'>('sku');
await storage.create({ sku: 'ABC123', name: 'Widget', price: 9.99 });

License

MIT License - See LICENSE file for details.

Built with 📦 by FimbulWorks