@fimbul-works/storage

A type-safe, abstract storage system for TypeScript with a unified interface for CRUD operations across multiple backends.

Features

• 🔷 Type-safe — Full TypeScript support with generics
• 🗄️ Multiple Backends — In-memory, file-based, Redis, and layered storage
• 🔌 Custom Serialization — Pluggable adapters for different data formats
• 📊 Efficient Data Access — Stream large datasets or retrieve all keys without loading entries
• ⚠️ Error Handling — Specific error types for duplicate keys and missing entries

Installation

npm install @fimbul-works/storage

For Redis support:

npm install redis

Quick Start

import { createMemoryStorage } from '@fimbul-works/storage';

interface User {
  id: string;
  name: string;
  email: string;
}

const storage = createMemoryStorage<User, 'id'>('id');

await storage.create({ id: '1', name: 'John Doe', email: '[email protected]' });
const user = await storage.get('1');
await storage.update({ id: '1', name: 'John Updated', email: '[email protected]' });

// For small datasets
const allUsers = await storage.getAll();

// For large datasets, use streaming
for await (const user of storage.streamAll()) {
  console.log(user.name);
}

await storage.delete('1');

Storage Backends

In-Memory Storage

Fast storage using JavaScript's Map. Perfect for testing or temporary data.

const storage = createMemoryStorage<User, 'id'>('id');

File-Based Storage

Persistent storage using the filesystem. Each entity is stored as a separate file.

import { createFileStorage } from '@fimbul-works/storage';

const storage = createFileStorage<User, 'id'>('id', { path: './data/users' });
await storage.create({ id: '1', name: 'John', email: '[email protected]' });
// Creates: ./data/users/1.json

Redis Storage

Distributed storage with automatic connection management.

import { createRedisStorage } from '@fimbul-works/storage/redis';

const storage = await createRedisStorage<User, 'id'>('id', {
  url: 'redis://localhost:6379',
  keyPrefix: 'users:',
});

await storage.create({ id: '1', name: 'John', email: '[email protected]' });

// Close connection when done
storage.close();

Layered Storage (Caching)

Combine backends for cache-aside patterns. Layers are ordered top to bottom (fastest first).

import { createLayeredStorage, createMemoryStorage, createFileStorage } from '@fimbul-works/storage';

const cache = createMemoryStorage<User, 'id'>('id');
const persistent = createFileStorage<User, 'id'>('id', { path: './data/users' });
const storage = createLayeredStorage([cache, persistent]);

// Reads check layers top-down (cache first)
const user = await storage.get('1');

// Writes persist to all layers
await storage.create({ id: '2', name: 'Jane', email: '[email protected]' });

All layers must share the same key field, which is automatically determined from the first layer.

Layer behavior:

• exists/get: Check layers top-down, return first match
• create/update: Write to all layers
• delete: Remove from all layers that have the key
• getAll/streamAll/getKeys: Merge all layers (top layer wins for duplicates)

API Reference

All storage implementations implement the Storage<T, K> interface:

| Property/Method | Description | Type/Returns | |----------------|-------------|--------------| | keyField | Read-only field indicating which property is used as the key | K | | exists(key) | Check if entry exists | Promise<boolean> | | create(entry) | Create new entry | Promise<void> | | get(key) | Retrieve entry by key | Promise<T \| null> | | getAll() | Retrieve all entries | Promise<T[]> | | getKeys() | Retrieve all keys | Promise<T[K][]> | | streamAll() | Stream all entries | AsyncIterableIterator<T> | | update(entry) | Update existing entry | Promise<void> | | delete(key) | Delete entry | Promise<void> |

Error Types

• DuplicateKeyError: Thrown when creating an entry with an existing key
• KeyNotFoundError: Thrown when updating/deleting a non-existent entry

Advanced Usage

Streaming Large Datasets

For large datasets, use streamAll() to process entries efficiently without loading everything into memory:

const storage = createFileStorage<User, 'id'>('id', { path: './data/users' });

// Process users one at a time
for await (const user of storage.streamAll()) {
  console.log(`Processing: ${user.name}`);
  // Send to API, perform calculations, etc.
}

// Early termination - stop after finding what you need
for await (const user of storage.streamAll()) {
  if (user.email === '[email protected]') {
    console.log('Found target user!');
    break; // Stops iteration, saves resources
  }
}

Working with Keys

Sometimes you only need the keys without loading the full entries:

const storage = createFileStorage<User, 'id'>('id', { path: './data/users' });

// Get all user IDs
const userIds = await storage.getKeys();
console.log(`Found ${userIds.length} users`);

Key Type Coercion

File and Redis storage store keys as strings, but your application might use numbers or other types. Use keyFromStorage to convert keys back to your application type:

interface User {
  id: number;  // Application uses numbers
  name: string;
}

// File storage with number keys
const storage = createFileStorage<User, 'id'>('id', {
  path: './data/users',
  keyFromStorage: (raw) => Number.parseInt(raw, 10),
});

await storage.create({ id: 123, name: 'John' });
const keys = await storage.getKeys();  // Returns [123] as number[]

// Redis storage with number keys
const redisStorage = await createRedisStorage<User, 'id'>('id', {
  url: 'redis://localhost:6379',
  keyFromStorage: (raw) => Number.parseInt(raw, 10),
});

await redisStorage.create({ id: 456, name: 'Jane' });
const redisKeys = await redisStorage.getKeys();  // Returns [456] as number[]
redisStorage.close();

Custom Serialization

Create custom serialization adapters for different data formats:

import { createFileStorage, type FileAdapter } from '@fimbul-works/storage';

const csvAdapter: FileAdapter<User, 'id'> = {
  encoding: 'utf-8',
  fileName: (key) => `user_${key}.csv`,
  serialize: (user) => `${user.id},${user.name},${user.email}`,
  deserialize: (str) => {
    const [id, name, email] = str.split(',');
    return { id, name, email };
  },
};

const storage = createFileStorage('id', { path: './data/users', adapter: csvAdapter });

Different Key Fields

Use any field as the unique key:

interface Product {
  sku: string;
  name: string;
  price: number;
}

const storage = createMemoryStorage<Product, 'sku'>('sku');
await storage.create({ sku: 'ABC123', name: 'Widget', price: 9.99 });

License

MIT License - See LICENSE file for details.

Built with 📦 by FimbulWorks