hashstorage-ts

Content-addressable version-controlled file storage for TypeScript/Node.js

A Git-inspired storage system providing:

• ✓ Content-based deduplication
• ✓ Full version history with commits
• ✓ Multiple storage backends (local filesystem, cloud storage, memory)
• ✓ Tagging ("named" commits)
• ✓ TypeScript-first with complete type definitions

Installation

npm install @btmnstr/hashstorage-ts

Quick Start

import { HashStorage, LocalFSOperations } from '@btmnstr/hashstorage-ts';

// Initialize storage
const storage = new HashStorage(new LocalFSOperations('/path/to/storage'));

// Create or get repository
const repo = await storage.findOrCreateRepository('my-repo');

// Add files
await repo.addFile('/documents/hello.txt', Buffer.from('Hello!'), 'text/plain');

// Commit changes
const updated = await repo.commit('[email protected]', 'Initial commit');

// Push to storage
await storage.pushRepository(updated);

// List files
const files = await updated.listAllFiles('/');
console.log(files);

Features

Content-Addressable Storage

Files are stored by their SHA-1 hash, providing automatic deduplication:

// Same content = same hash = stored once
await repo.addFile('/doc1.txt', Buffer.from('Hello'));
await repo.addFile('/doc2.txt', Buffer.from('Hello')); // Not stored again

Reading Files

// Read binary file (throws NotFoundError if missing)
const data = await repo.readFile('/documents/image.png');

// Read JSON file (parsed)
const config = await repo.readJsonFile<Config>('/config.json');

// Check if file exists (returns undefined if missing)
const file = await repo.findFile('/maybe.txt');

Version History

Full Git-like commit history:

// Make multiple commits
let repo = await storage.findOrCreateRepository('my-repo');
await repo.addFile('/file.txt', Buffer.from('v1'));
repo = await repo.commit('[email protected]', 'Version 1');

await repo.addFile('/file.txt', Buffer.from('v2'));
repo = await repo.commit('[email protected]', 'Version 2');

// View history
for await (const commit of repo.log()) {
    console.log(commit.message, commit.time);
}

Multiple Storage Backends

Choose your storage backend:

// Local filesystem (uses file:// URLs by default)
new LocalFSOperations('/path/to/storage');

// Local filesystem with custom URL prefix (for web serving)
new LocalFSOperations('/var/www/assets', '/assets');

// In-memory (for testing)
new MemoryFSOperations();

Cloud Storage: See Cloud Storage Examples below for Azure, Google Cloud, AWS S3, Cloudflare R2, and Backblaze B2.

Tagging

Tag important commits:

await storage.addTag(commit, 'v1.0.0');
const tagged = await storage.findExistingRepository('my-repo', 'v1.0.0');

Cloud Storage Examples

HashStorage can work with any cloud storage provider by implementing the Operations interface. The package includes example implementations for common providers:

Azure Blob Storage

See examples/operations-azure.ts for a complete implementation.

import { AzureBlobOperations } from './examples/operations-azure';

const ops = new AzureBlobOperations(
    'https://myaccount.blob.core.windows.net',
    'myaccount',
    'mycontainer',
    'accountkey'
);
const storage = new HashStorage(ops);

Installation: npm install @azure/storage-blob

Google Cloud Storage

See examples/operations-gcs.ts for a complete implementation.

import { GoogleCloudStorageOperations } from './examples/operations-gcs';

const ops = new GoogleCloudStorageOperations('my-bucket-name');
const storage = new HashStorage(ops);

Installation: npm install @google-cloud/storage

Authentication: Set GOOGLE_APPLICATION_CREDENTIALS environment variable to your service account key file.

AWS S3

See examples/operations-s3.ts for a complete implementation.

import { S3Operations } from './examples/operations-s3';

const ops = new S3Operations({
    region: 'us-east-1',
    bucket: 'my-bucket',
    credentials: {
        accessKeyId: 'YOUR_ACCESS_KEY',
        secretAccessKey: 'YOUR_SECRET_KEY',
    },
});
const storage = new HashStorage(ops);

Installation: npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

Cloudflare R2

Cloudflare R2 is S3-compatible. Use the same S3Operations class with a custom endpoint:

const ops = new S3Operations({
    region: 'auto',
    bucket: 'my-bucket',
    endpoint: 'https://ACCOUNT_ID.r2.cloudflarestorage.com',
    credentials: {
        accessKeyId: 'YOUR_R2_ACCESS_KEY_ID',
        secretAccessKey: 'YOUR_R2_SECRET_ACCESS_KEY',
    },
});

Benefits: Free egress, $0.015/GB storage

Backblaze B2

Backblaze B2 is S3-compatible. Use the same S3Operations class with a custom endpoint:

const ops = new S3Operations({
    region: 'us-west-000',
    bucket: 'my-bucket',
    endpoint: 'https://s3.us-west-000.backblazeb2.com',
    credentials: {
        accessKeyId: 'YOUR_B2_KEY_ID',
        secretAccessKey: 'YOUR_B2_APPLICATION_KEY',
    },
});

Benefits: Free egress via CDN partners, low-cost storage

Custom Implementations

To implement your own storage backend, implement the Operations interface:

interface Operations {
    mkdir(dirPath: string): Promise<void>;
    listFiles(dirPath: string): Promise<string[]>;
    readBuffer(filePath: string, position?: number, length?: number): Promise<Buffer>;
    writeBuffer(filePath: string, data: Buffer, mimeType: string): Promise<void>;
    composeAccessURL(objectPath: string): Promise<string>;
}

Note: The example implementations are minimal starting points. For production use, consider adding:

• Error handling and retries
• Connection pooling
• Monitoring and logging
• Rate limiting
• Cost optimization

Logging

HashStorage includes structured logging support with three logger implementations:

• WarningsLogger (default) - Logs only warnings and errors, silent for normal operations
• ConsoleLogger - Logs all messages (debug, info, warn, error) for verbose debugging
• NoOpLogger - Completely silent, discards all log messages

import { HashStorage, WarningsLogger, ConsoleLogger, NoOpLogger } from '@btmnstr/hashstorage-ts';

// Default: WarningsLogger (recommended for production and testing)
const storage = new HashStorage(
    new LocalFSOperations('/path/to/storage')
    // Uses WarningsLogger by default - clean output, shows issues
);

// Verbose logging for development/debugging
const verboseStorage = new HashStorage(
    new LocalFSOperations('/path/to/storage'),
    undefined,
    undefined,
    new ConsoleLogger() // Shows all debug/info/warn/error messages
);

// Completely silent (for specific test scenarios)
const silentStorage = new HashStorage(
    new LocalFSOperations('/path/to/storage'),
    undefined,
    undefined,
    new NoOpLogger() // No output at all
);

Custom Logger

Implement the Logger interface for custom logging:

import { Logger } from '@btmnstr/hashstorage-ts';

class MyCustomLogger implements Logger {
    debug(message: string, context?: Record<string, any>): void {
        // Your debug logging implementation
    }

    info(message: string, context?: Record<string, any>): void {
        // Your info logging implementation
    }

    warn(message: string, context?: Record<string, any>): void {
        // Your warning logging implementation
    }

    error(message: string, context?: Record<string, any>): void {
        // Your error logging implementation
    }
}

const storage = new HashStorage(
    new LocalFSOperations('/path/to/storage'),
    undefined,
    undefined,
    new MyCustomLogger()
);

The logger receives structured context objects that include relevant details like repository IDs, file paths, and error information.

Error Handling

HashStorage uses typed errors for better error handling:

import {
    HashStorageError,
    NotFoundError,
    ValidationError,
    StorageError,
    LockError,
    ConflictError,
} from '@btmnstr/hashstorage-ts';

try {
    const repo = await storage.findExistingRepository('nonexistent');
} catch (error) {
    if (error instanceof NotFoundError) {
        // Handle missing repository
        console.log('Repository not found:', error.context);
    } else if (error instanceof ConflictError) {
        // Handle merge conflicts
        console.log('Conflict detected:', error.context);
    }
}

Error Types

• NotFoundError - Resource not found (repository, commit, file)
• ValidationError - Data validation failure (hash mismatch, invalid format)
• StorageError - Storage operation failure (I/O errors, permissions)
• LockError - Lock acquisition or release failure
• ConflictError - Concurrent modification detected (diverged HEAD)

All errors extend HashStorageError and include a context object with relevant details.

API Documentation

Public vs Internal API

This package provides two entry points:

Public API (recommended for all users):

import { HashStorage, Repository } from '@btmnstr/hashstorage-ts';

• Stable, well-documented API
• Semantic versioning guarantees
• Safe for production use

Internal API (advanced users only):

import { FileStorage, makeTreeBlob } from '@btmnstr/hashstorage-ts/internal';

• Includes internal implementation details
• May change without notice in minor versions
• Use only if you need specific internal functionality

Functions marked with @internal in the documentation may change between minor versions.

HashStorage

Main storage coordinator:

• findOrCreateRepository(name, commitSpec?) - Find or create repository
• pushCommit(repoId, commit) - Update repository HEAD
• addTag(commit, tag) - Tag a commit
• getTags(commit) - Get tags for commit
• findHash(partial) - Find full hash from partial

Repository

Version-controlled file management:

• addFile(path, data, mime) - Add or update file
• addJsonFile(path, data) - Add JSON file
• readFile(path) - Read file (throws if not found)
• readJsonFile<T>(path) - Read and parse JSON file
• findFile(path) - Find file (returns undefined if not found)
• remove(path) - Delete file
• move(oldPath, newPath) - Rename/move file
• commit(email, message) - Create commit
• log() - Iterate commit history
• stepBack() - Navigate to previous commit
• listAllFiles(dir) - List all files

Operations

Filesystem abstraction:

• mkdir(path) - Create directory
• listFiles(path) - List files
• readBuffer(path, pos?, len?) - Read file
• writeBuffer(path, data, mime) - Write file
• composeAccessURL(path) - Get access URL

Examples

See the examples/ directory for more:

• 01-basic-usage.ts - Basic file operations
• 02-versioning.ts - Multiple commits and history

Architecture

HashStorage
  ├─ RootStorage (HEAD references)
  └─ ObjectStorage (content-addressed files)
      └─ Operations (filesystem abstraction)
          ├─ LocalFSOperations
          ├─ MemoryFSOperations
          └─ CloudFSOperations (optional)

Repository
  └─ Commit (HEAD) ──> Commit ──> Commit
       └─> Tree (directory)
           ├─> Blob (file)
           └─> Blob (subdirectory)
               └─> Tree...

Testing

npm test
npm run test:coverage

Building

npm run build

Running TypeScript Files

This package uses ES modules. Use tsx instead of ts-node:

npx tsx examples/01-basic-usage.ts

Alternatively, use ts-node with the ESM flag:

npx ts-node --esm examples/01-basic-usage.ts

Why? The package is configured with "type": "module" (ESM). Standard ts-node defaults to CommonJS resolution, which conflicts with ESM imports. tsx handles this automatically.

Requirements

• Node.js ≥ 18.0.0
• ES Modules (this package is ESM-only, not compatible with CommonJS require())
• TypeScript ≥ 5.0 (for development)

Optional Dependencies

Cloud storage SDKs (install only what you need):

• @azure/storage-blob - For Azure Blob Storage
• @google-cloud/storage - For Google Cloud Storage
• @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner - For AWS S3, Cloudflare R2, or Backblaze B2

License

MIT

Credits

Originally developed as part of a larger project, extracted and published as a standalone library.