@x12i/hybrid-docstore

The Tier 1 base of the @x12i Edge Data Stack. A schema-less, MongoDB-like document database that transparently offloads heavy payloads (binary buffers, large text) to object storage (Cloudflare R2 or local disk) while indexing lightweight JSON records in a relational database (Cloudflare D1 or SQLite).

Features

• Transparent Content Interceptor: Converts { title: "Post", content: Buffer } into { title: "Post", contentKey: "articles/123/content-..." }.
• Atomicity Rollback: If a database write fails after storage upload, uploaded objects are immediately deleted to avoid orphaned storage leaks.
• Update Overwrite Cleanup: Updating a document with a new heavy payload replaces the key in SQLite and asynchronously deletes the old object from storage.
• Resilient Delete Lifecycle: Database rows are deleted first; failed storage deletions are routed to a dead-letter handler without crashing the app.
• Concurrency-Capped Lazy Loading: .withContent(['content']) batches storage fetches (default: 16 concurrent requests) to prevent socket or rate-limit exhaustion.
• Zero-Config .env Support: Automatically connects using credentials declared in .env.

Environment Variables (.env)

| Variable | Description | Default | | :--- | :--- | :--- | | STORAGE_PROVIDER | "local", "r2", or "bucket" | "local" | | STORAGE_LOCAL_DIR | Local folder for object files (when STORAGE_PROVIDER=local) | ./.storage | | R2_ACCOUNT_ID | Cloudflare Account ID (when STORAGE_PROVIDER=r2) | - | | R2_BUCKET_NAME | Cloudflare R2 bucket name | - | | HYBRID_BUCKET_URL | HTTP bucket base URL used from Node. Inferred as STORAGE_PROVIDER=bucket when set | - | | DATABASE_PROVIDER | "sqlite", "d1", or "worker" | "sqlite" | | SQLITE_DB_PATH | Path to SQLite file or ":memory:" | ":memory:" | | CLOUDFLARE_D1_DATABASE_ID | Cloudflare D1 Database UUID | - | | CLOUDFLARE_ACCOUNT_ID | Cloudflare Account ID | - | | CLOUDFLARE_API_TOKEN | Cloudflare API Token | - | | HYBRID_DB_API_URL | Worker URL for remote D1. Requests go to {url}/query with Cloudflare Access service auth | - | | HYBRID_DB_CLIENT_ID | Cloudflare Access service token client id | Required with HYBRID_DB_API_URL | | HYBRID_DB_SECRET | Cloudflare Access service token secret | Required with HYBRID_DB_API_URL | | HYBRID_DB_FOLDER | Optional folder. Empty uses the bucket root and omits folder from queries. A value scopes bucket keys and queries to that folder only | "" | | DOCSTORE_CONCURRENCY | Max concurrent storage hydration requests | 16 |

Quick Start

import { createDocStoreFromEnv } from '@x12i/hybrid-docstore';

// Reads from .env automatically
const store = await createDocStoreFromEnv();

interface Article {
  id: string;
  title: string;
  content: Buffer;
}

const articles = store.collection<Article, 'content'>('articles', {
  heavyProperties: ['content'],
});

// 1. Insert: Heavy payload 'content' is transparently offloaded
const post = await articles.insert({
  title: 'Hello World',
  content: Buffer.from('Rich markdown body...'),
});
// post -> { id: '...', title: 'Hello World', contentKey: 'articles/.../content-...' }

// 2. Query: Lightweight documents returned natively
const list = await articles.find().limit(10);
// list[0].content is undefined, list[0].contentKey is present

// 3. Explicit Materialization with concurrency limiting
const full = await articles.find().withContent(['content']);
// full[0].content is Buffer