@s3p/s3db

Substrate-agnostic data layer over S3-compatible object storage. Bring your own S3 client (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Tigris, …) and your own auth — s3db only needs an S3Client instance and a bucket name.

See ../../DATA.md for the design rationale: the two-tier (document + analytical) model, per-author rollup shards, encryption-aware Parquet schema, and how the platform composes these into @s3p/sdk.

What's shipped

• Document store (createDocuments) — list / get / append / remove with ULID-keyed records and an optional author-partition layout.
• Parquet writer (writeParquetBytes) — hybrid plaintext-key + encrypted-body schema; predicate pushdown works on the plaintext columns.
• Rollup primitive (createRollups) — per-author Parquet shards at {prefix}/_rollups/{authorSub}/v1-{ulidLo}-{ulidHi}.parquet. parseShardKey / shardFilename are the path conventions.
• Analytical store (createAnalytics) — DuckDB-WASM over the rollup glob, with dynamic-import so apps that don't query analytics don't pay the 33 MB cost at SDK init.
• Envelope encryption adapter (createEnvelopeAdapter) — ECDH P-256 + AES-GCM-256 + HKDF-SHA256, with wrap / unwrap hooks on the document store. Recipient sets are RecipientPublicKey[]; resolution forms ({ group }, { role }, { subs }, function) live in the SDK.
• ULID (ulid, ulidTimestamp, ulidLowerBound) — monotonic, time-prefixed; the key sort order matches event order.
• RBAC tests (rbac.test.ts) — pin the truth table for who can decrypt what under combinations of partition × encryptedFor × membership.

Usage

import { createDocuments } from "@s3p/s3db";
import { S3Client } from "@aws-sdk/client-s3";

const s3 = new S3Client({ /* your creds, your endpoint */ });

const messages = createDocuments<{ body: string }, { roomId: string }>({
  bucket: "app-data",
  getClient: async () => s3,
  prefix: (ctx) => `rooms/${ctx.roomId}/messages`,
  partition: "author",
});

await messages.append(
  { roomId: "abc" },
  { body: "hi" },
  { authorSub: "alice" },
);

for await (const m of messages.list({ roomId: "abc" }, { limit: 100 })) {
  console.log(m.key, m.body);
}

With envelope encryption:

import { createDocuments, createEnvelopeAdapter } from "@s3p/s3db";

const crypto = createEnvelopeAdapter({
  ownerSub: app.me.sub,
  ownerKeypair: await app.identity.getKeypair(),
  resolveRecipients: async () => [
    { sub: "alice", publicKey: await app.identity.publicKeyOf("alice") },
    { sub: "bob",   publicKey: await app.identity.publicKeyOf("bob") },
  ],
});

const posts = createDocuments<PostBody, { topicId: string }>({
  bucket: "my-bucket",
  getClient: async () => s3,
  prefix: ({ topicId }) => `topics/${topicId}/posts`,
  partition: "author",
  crypto,
});

Why "bring your own S3 client"

s3db deliberately doesn't know about identity, STS, OIDC, or per-platform quirks. That's the whole point of the substrate- agnostic split — any auth strategy that can produce an authenticated S3Client can use this package.

Inside @s3p/sdk the client is built from a Cognito OIDC session and a Cognito Identity Pool → STS exchange. Outside, you can use raw AWS keys, R2 keys, MinIO admin keys, whatever your stack already produces.

Dev loop

cd packages/s3db
npm run build         # tsup -> dist/index.js + index.d.ts
npm test              # vitest run
npm run typecheck     # tsc --noEmit
npm run lint          # biome check

Tests

Vitest covers:

• Document store (list/get/append/remove with mock S3).
• ULID monotonicity + time extraction.
• Parquet writer round-trip via hyparquet.
• Rollup primitive (single-writer-shard idempotency, shard discovery, high-watermark resumption).
• Analytical store (DuckDB-WASM virtual-file registration, multi- shard union, pushdown).
• Envelope encryption — wrap / unwrap permutations across partition × encryption × membership combos (rbac.test.ts, story.test.ts).