CloudRoaring

An easy-to-use, open-source library that takes the Roaring Bitmap data structure distributed and cloud-scale — so bitmaps that are too big for a single machine's RAM can live durably across tiered cloud storage, while keeping the fast, familiar in-memory API.

npm install cloud-roaring      # or: pnpm add cloud-roaring

Pre-release (0.0.0). The package name is reserved and the repo is scaffolded, but there is no functional API yet — Phase 0 (design + scaffold) is wrapping up. Follow the roadmap for progress.

Status: 🛠️ Phase 0 — all 7 design specs locked; repo scaffold (TypeScript, CI, lint, tests) landing. First functional code (the in-memory core) is Phase 1.

Table of contents

• The idea (in one paragraph)
• Documentation
• Conventions
• Getting started

The idea (in one paragraph)

The original roaring-node (C++ bindings) is a brilliant local data structure engine — but it's bound by a single process's RAM and is ephemeral (lose the process, lose the data). CloudRoaring wraps that same engine in a distributed storage architecture: a Hot tier (local RAM + LRU cache), a Warm tier (a NoSQL DB such as DynamoDB holding 8 KB binary chunks), and a Cold tier (compacted archive files in object storage such as S3). A background compaction daemon keeps the warm tier from fragmenting, and a serverless intersection engine lets thousands of functions query enormous bitmaps by pulling only the specific chunks they need. Same developer experience as roaring-node; virtually unlimited capacity and cloud durability underneath.

These are the concepts explored in the origin conversation — they are not yet finalized design decisions. See the docs below.

Documentation

All planning lives in docs/, written one document at a time as we work through the design — the docs index is the map and also defines our working process.

| Doc | What it is | |---|---| | docs/00-NORTH-STAR.md | The vision: problem, market gap, product pillars, what we're NOT building, success metrics, license — read first | | docs/01-ARCHITECTURE.md | System design: key routing, the three tiers, read/write paths, the .crbm format, pluggable drivers, the intersection engine, the compaction daemon, runtime/daemon modes, ADRs | | docs/02-POSITIONING.md | Why CloudRoaring & when — fact-checked cost/performance vs Redis & ClickHouse, where we win/lose, design levers | | docs/03–09 | The build specs (locked): data model, .crbm format, driver SDK, lifecycle/erasure, threat model, cost model, testing/determinism | | docs/10-SEO-AND-RELEASE.md | Naming/SEO strategy, npm + GitHub discoverability, the release & launch checklist | | docs/research/adversarial/ | Adversarial pre-build review (8 lenses) — critical correctness/cost/security findings + the data-model revision they force (research) | | docs/research/02-COST-PERFORMANCE-VERIFICATION.md | Source-cited fact-check of every cost/scale/performance claim (research) | | docs/99-ROADMAP.md | The phased build plan — every architecture piece + every stress-test fix, chunked into phases. The living source of project state | | docs/research/00-ORIGIN-CONVERSATION.md | Verbatim capture of the origin conversation (Google AI Mode / "Gemini") that started the project — all 16 turns, every diagram, table, and formula. Raw research input. |

More docs (file-format spec, storage-driver SDK, compaction reliability) will be added as we design each piece.

Conventions

This project follows the personal-repo conventions in ~/sharvil/README.md: kebab-case folder, own git repo, personal git identity, and — once implementation starts — the full engineering-standards baseline (TypeScript + strict typecheck, lint, format, tests, CI, pre-commit hooks). Per-phase working process (branch off main, test, commit/push, adversarial subagent review) lives in CLAUDE.md and the docs index. License: Apache-2.0 (decided) + trademark the name — see the North Star.

Getting started

The library has no functional API yet, but the repo is scaffolded — a fresh clone passes the full gate with no manual setup:

pnpm install
pnpm lint && pnpm lint:arch && pnpm format:check && pnpm typecheck && pnpm test && pnpm build
pnpm test:integration   # spins up DynamoDB-Local + MinIO via `docker compose`

To understand the project, read docs/00-NORTH-STAR.md; for how it will be used end to end (with a live scenario), see USAGE.md.