GitDB

English | 한국어 | Website

GitDB is a first-party database runtime that uses repository storage as durable history. Queries execute locally. Writes commit through a manifest-gated mutation log. Plaintext stores expose reviewable page snapshots, while GitHub stores sync durable state through Git tree commits.

App code
  -> GitDB DataSource / Repository
  -> Local SQL engine + equality indexes
  -> Manifest, mutation log, page snapshots, compaction
  -> GitHub repository for durable history and audit

GitDB is not a single JSON file with a SQL-shaped API. Its identity is the runtime and storage engine: transaction boundaries, single-writer control, replayable logs, checkpointed snapshots, rebuildable indexes, compaction, and GitHub audit sync.

Why GitDB

Use GitDB when you want:

• A database repository per project, for example my-app-db
• Local query execution without per-query network round trips
• A TypeORM-style DataSource and repository API included in the package
• Human-inspectable plaintext snapshots for public demos and reviewable data
• Encrypted manifest and mutation logs for private local data
• Auditable Git commits for agents, demos, content tools, config tools, and low-frequency app data

GitDB is still experimental. Do not use it for high-throughput OLTP, low-latency distributed writers, range-heavy analytics, or workloads that require mature multi-node coordination.

Current Surface

| Area | Current behavior | | --- | --- | | App API | createGitDbDataSource, defineEntity, typed repositories | | ORM | save, insert, insertMany, saveMany, find, findOne, delete | | SQL engine | CREATE TABLE, INSERT, DELETE, SELECT, joins, grouping, ordering, aggregates | | Indexes | Rebuildable primary/equality indexes for simple equality lookup | | Storage | Local plaintext, local encrypted, GitHub plaintext, GitHub encrypted | | Durability | Manifest-gated mutation log replay with checkpointed visible snapshots | | Concurrency | Local single-writer with multiple readers and stale lock recovery | | Compaction | Local plaintext log compaction behind page snapshot checkpoints | | GitHub sync | Git Database tree, commit, and non-force ref updates | | CLI | gitdb keygen, gitdb query, gitdb check | | Package | npm exports, bin, pack dry-run, publish dry-run scripts |

Guarantee Matrix

| Capability | Guarantee | Boundary | | --- | --- | --- | | Transactions | Mutations are serialized and manifest-gated before becoming committed state. | In-process queue plus store commit boundary | | Concurrency | Local stores own a single writer lock with stale recovery; readers can reopen committed state. | Not distributed OLTP | | Durability | Reopen uses a checkpoint only when its sequence matches the manifest; otherwise logs replay. | Manifest sequence | | Snapshots | Plaintext snapshots are page-level JSON files plus snapshot.json. | Optimization, not source of truth | | Indexes | Equality indexes rebuild from committed rows and are ignored when stale or missing. | Derived data | | Compaction | Checkpoint first, manifest advance second, obsolete log deletion last. | Local plaintext store | | GitHub remote | Writes use tree commit batches and non-force ref updates. | Remote audit sync, not SELECT hot path |

Repository Layout

Plaintext mode writes internal state plus human-readable snapshots:

gitdb/v1/
  manifest.json
  snapshot.json
  people/
    schema.json
    pages.json
    pages/
      000000.json
    indexes.json
  log/
    00000000000000000001.json

schema.json contains only schema:

{
  "name": "people",
  "columns": ["id", "name", "team_id"]
}

Page files contain rows:

[
  { "id": "p1", "name": "Lin", "team_id": "t1" },
  { "id": "p2", "name": "Ada", "team_id": "t2" }
]

Encrypted mode writes opaque files:

gitdb/v1/
  manifest.enc
  log/
    00000000000000000001.enc

Quick Start

Install:

npm install @3xhaust/gitdb

Use the first-party repository API:

import { LocalPlaintextStore, createGitDbDataSource, defineEntity } from "@3xhaust/gitdb"

type Person = {
  readonly id: string
  readonly name: string
  readonly team_id: string
}

const PersonEntity = defineEntity<Person>({
  columns: { id: "STRING", name: "STRING", team_id: "STRING" },
  indexes: [{ columns: ["team_id"], name: "people_team_id_idx" }],
  primaryKey: "id",
  tableName: "people",
})

const dataSource = await createGitDbDataSource({
  entities: [PersonEntity],
  store: new LocalPlaintextStore({ root: ".gitdb" }),
  synchronize: true,
})

const people = dataSource.getRepository(PersonEntity)
await people.insertMany([
  { id: "p1", name: "Lin", team_id: "storage" },
  { id: "p2", name: "Ada", team_id: "runtime" },
])
await people.saveMany([{ id: "p2", name: "Ada Lovelace", team_id: "runtime" }])

const storagePeople = await people.find({ where: { team_id: "storage" } })

Secondary indexes are equality-only metadata on defineEntity. The runtime uses rebuilt equality indexes for simple find({ where }) and SELECT * ... WHERE lookups when the query shape is safe; other SQL falls back to the local SQL engine.

CLI

Generate an encryption key:

gitdb keygen

Check the configured store:

GITDB_ENCRYPTION=off GITDB_ROOT=.gitdb gitdb check

Execute one SQL statement:

GITDB_ENCRYPTION=off GITDB_ROOT=.gitdb \
  gitdb query "CREATE TABLE people (id STRING, name STRING)"

Environment Model

Local plaintext mode:

GITDB_ENCRYPTION=off
GITDB_ROOT=.gitdb

Local encrypted mode:

GITDB_ENCRYPTION=on
GITDB_KEY=generated-by-gitdb-keygen
GITDB_ROOT=.gitdb

GitHub-backed modes additionally use:

GITDB_GITHUB_OWNER=3x-haust
GITDB_GITHUB_REPO=my-project-db
GITDB_GITHUB_BRANCH=main
GITDB_GITHUB_PREFIX=gitdb/v1
GITDB_GITHUB_TOKEN=github_token_with_contents_write_access

Leave GITDB_GITHUB_TOKEN blank for local-only development. Use GITDB_ENCRYPTION=off only for intentional public demos where table names, columns, and rows should be visible.

Operations

Run the bundled example:

corepack pnpm example

This builds the package and runs the plaintext API example plus two encrypted API examples:

• examples/api-local
• examples/api-encrypted
• examples/api-encrypted-reopen

Run local benchmarks:

corepack pnpm benchmark

Compare current benchmark evidence with the previous documented run:

GITDB_BENCH_ROWS=250 corepack pnpm benchmark:compare

Compact committed logs on a local plaintext engine:

const engine = await GitDbEngine.open({ store: new LocalPlaintextStore({ root: ".gitdb" }) })
await engine.compact()

Compaction writes a matching page snapshot first, advances the manifest to an empty log list second, and deletes obsolete log segments last.

Documentation

• Architecture
• API
• Benchmarks
• Migration
• Release Handoff
• Release Notes

Current Limitations

• SQL support is intentionally limited to the subset GitDB currently executes.
• Local writes are single-writer with stale lock recovery. This is not distributed OLTP.
• GitHub tree commits are for durable audit sync, not the SELECT hot path.
• Local encrypted compaction is not enabled until encrypted checkpoint restore lands.
• Public plaintext mode is not private mode.

Commands

corepack pnpm check
corepack pnpm test
corepack pnpm build
corepack pnpm benchmark
corepack pnpm benchmark:evaluate
corepack pnpm pack:dry-run
corepack pnpm publish:dry-run
corepack pnpm example

License

MIT