@x12i/metagit

Git-like version control for MongoDB-backed metadata repositories (Mongo-first, DB-as-repo).

Metagit versions the full selected MongoDB state after applying .metagitignore (records included unless excluded). It is not positioned as a generic operational database backup system.

Sync vs pipeline deploy (mental model)

| Layer | What it is | Analog | |-------|----------------|--------| | Object store sync | commit snapshots eligible Mongo data into immutable objects; push / pull / clone move that history with GCS (your “remote”). | Git commits + fetch/push; another checkout is like another clone/pull of the same repo. | | Pipeline deploy | Named releases point at commits; deploy / promote / rollback apply a chosen commit’s manifests onto a named environment (another Mongo), with plans, protections, and checkpoints. | CI/CD promoting a release artifact to QA/staging/prod—not “clone the repo onto the server.” |

Managed slice on targets: For deploy/diff-to-live/applyCommit in exact mode, live Mongo reads and extraneous-record detection use the same rules as diff --live: .metagitignore at the repo root, optionally merged with per-environment deployMetagitignorePath / deployMetagitignoreExtraPath in .metagit/pipeline.json (see metagit env add --help). Collections excluded by those rules are skipped for deploy even if present in the commit object.

Multi-repo releases: Use metagit release-bundle to capture the same release name across several metagit repos under one workspace, then release-bundle plan / release-bundle apply toward a shared pipeline environment (each component repo must define that environment). Bundle manifests live under .metagit/store/release-bundles/ in the repo where you run release-bundle create.

Pipeline config location: Default is .metagit/pipeline.json. Override with env var METAGIT_PIPELINE_CONFIG (absolute path)—useful when CI generates the file per run.

Documentation: Design specification · Large databases & operators · Appendix · Open feature requests (large deployments) · Planning: Storage tier · Public API (token)

Install

npm i @x12i/metagit

Runtime: Node.js ≥ 18. Integration tests and smoke scripts expect MongoDB and Google Cloud Storage (see Develop / test).

CLI overview

npx metagit --help
npx metagit --version

Machine-readable output (two channels)

| Flag | Stream | Purpose | |------|--------|---------| | --json | stdout | Final command result as JSON where the command supports it (e.g. metagit --json diff --live, metagit --json commit -m "…"). | | --json-progress | stderr | Newline-delimited JSON progress events (schemaVersion: "metagit.progress.v1") for long-running work. Independent of --json; you may use both together. |

Progress on stderr (human or NDJSON)

Long-running commands emit progress on stderr so stdout stays pipe-friendly when using --json.

| Root flag | Behavior | |-----------|----------| | (default, auto) | Human-readable lines when stderr or stdout is a TTY, or when FORCE_COLOR is enabled — otherwise quiet (typical CI with both streams piped) | | --progress | Force human progress even when stderr is not a TTY | | --no-progress | Disable human progress | | --quiet | Same as --no-progress (still overridden by --json-progress) | | --json-progress | NDJSON events on stderr (stable schema; see specs §38a) | | --progress-interval-ms <n> | Coalesce phase-tick events (default 1000). Use 0 to emit every tick. | | --verbose | Extra meta on human progress lines | | METAGIT_PROGRESS | When the CLI would otherwise use auto progress: human, json, or off (same as quiet). Overrides TTY / FORCE_COLOR heuristics. CLI flags above still win. |

npm / subprocess wrappers: many spawn Node with a non-TTY stderr; auto still prints human progress when stdout is a TTY (interactive terminal). For fully piped runs, use --progress, --json-progress, or METAGIT_PROGRESS. To force silence when stdout is a TTY, use --quiet or METAGIT_PROGRESS=off.

Commands that honor progress and safety flags: commit, diff (--live or --from/--to), push, pull, clone, checkout, promote, deploy plan, deploy apply, rollback plan, rollback apply, release-bundle plan, release-bundle apply.

Integrity gates: Declarative rulesets + optional read-only .mjs checks in .metagit/pipeline.json (integrity.rulesets, per-env integrityRuleset, optional requiresIntegrity). Gates run automatically around release create, deploy/promote/release-bundle plan, and apply (rollback apply also runs preRollbackApply / postRollbackApply). metagit integrity (validate-config, explain, run) is for inspection and manual runs. Reports live under .metagit/store/integrity-reports/. Apply-style commands support --skip-integrity unless the environment sets requiresIntegrity: true. Details: specs §6.1b (integrity subsection).

Apply-style flags: deploy/promote/rollback/release-bundle apply accept --confirm-large-target when the target environment sets requireConfirmAboveEstimatedDocs and the preflight estimate exceeds it (see metagit env add). They also accept --skip-integrity where allowed (see integrity gates above).

Core commands (v1)

| Command | Purpose | |--------|---------| | init | Create .metagit/, config, local store; optional GCS remote/* bootstrap | | commit | Snapshot current Mongo state into a commit | | push / pull | Sync object store with .metagit/store | | clone | init + pull from a gs://… URL (use --gcs-credentials-base64-env when using SA-in-env, same as init) | | checkout | Apply a commit (indexes optional) | | diff --live | Compare HEAD to live Mongo | | diff --from <h> --to <h> | Compare two object-store commits | | log | Print commits from HEAD (first-parent chain); -n / --max-count; --json |

Initialize (Mongo + GCS; env vars hold secrets, not the config file):

metagit init \
  --driver mongo \
  --url-env MONGO_URI \
  --db mydb \
  --name my-repo \
  --gcs-bucket my-bucket \
  --gcs-base-path my-prefix \
  --gcs-credentials-base64-env GOOGLE_SERVICE_ACCOUNT_BASE64 \
  --env local

Useful init options (all optional):

• --gcs-root-folder <segment> — first path segment under the bucket for keys (default: metagit)
• --strict-remote — fail if remote/* cannot be written to GCS (default: warn and continue)
• --remote-secrets-json <path> — upload wrapped secrets to GCS (high risk; strict IAM)
• --repo-metadata-json <path> — non-secret operator metadata (tier, purpose, hosting, …); see specs §6.1

After init, remote/repo.v1.json and remote/client-hints.v1.json are written under .metagit/store/remote/ and uploaded when GCS is reachable. Each push refreshes lastPushedAt on the remote copy. Layout and behavior are documented in specs §5.3–5.4.

Optional repo metadata

You can document deployment tier, purpose, related systems, hosting, exposure, access notes, etc. (never secrets) via --repo-metadata-json or the repo.metadata field in .metagit/config.json. The same block is mirrored on remote/repo.v1.json when present. Details: specs §6.1.

Optional record annotations

Opt-in Mongo metadata under _<name> (default _metagit; name is configurable). It is written only when applyCommit runs—i.e. pipeline deploy / promote / rollback / release-bundle apply, not when you commit alone. Enable at metagit init (--record-annotation, --record-annotation-name, …) or via recordAnnotation in .metagit/config.json, with optional overrides on pipeline environments. Behavior (merge rules, diff stripping, fast path): specs §2.1b.

Pipeline layer (environments, releases, deploy, rollback, merge, promote, release-bundle)

The pipeline layer moves a selected metagit commit across named MongoDB environments in a controlled, auditable, reversible way.

Command groups

metagit env
metagit release
metagit release-bundle
metagit deploy
metagit rollback
metagit merge
metagit promote
metagit integrity

Configure environments (stored in .metagit/pipeline.json; Mongo URIs are referenced by env var name, not stored inline)

metagit env add dev \
  --mongo-url-env MONGO_DEV_URI \
  --db mydb

metagit env add production \
  --mongo-url-env MONGO_PROD_URI \
  --db mydb \
  --protected \
  --requires-plan \
  --requires-confirmation \
  --requires-pre-deploy-snapshot \
  --hosting cloud \
  --preflight-warn-estimated-docs 500000 \
  --require-confirm-above-estimated-docs 2000000

# Optional: attach a named integrity ruleset and require gates (see specs §6.1b)
# metagit env add production ... --integrity-ruleset production-safe --requires-integrity

Optional --deploy-metagitignore / --deploy-metagitignore-extra: repo-relative paths to additional ignore files merged after .metagitignore when planning/applying to that environment only.

If two environment names share the same mongoUrlEnv + db, metagit emits a warning (often accidental aliases).

Create a version in dev and deploy it to production

# Create a commit from dev (whatever `.metagit/config.json` points at)
metagit commit -m "prepare catalog v3"
metagit push

# Name it as a release
metagit release create catalog-v3 --commit HEAD --from dev

# Plan (read-only)
metagit deploy plan --release catalog-v3 --to production

# Apply
metagit deploy apply --release catalog-v3 --to production --plan <planHash>

Mandatory pre-deploy snapshot (rollback point)

Before deploy apply modifies the target environment, metagit will:

• create a pre-deploy snapshot commit of the target environment (e.g. production)
• push it successfully
• store it as beforeCommit in deployments/<deploymentId>.json

That beforeCommit is the rollback point.

Rollback

metagit rollback plan --deployment <deploymentId>
metagit rollback apply --deployment <deploymentId> --mode exact

Merge before deploy

Merge creates a new metagit commit first (it does not modify any Mongo environment).

metagit merge plan --left env:dev --right env:production
metagit merge resolve --merge <mergeId> --conflict users/64f... --take right
metagit merge create --merge <mergeId> -m "merge production hotfixes into dev"

Promote shortcut

# Plan only
metagit promote dev production --plan

# Apply (requires confirmation for protected env)
metagit promote dev production --apply

Release bundle (monorepo)

# From a repo that will store the bundle manifest (cwd has .metagit):
metagit release-bundle create catalog-2026 --root /path/to/workspace --component packages/svc-a --component packages/svc-b

metagit release-bundle plan --bundle catalog-2026 --to staging
# Save JSON map label -> planHash for environments with --requires-plan
metagit release-bundle apply --bundle catalog-2026 --to production --plans-json ./plans-by-label.json

Large databases: progress, preflight, safety

For large MongoDB deployments and slow networks, metagit reports how big the job is before heavy work and lets you bound batching and concurrency.

Preflight

Before the main loops, applicable commands emit a preflight progress phase whose phase-end summarizes scope (e.g. collection counts, sums of Mongo estimatedDocumentCount, record counts from manifests, file/object counts for sync). Human stderr prints a Preparing line with summary …; --json-progress emits phase-end with the same fields under meta. Details and keys: specs §38a.2.1.

Examples

# Human progress on stderr (default in auto mode when stderr or stdout is a TTY, or FORCE_COLOR)
metagit promote dev production --apply

# NDJSON on stderr for CI / wrappers (stdout unchanged)
metagit --json-progress promote dev production --apply 2>events.ndjson

# Final JSON on stdout + progress NDJSON on stderr
metagit --json --json-progress deploy apply --release catalog-v3 --to production --plan <hash> \
  2>events.ndjson | jq .

# Silence progress
metagit --quiet commit -m "snapshot"

# Throttle tick spam (or use 0 for every tick)
metagit --progress-interval-ms 250 commit -m "snapshot"

Safety flags (on relevant subcommands)

Defaults are conservative (e.g. --max-collection-concurrency default 1). Full tables and tuning guidance: docs/large-databases.md.

| Flag | Role | |------|------| | --batch-size <n> | Records per bulkWrite / deleteMany chunk in apply (default 1000) | | --max-collection-concurrency <n> | Parallel collections during apply (default 1) | | --throttle-ms <n> | Pause between apply batches (default 0) | | --cursor-batch-size <n> | Mongo read cursor batch size (default 1000) | | --max-time-ms <n> | Mongo maxTimeMS on batched operations | | --mongo-max-pool-size <n> | Driver pool (also MONGO_MAX_POOL_SIZE) | | --mongo-socket-timeout-ms <n> | Also MONGO_SOCKET_TIMEOUT_MS | | --mongo-server-selection-timeout-ms <n> | Also MONGO_SERVER_SELECTION_TIMEOUT_MS | | --push-concurrency <n> | Parallel uploads in push (default 4) | | --pull-concurrency <n> | Parallel downloads in pull / clone (default 4) | | --dry-run | Apply paths: compute progress, no Mongo writes | | --resume <deploymentId> | Skip collections already listed in .metagit/store/checkpoints/<id>.json | | --confirm-large-target | Acknowledge large target when env requireConfirmAboveEstimatedDocs would otherwise abort (deploy apply, rollback apply, promote --apply, release-bundle apply) |

Apply-style commands (deploy apply, rollback apply, promote --apply, release-bundle apply) pass the active deploymentId (or rollback id) as the resume key automatically; use --resume to target another checkpoint explicitly.

Checkpoints and --records upsert-only|exact: see docs/large-databases.md. Progress schema and phase names: specs §38a. Failures include an error progress event with a metagit rollback apply --deployment <id> hint where applicable.

Library

import {
  metagit,
  createProgressReporter,
  noopProgressReporter,
  resolveSafetyOptions,
  type MetagitRepoMetadataV1,
  type DiffBetweenCommitsResult,
  type ProgressEventV1,
  type ProgressReporter,
  type SafetyOptions
} from "@x12i/metagit";

const repo = await metagit.open();
const diff = await repo.diffLive();
console.log(diff);

// Optional: pass a reporter + safety into repo methods (CLI does this for you).
const reporter = createProgressReporter({ mode: "json", intervalMs: 1000 });
reporter.start("commit", {});
try {
  await repo.commit({
    message: "snapshot",
    reporter,
    safety: resolveSafetyOptions({ batchSize: 500, cursorBatchSize: 1000 })
  });
} finally {
  reporter.end();
}

// Compare two object-store commits (no Mongo). Result includes `schemaVersion: "metagit.diffBetweenCommits.v1"`.
const between: DiffBetweenCommitsResult = await repo.diffBetweenCommits({
  from: "<commit-hash-a>",
  to: "<commit-hash-b>",
  recordFull: false
});
console.log(between);

CLI equivalents: metagit diff --from <hash-a> --to <hash-b> (add --json and/or --record-full as needed). Default metagit diff / metagit diff --live compare HEAD to live MongoDB.

Also exported: humanProgressReporter, jsonProgressReporter, PhaseHandle, ProgressMode, ProgressEventType, ResolvedSafetyOptions, safetyDefaults, ApplyCheckpointV1, DiffBetweenCommitsOptions, DiffCommitToLiveOptions, DiffCollectionDelta, DiffLiveOptions, DiffLiveResult, DeployIgnoreLoadOptions, mergeMetagitIgnoreRules, loadEffectiveDeployIgnoreRules, eligibleCommitCollections, recordIncludeFilterForCollection, deployIgnoreLoadOptionsFromPipelineEnv, pipeline helpers (resolvePipelineConfigPath, duplicateMongoTargetWarningLines, …), release-bundle types, and CreateProgressReporterOptions.

Use repo.listCommitsFromHead({ maxCount }) for commit history; repo.diffCommitToLive / repo.applyCommit accept deployIgnore and optional deployPreflightContext for library-driven deploy tooling.

MetagitRepoMetadataV1 is optional metadata you can attach at init or in config; see specs §6.1. Diff shapes are documented in specs §24.5–24.5.1.

Develop / test

From the package repository root, typical env vars are MONGO_URI, STORAGE_BUCKET, and optional GOOGLE_SERVICE_ACCOUNT_BASE64 (see scripts/live-smoke.mjs and tests/scenarios/*/.env patterns).

| Script | What it runs | |--------|----------------| | npm test | typecheck → build → tests/run-all.mjs → npm pack --dry-run | | npm run test:cli | build + scenario tests only (no pack:check) | | npm run typecheck | TypeScript --noEmit | | npm run build | tsup → dist/ | | npm run dev | tsup --watch | | npm run pack:check | npm pack --dry-run (publish tarball dry run) | | npm run live:smoke | Build + end-to-end Mongo + GCS smoke (scripts/live-smoke.mjs) | | npm run pipeline:smoke | Build + pipeline smoke (scripts/pipeline-smoke.mjs; expects an initialized .metagit at repo root and MONGO_DEV_URI / MONGO_PROD_URI) | | npm run cleanup:test-dbs | Drop disposable test Mongo DBs: test-metagit_cli_*, test-metagit_live_test_*, and legacy unprefixed names (uses repo .env MONGO_URI). Add --dry-run on the script to list only. |

Integration scenarios use disposable DB names prefixed with test- (e.g. test-metagit_cli_<scenario>_<timestamp>). Each scenario drops its Mongo database after a successful run. Use npm run cleanup:test-dbs for leftovers from failed runs, old unprefixed names, or scripts like npm run live:smoke (which does not auto-drop).

npm test

Shell completions for deploy / rollback subcommands ship under scripts/completions/ (metagit.bash, metagit.zsh). Source the appropriate file from your shell rc after the usual completion setup (compinit on zsh).

Publishing notes

• 1.8.0 aligns deploy / diff-to-live / exact apply with .metagitignore (plus optional per-env deploy ignore merges), adds METAGIT_PIPELINE_CONFIG, duplicate-env warnings, hosting / preflight thresholds / --confirm-large-target, metagit log, metagit release-bundle, and expanded library exports. Breaking vs older previews: deploy semantics on targets previously scanned whole collections for live diffs.
• 1.6.0 adds repo.diffBetweenCommits, metagit diff --from / --to, and named TypeScript exports for diff-related types (DiffBetweenCommitsResult, DiffLiveResult, DiffCollectionDelta, …).
• Later 1.6.x adds stderr progress (metagit.progress.v1), preflight scope summaries, safety knobs (batching, concurrency, throttles, checkpoints, --dry-run / --resume), GCS upload/download concurrency, and Mongo driver options via flags and env vars; see specs §38a and docs/large-databases.md.
• Auto progress: besides stderr TTY, auto enables human lines when stdout is a TTY or FORCE_COLOR is truthy; METAGIT_PROGRESS overrides when the CLI stays on auto. Scripts that require a silent stderr should pass --quiet or METAGIT_PROGRESS=off.
• The package is configured as public via publishConfig.access = "public".
• Published files: dist/, docs/specs.md, docs/large-databases.md, docs/open-feature-requests.md, scripts/completions/, README.md, LICENSE (see package.json files).
• Do not commit a real .npmrc token file; use .npmrc.example if you document registry setup.
• prepublishOnly runs build, typecheck, and pack:check (no live Mongo required).
• Run npm test before tagging a release when you change behavior that touches Mongo or GCS.