@x12i/magit

Git-like version control for MongoDB-backed metadata repositories (Mongo-first, DB-as-repo).

MaGit versions the full selected MongoDB state after applying .metagitignore (records included unless excluded). It is not positioned as a generic operational database backup system.

Sync vs pipeline deploy (mental model)

| Layer | What it is | Analog | |-------|----------------|--------| | Object store sync | commit snapshots eligible Mongo data into immutable objects; push / pull / clone move that history with GCS (your “remote”). Content-addressed objects are immutable; refs and workflow manifests are overwritten on push. | Git commits + fetch/push; another checkout is like another clone/pull of the same repo. | | Pipeline deploy | Named releases point at commits; deploy / promote / rollback apply a chosen commit’s manifests onto a named environment (another Mongo), with plans, protections, and checkpoints. | CI/CD promoting a release artifact to QA/staging/prod—not “clone the repo onto the server.” |

Managed slice on targets: For deploy/diff-to-live/applyCommit in exact mode, live Mongo reads and extraneous-record detection use the same rules as diff --live: .magitignore at the repo root (legacy .metagitignore is still honored), optionally merged with per-environment deployMetagitignorePath / deployMetagitignoreExtraPath in .magit/pipeline.json (legacy .metagit/pipeline.json fallback; see magit env add --help). Collections excluded by those rules are skipped for deploy even if present in the commit object.

Multi-repo releases: Use magit release-bundle to capture the same release name across several MaGit repos under one workspace, then release-bundle plan / release-bundle apply toward a shared pipeline environment (each component repo must define that environment). Bundle manifests live under .magit/store/release-bundles/ in the repo where you run release-bundle create.

Pipeline config location: Default is .magit/pipeline.json (legacy .metagit/pipeline.json fallback). Override with env var METAGIT_PIPELINE_CONFIG (absolute path)—useful when CI generates the file per run.

Documentation: Test guide · Design specification · Large databases & operators · Appendix · Open feature requests (large deployments) · Planning: Storage tier · Public API (token)

Install

npm i @x12i/magit

Runtime: Node.js ≥ 20. Integration tests and smoke scripts expect MongoDB and Google Cloud Storage (see Develop / test).

Compatibility names

MaGit is the public package and CLI name. New local checkouts use MaGit names, while existing Metagit-era repos are still read in place:

• Local repo directory: .magit/ (legacy .metagit/ fallback)
• Ignore file: .magitignore (legacy .metagitignore fallback)
• Environment variables: METAGIT_*
• JSON schemas and progress events: metagit.*.v1
• Built-in integrity rule keys: metagit/...
• Default GCS root folder: metagit
• Legacy CLI alias: metagit

Treat the remaining metagit.* values as wire-format identifiers. They should change only through a schema/storage migration that can read old repos and write the new layout intentionally.

CLI overview

npx magit --help
npx magit --version

Machine-readable output (two channels)

| Flag | Stream | Purpose | |------|--------|---------| | --json | stdout | Final command result as JSON where the command supports it (e.g. magit --json diff --live, magit --json commit -m "…"). | | --json-progress | stderr | Newline-delimited JSON progress events (schemaVersion: "metagit.progress.v1") for long-running work. Independent of --json; you may use both together. |

Progress on stderr (human or NDJSON)

Long-running commands emit progress on stderr so stdout stays pipe-friendly when using --json.

| Root flag | Behavior | |-----------|----------| | (default, auto) | Human-readable lines when stderr or stdout is a TTY, or when FORCE_COLOR is enabled — otherwise quiet (typical CI with both streams piped) | | --progress | Force human progress even when stderr is not a TTY | | --no-progress | Disable human progress | | --quiet | Same as --no-progress (still overridden by --json-progress) | | --json-progress | NDJSON events on stderr (stable schema; see specs §38a) | | --progress-interval-ms <n> | Coalesce phase-tick events (default 1000). Use 0 to emit every tick. | | --verbose | Extra meta on human progress lines | | METAGIT_PROGRESS | When the CLI would otherwise use auto progress: human, json, or off (same as quiet). Overrides TTY / FORCE_COLOR heuristics. CLI flags above still win. |

npm / subprocess wrappers: many spawn Node with a non-TTY stderr; auto still prints human progress when stdout is a TTY (interactive terminal). For fully piped runs, use --progress, --json-progress, or METAGIT_PROGRESS. To force silence when stdout is a TTY, use --quiet or METAGIT_PROGRESS=off.

Commands that honor progress and safety flags: commit, diff (--live or --from/--to), push, pull, clone, checkout, promote, deploy plan, deploy apply, rollback plan, rollback apply, release-bundle plan, release-bundle apply.

Integrity gates: Declarative rulesets + optional read-only .mjs checks in .magit/pipeline.json (integrity.rulesets, per-env integrityRuleset, optional requiresIntegrity). Gates run automatically around release create, deploy/promote/release-bundle plan, and apply (rollback apply also runs preRollbackApply / postRollbackApply). magit integrity (validate-config, explain, run) is for inspection and manual runs. Reports live under .magit/store/integrity-reports/. Apply-style commands support --skip-integrity unless the environment sets requiresIntegrity: true. Details: specs §6.1b (integrity subsection).

Apply-style flags: deploy/promote/rollback/release-bundle apply accept --confirm-large-target when the target environment sets requireConfirmAboveEstimatedDocs and the preflight estimate exceeds it (see magit env add). They also accept --skip-integrity where allowed (see integrity gates above).

Core commands (v1)

| Command | Purpose | |--------|---------| | init | Create .magit/, config, local store; optional GCS remote/* bootstrap | | commit | Snapshot current Mongo state into a commit | | push / pull | Sync object store with .magit/store | | clone | init + pull from a gs://… URL (use --gcs-credentials-base64-env when using SA-in-env, same as init) | | checkout | Apply a commit (indexes optional) | | diff --live | Compare HEAD to live Mongo | | diff --from <h> --to <h> | Compare two object-store commits | | log | Print commits from HEAD (first-parent chain); -n / --max-count; --json |

Initialize (Mongo + GCS; env vars hold secrets, not the config file):

magit init \
  --driver mongo \
  --url-env MONGO_URI \
  --db mydb \
  --name my-repo \
  --gcs-bucket my-bucket \
  --gcs-base-path my-prefix \
  --gcs-credentials-base64-env GOOGLE_SERVICE_ACCOUNT_BASE64 \
  --env local

Initialize with X12I-managed HTTP storage (customers keep Mongo credentials locally/CI-side; MaGit objects go to your storage API):

export X12I_MAGIT_TOKEN="..."

magit login \
  --storage-url https://api.x12i.com/magit-storage \
  --token-env X12I_MAGIT_TOKEN \
  --tenant my-workspace

magit init \
  --driver mongo \
  --url-env MONGO_URI \
  --db mydb \
  --name my-repo \
  --storage-provider x12i-http \
  --env dev

The command workflow stays the same after initialization:

magit commit -m "update metadata"
magit push
magit pull
magit release create catalog-v3 --commit HEAD --from dev
magit deploy plan --release catalog-v3 --to production

The HTTP storage API is expected to expose repo object operations:

GET    /v1/repos/:repoId/objects?prefix=<prefix>  -> { "keys": ["..."] }
HEAD   /v1/repos/:repoId/objects/:objectPath
GET    /v1/repos/:repoId/objects/:objectPath
PUT    /v1/repos/:repoId/objects/:objectPath
DELETE /v1/repos/:repoId/objects/:objectPath

When --storage-token-env is set, MaGit sends Authorization: Bearer <token>. When --storage-tenant is set, MaGit sends x-metagit-tenant.

For local development, this repo includes a filesystem-backed reference storage API:

export PORT=4000
export X12I_MAGIT_TOKEN="dev-token"
export X12I_MAGIT_TENANT="demo-workspace"
export X12I_MAGIT_STORAGE_ROOT="/private/tmp/x12i-magit-storage"

npm run storage:dev

Useful init options (all optional):

• --gcs-root-folder <segment> — first path segment under the bucket for keys (default: metagit)
• --strict-remote — fail if remote/* cannot be written to remote storage (default: warn and continue)
• --remote-secrets-json <path> — upload wrapped secrets to GCS (high risk; strict IAM)
• --repo-metadata-json <path> — non-secret operator metadata (tier, purpose, hosting, …); see specs §6.1

After init, remote/repo.v1.json and remote/client-hints.v1.json are written under .magit/store/remote/ and uploaded when GCS is reachable. Each push uploads missing immutable objects, overwrites mutable refs and workflow manifests, and refreshes lastPushedAt on the remote copy. Before overwriting refs, push rejects if the remote ref is not already contained in the local commit history; run magit pull first or use magit push --force when intentionally replacing remote state. Layout and behavior are documented in specs §5.3–5.4.

Optional repo metadata

You can document deployment tier, purpose, related systems, hosting, exposure, access notes, etc. (never secrets) via --repo-metadata-json or the repo.metadata field in .magit/config.json. The same block is mirrored on remote/repo.v1.json when present. Details: specs §6.1.

Optional record annotations

Opt-in Mongo metadata under _<name> (default _metagit; name is configurable). It is written only when applyCommit runs—i.e. pipeline deploy / promote / rollback / release-bundle apply, not when you commit alone. Enable at magit init (--record-annotation, --record-annotation-name, …) or via recordAnnotation in .magit/config.json, with optional overrides on pipeline environments. Behavior (merge rules, diff stripping, fast path): specs §2.1b.

Pipeline layer (environments, releases, deploy, rollback, merge, promote, release-bundle)

The pipeline layer moves a selected MaGit commit across named MongoDB environments in a controlled, auditable, reversible way.

Command groups

magit env
magit release
magit release-bundle
magit deploy
magit rollback
magit merge
magit promote
magit integrity

Configure environments (stored in .magit/pipeline.json; Mongo URIs are referenced by env var name, not stored inline)

magit env add dev \
  --mongo-url-env MONGO_DEV_URI \
  --db mydb

magit env add production \
  --mongo-url-env MONGO_PROD_URI \
  --db mydb \
  --protected \
  --requires-plan \
  --requires-confirmation \
  --requires-pre-deploy-snapshot \
  --hosting cloud \
  --preflight-warn-estimated-docs 500000 \
  --require-confirm-above-estimated-docs 2000000

# Optional: attach a named integrity ruleset and require gates (see specs §6.1b)
# magit env add production ... --integrity-ruleset production-safe --requires-integrity

Optional --deploy-metagitignore / --deploy-metagitignore-extra: repo-relative paths to additional ignore files merged after .metagitignore when planning/applying to that environment only.

If two environment names share the same mongoUrlEnv + db, MaGit emits a warning (often accidental aliases).

Create a version in dev and deploy it to production

# Create a commit from dev (whatever `.magit/config.json` points at)
magit commit -m "prepare catalog v3"
magit push

# Name it as a release
magit release create catalog-v3 --commit HEAD --from dev

# Plan (read-only)
magit deploy plan --release catalog-v3 --to production

# Apply
magit deploy apply --release catalog-v3 --to production --plan <planHash>

Mandatory pre-deploy snapshot (rollback point)

Before deploy apply modifies the target environment, MaGit will:

• create a pre-deploy snapshot commit of the target environment (e.g. production)
• push it successfully
• store it as beforeCommit in deployments/<deploymentId>.json

That beforeCommit is the rollback point.

Rollback

magit rollback plan --deployment <deploymentId>
magit rollback apply --deployment <deploymentId> --mode exact

Merge before deploy

Merge creates a new MaGit commit first (it does not modify any Mongo environment).

magit merge plan --left env:dev --right env:production
magit merge resolve --merge <mergeId> --conflict users/64f... --take right
magit merge create --merge <mergeId> -m "merge production hotfixes into dev"

Promote shortcut

# Plan only
magit promote dev production --plan

# Apply (requires confirmation for protected env)
magit promote dev production --apply

Release bundle (monorepo)

# From a repo that will store the bundle manifest (cwd has .magit):
magit release-bundle create catalog-2026 --root /path/to/workspace --component packages/svc-a --component packages/svc-b

magit release-bundle plan --bundle catalog-2026 --to staging
# Save JSON map label -> planHash for environments with --requires-plan
magit release-bundle apply --bundle catalog-2026 --to production --plans-json ./plans-by-label.json

Large databases: progress, preflight, safety

For large MongoDB deployments and slow networks, MaGit reports how big the job is before heavy work and lets you bound batching and concurrency.

Preflight

Before the main loops, applicable commands emit a preflight progress phase whose phase-end summarizes scope (e.g. collection counts, sums of Mongo estimatedDocumentCount, record counts from manifests, file/object counts for sync). Human stderr prints a Preparing line with summary …; --json-progress emits phase-end with the same fields under meta. Details and keys: specs §38a.2.1.

Examples

# Human progress on stderr (default in auto mode when stderr or stdout is a TTY, or FORCE_COLOR)
magit promote dev production --apply

# NDJSON on stderr for CI / wrappers (stdout unchanged)
magit --json-progress promote dev production --apply 2>events.ndjson

# Final JSON on stdout + progress NDJSON on stderr
magit --json --json-progress deploy apply --release catalog-v3 --to production --plan <hash> \
  2>events.ndjson | jq .

# Silence progress
magit --quiet commit -m "snapshot"

# Throttle tick spam (or use 0 for every tick)
magit --progress-interval-ms 250 commit -m "snapshot"

Safety flags (on relevant subcommands)

Defaults are conservative (e.g. --max-collection-concurrency default 1). Full tables and tuning guidance: docs/large-databases.md.

| Flag | Role | |------|------| | --batch-size <n> | Records per bulkWrite / deleteMany chunk in apply (default 1000) | | --max-collection-concurrency <n> | Parallel collections during apply (default 1) | | --throttle-ms <n> | Pause between apply batches (default 0) | | --cursor-batch-size <n> | Mongo read cursor batch size (default 1000) | | --max-time-ms <n> | Mongo maxTimeMS on batched operations | | --mongo-max-pool-size <n> | Driver pool (also MONGO_MAX_POOL_SIZE) | | --mongo-socket-timeout-ms <n> | Also MONGO_SOCKET_TIMEOUT_MS | | --mongo-server-selection-timeout-ms <n> | Also MONGO_SERVER_SELECTION_TIMEOUT_MS | | --push-concurrency <n> | Parallel uploads in push (default 4) | | --force | Push only: allow overwriting divergent remote refs | | --pull-concurrency <n> | Parallel downloads in pull / clone (default 4) | | --dry-run | Apply paths: compute progress, no Mongo writes | | --transaction | Apply paths: one Mongo transaction across eligible record writes; requires --indexes skip | | --snapshot-read | Commit only: read records through one Mongo snapshot transaction when supported | | --resume <deploymentId> | Skip collections already listed in .magit/store/checkpoints/<id>.json | | --confirm-large-target | Acknowledge large target when env requireConfirmAboveEstimatedDocs would otherwise abort (deploy apply, rollback apply, promote --apply, release-bundle apply) |

Apply-style commands (deploy apply, rollback apply, promote --apply, release-bundle apply) pass the active deploymentId (or rollback id) as the resume key automatically; use --resume to target another checkpoint explicitly.

Checkpoints and --records upsert-only|exact: see docs/large-databases.md. Progress schema and phase names: specs §38a. Failures include an error progress event with a magit rollback apply --deployment <id> hint where applicable.

Library

import {
  magit,
  createProgressReporter,
  noopProgressReporter,
  resolveSafetyOptions,
  type MetagitRepoMetadataV1,
  type DiffBetweenCommitsResult,
  type ProgressEventV1,
  type ProgressReporter,
  type SafetyOptions
} from "@x12i/magit";

const repo = await magit.open();
const diff = await repo.diffLive();
console.log(diff);

// Optional: pass a reporter + safety into repo methods (CLI does this for you).
const reporter = createProgressReporter({ mode: "json", intervalMs: 1000 });
reporter.start("commit", {});
try {
  await repo.commit({
    message: "snapshot",
    reporter,
    safety: resolveSafetyOptions({ batchSize: 500, cursorBatchSize: 1000 })
  });
} finally {
  reporter.end();
}

// Compare two object-store commits (no Mongo). Result includes `schemaVersion: "metagit.diffBetweenCommits.v1"`.
const between: DiffBetweenCommitsResult = await repo.diffBetweenCommits({
  from: "<commit-hash-a>",
  to: "<commit-hash-b>",
  recordFull: false
});
console.log(between);

CLI equivalents: magit diff --from <hash-a> --to <hash-b> (add --json and/or --record-full as needed). Default magit diff / magit diff --live compare HEAD to live MongoDB.

Also exported: humanProgressReporter, jsonProgressReporter, PhaseHandle, ProgressMode, ProgressEventType, ResolvedSafetyOptions, safetyDefaults, ApplyCheckpointV1, DiffBetweenCommitsOptions, DiffCommitToLiveOptions, DiffCollectionDelta, DiffLiveOptions, DiffLiveResult, DeployIgnoreLoadOptions, mergeMetagitIgnoreRules, loadEffectiveDeployIgnoreRules, eligibleCommitCollections, recordIncludeFilterForCollection, deployIgnoreLoadOptionsFromPipelineEnv, pipeline helpers (resolvePipelineConfigPath, duplicateMongoTargetWarningLines, …), release-bundle types, and CreateProgressReporterOptions.

Use repo.listCommitsFromHead({ maxCount }) for commit history; repo.diffCommitToLive / repo.applyCommit accept deployIgnore and optional deployPreflightContext for library-driven deploy tooling.

MetagitRepoMetadataV1 is optional metadata you can attach at init or in config; see specs §6.1. Diff shapes are documented in specs §24.5–24.5.1.

Develop / test

From the package repository root, typical env vars are MONGO_URI, STORAGE_BUCKET, and optional GOOGLE_SERVICE_ACCOUNT_BASE64 (see scripts/live-smoke.mjs and tests/scenarios/*/.env patterns). Scenario tests are env-aware: tests/run-all.mjs runs scenarios whose required external env vars are present and reports skipped scenarios explicitly.

| Script | What it runs | |--------|----------------| | npm test | typecheck → build → tests/run-all.mjs → npm pack --dry-run | | npm run test:cli | build + scenario tests only (no pack:check) | | npm run typecheck | TypeScript --noEmit | | npm run build | tsup → dist/ | | npm run dev | tsup --watch | | npm run pack:check | npm pack --dry-run with repo-local npm cache/logs (publish tarball dry run) | | npm run live:smoke | Build + end-to-end Mongo + GCS smoke (scripts/live-smoke.mjs) | | npm run pipeline:smoke | Build + pipeline smoke (scripts/pipeline-smoke.mjs; expects an initialized .magit or legacy .metagit at repo root and MONGO_DEV_URI / MONGO_PROD_URI) | | npm run cleanup:test-dbs | Drop disposable test Mongo DBs: test-metagit_cli_*, test-metagit_live_test_*, and legacy unprefixed names (uses repo .env MONGO_URI). Add --dry-run on the script to list only. |

Integration scenarios use disposable DB names prefixed with test- (e.g. test-metagit_cli_<scenario>_<timestamp>). Each scenario drops its Mongo database after a successful run. Use npm run cleanup:test-dbs for leftovers from failed runs, old unprefixed names, or scripts like npm run live:smoke (which does not auto-drop).

npm test

Shell completions for deploy / rollback subcommands ship under scripts/completions/ (magit.bash, magit.zsh; legacy metagit.* names are kept as aliases). Source the appropriate file from your shell rc after the usual completion setup (compinit on zsh).

Publishing notes

• Rebrand release: publish @x12i/magit as the primary package and keep the metagit binary alias during the transition. If @x12i/metagit remains published, make the next release a deprecation/redirect package or clearly document the last supported version there.

• 1.8.0 aligns deploy / diff-to-live / exact apply with .metagitignore (plus optional per-env deploy ignore merges), adds METAGIT_PIPELINE_CONFIG, duplicate-env warnings, hosting / preflight thresholds / --confirm-large-target, magit log, magit release-bundle, and expanded library exports. Breaking vs older previews: deploy semantics on targets previously scanned whole collections for live diffs.

• 1.6.0 adds repo.diffBetweenCommits, magit diff --from / --to, and named TypeScript exports for diff-related types (DiffBetweenCommitsResult, DiffLiveResult, DiffCollectionDelta, …).

• Later 1.6.x adds stderr progress (metagit.progress.v1), preflight scope summaries, safety knobs (batching, concurrency, throttles, checkpoints, --dry-run / --resume), GCS upload/download concurrency, and Mongo driver options via flags and env vars; see specs §38a and docs/large-databases.md.

• Auto progress: besides stderr TTY, auto enables human lines when stdout is a TTY or FORCE_COLOR is truthy; METAGIT_PROGRESS overrides when the CLI stays on auto. Scripts that require a silent stderr should pass --quiet or METAGIT_PROGRESS=off.

• The package is configured as public via publishConfig.access = "public".

• Published files: dist/, docs/specs.md, docs/large-databases.md, docs/open-feature-requests.md, scripts/completions/, README.md, LICENSE (see package.json files).

• Do not commit a real .npmrc token file; use .npmrc.example if you document registry setup.

• prepublishOnly runs build, typecheck, and pack:check (no live Mongo required).

• Run npm test before tagging a release when you change behavior that touches Mongo or GCS.