@x12i/magit

Git-like version control for structured metadata repositories — a MongoDB database (the primary, DB-as-repo source) or a folder of JSON / Markdown files.

MaGit versions the full selected source state after applying .magitignore (records included unless excluded). For MongoDB it snapshots the selected DB state; for the folder driver it snapshots JSON/Markdown records under a directory. It is not positioned as a generic operational database backup system.

Hosted: an Authix-authenticated storage service and a self-service token dashboard are available at magit-storage.x12i.com / magit.x12i.com — see Hosted usage.

Sync vs pipeline deploy (mental model)

| Layer | What it is | Analog | |-------|----------------|--------| | Object store sync | commit snapshots eligible Mongo data into immutable objects; push / pull / clone move that history with GCS (your “remote”). Content-addressed objects are immutable; refs and workflow manifests are overwritten on push. | Git commits + fetch/push; another checkout is like another clone/pull of the same repo. | | Pipeline deploy | Named releases point at commits; deploy / promote / rollback apply a chosen commit’s manifests onto a named environment (another Mongo), with plans, protections, and checkpoints. | CI/CD promoting a release artifact to QA/staging/prod—not “clone the repo onto the server.” |

Managed slice on targets: For deploy/diff-to-live/applyCommit in exact mode, live Mongo reads and extraneous-record detection use the same rules as diff --live: .magitignore at the repo root (legacy .metagitignore is still honored), optionally merged with per-environment deployMetagitignorePath / deployMetagitignoreExtraPath in .magit/pipeline.json (legacy .metagit/pipeline.json fallback; see magit env add --help). Collections excluded by those rules are skipped for deploy even if present in the commit object.

Multi-repo releases: Use magit release-bundle to capture the same release name across several MaGit repos under one workspace, then release-bundle plan / release-bundle apply toward a shared pipeline environment (each component repo must define that environment). Bundle manifests live under .magit/store/release-bundles/ in the repo where you run release-bundle create.

Release slots: A release can name sub-slices of one repo's records (metagit.release-slots.v1) so a product version is deployable and rollback-able by component, not just by commit pointer: magit release create catalox-v3 --commit HEAD --from dev --slot planner=agentId:plnr@2 --slot researcher=agentId:rsch@1 (also --slots-json <path> and --auto-slots by=<field>). Then deploy plan/apply --slot planner, rollback apply --slot planner, and diff --release A --release B --by slot. A slot selector is the same descriptor as a record-slice include, ANDed onto the slice at the same scan seam — so a slot-scoped apply never reads, writes or deletes another slot's records, and can never reach outside the repo's slice (a selector that tries is rejected at authoring time). Slot version is a UX label only: it moves no hash and no gate. Opt-in metagit/slotsCoverSlice fails when a sliced record belongs to no slot or to more than one. Bundles group repos; slots group records within a repo — release-bundle apply is unchanged. Details: specs §7.6.

Partial commit / sparse apply (--only): A commit is a full-state snapshot — right for integrity, wrong ergonomics for "I changed one entry". magit commit --only <selector> [--base <ref>] writes a commit that equals its base (HEAD by default) with the selected records replaced by live values; everything else inherits base verbatim, by reference — unselected record hashes are copied off the base manifest and their bodies are never re-read, and a collection the selection did not touch keeps base's collectionSnapshotHash as-is (no collectionMeta round-trip). The result is still a valid full snapshot: it just differs from base only in the selection. Symmetrically, deploy plan/apply --only <selector> applies only the selected records and leaves every other record in the target untouched — on the transform driver that is a read-modify-write of the one container document, siblings intact, guarded by compare-and-swap. Delete semantics are explicit: --only is upsert-only, and --sparse-delete removes target records that match the selection and are absent from the commit, scoped to selection ∩ slice — never a slice-wide sweep (it is refused without --only). Selectors reuse the record-slice vocabulary — kind=persona, agentId:plnr, kind=persona,tool, agentId^=plnr-, and id=<canonical record id> — compiled by the same compiler, ANDed after slice and slot so --only can only narrow; one aimed outside the slice is rejected at authoring time. Library: repo.commit({ only, base }), repo.applyCommit({ only, sparseDelete }), and a selection block (matched counts, delete candidates) on the plan. Without --only, commit and apply are byte-identical to before, commit hashes included.

Proposals / patch import (metagit.patchset.v1): A producer — an automated agent, a UI, a migration script — computes "set persona greeter to X, delete tool legacy-search" and needs it to become a reviewable MaGit change. On mongo/transform repos there is no filesystem working tree to edit and commit from, and writing it live first defeats the purpose. So the change arrives as data: repo.proposePatchset(patchset) validates a metagit.patchset.v1 ({base, provenance, ops:[{op:"put"|"delete", collection?, id?|selector?, record?}], idempotencyKey}) and turns it into an overlay commit, returning its hash. Nothing goes live: no source session is opened at all, HEAD does not move, no ref is written — the proposal exists because it is content-addressed, and installing it stays the separate gated deploy / sparse-apply path. Ops name a record by canonical id or a set of base records by selector (the record-slice vocabulary again, same compiler), apply in order (last op wins), and are validated against the repo's slice before a single object is written — a put whose body, or a selector whose reach, falls outside the slice is refused with code: "out-of-slice", so a proposal can never smuggle in another team's records. Idempotent by idempotencyKey: resubmitting returns the identical commit hash with zero work; reusing a key for different content is refused. Optional { integrity: { ruleset } } runs an SDK-4 ruleset inline against the proposal's objects and returns findings, so a producer fails fast. CLI: magit import --patchset ./p.json imports one; magit import --from-diff <a> <b> turns a computed diff into a patchset applied to <a> — the labeled diff keeps collection + record id, so the round trip reproduces <b>'s records hash for hash — and --dry-run --out p.json emits the patchset as the reviewable artifact without creating a commit. Details: specs §7.8.

Labeled diff (operator view): Record deltas key on collection + record id, which reads as memorix_metadata_agents / plnr/persona/greeter. Configure diffLabels per collection in .magit/config.json (metagit.diffLabels.v1) to render the same deltas the way operators think — "persona greeter on agent planner changed":

{ "diffLabels": { "memorix_metadata_agents": {
    "group": "{agentId}", "label": "{kind}: {id}", "order": ["persona","tool","policy"] } } }

Then magit diff --by kind groups and orders the output, --group <field> regroups ad hoc, --label-template "{kind}: {id}" overrides the label, and --by slot groups by release slots (--slots-from <release> to group a live or commit diff by that release's slots). Labels are a pure projection over the existing diff shared by the CLI and library, so any future UI renders identically; label/group are additive keys, and a repo with no diffLabels produces byte-identical diff output. A field a record cannot answer renders as ? rather than crashing or dropping the delta; kinds absent from order sort after every listed one. Details: specs §7.7.

Soft locks (optional — most platforms should NOT use these): If you are embedding MaGit in a platform, own locks yourself. A platform with its own database and a per-session identity expresses "one editor at a time" with a unique index plus a TTL field, in a few lines, and — decisively — it knows who the user is. MaGit does not: it sees a CLI invocation and a saved auth profile. Locks are therefore P3/optional, excluded from the reference integration, and offered here for standalone / product use. What they are: advisory claims over a slice, slot or record (metagit.lock.v1) — magit lock acquire "agents:agentId=plnr" --ttl 30m, magit lock list, magit lock release <id>. The scope grammar is the record-slice vocabulary again (agents, agents:agentId=plnr, agents:agentId^=pl, agents:id=<recordId>, *) and overlap is decided with the same compiler and containment math slices/slots/--only use — where it cannot prove two selectors disjoint it warns anyway. Advisory: an overlapping acquire warns naming the holder by default; only a lock taken with --mode block refuses one, and --steal always wins and records who stole from whom. TTL is mandatory (default 30m, max 30d) and expiry is automatic — an expired lock blocks nothing and is pruned by the next write. A lock is coordination, never authorization: acquiring requires the same write capability a write requires (a read-only token cannot lock), and holding one grants nothing — no code path anywhere consults lock state when deciding whether an action is permitted. State is one small versioned object at .magit/locks.v1.json (outside store/, never pushed) guarded by an optimistic version check — no storage-API compare-and-set is required, and a repo that never locks has no lock file and behaves byte-identically. Details: specs §7.9.

Pipeline config location: Default is .magit/pipeline.json (legacy .metagit/pipeline.json fallback). Override with env var METAGIT_PIPELINE_CONFIG (absolute path)—useful when CI generates the file per run.

Documentation: Embedding MaGit as an SDK · Test guide · Auth experience · Authix token payload · Permission model · Design specification · Large databases & operators · Appendix · Open feature requests (large deployments) · Planning: Storage tier · Public API (token)

Install

npm i @x12i/magit

Runtime: Node.js ≥ 20. Integration tests and smoke scripts expect MongoDB and Google Cloud Storage (see Develop / test).

Compatibility names

MaGit is the public package and CLI name. New local checkouts use MaGit names, while existing Metagit-era repos are still read in place:

• Local repo directory: .magit/ (legacy .metagit/ fallback)
• Ignore file: .magitignore (legacy .metagitignore fallback)
• Environment variables: METAGIT_*
• JSON schemas and progress events: metagit.*.v1
• Built-in integrity rule keys: metagit/... (grouped-view payload: metagit.integrityRecordSet.v1; external validator request: metagit.integrityValidatorRequest.v1)
• Default GCS root folder: metagit
• Legacy CLI alias: metagit

Treat the remaining metagit.* values as wire-format identifiers. They should change only through a schema/storage migration that can read old repos and write the new layout intentionally.

Source drivers

MaGit reads the versioned state through a source driver (--driver at init). Two are built in:

| Driver | Source | Notes | |--------|--------|-------| | mongo (default) | A MongoDB database | Sub-collections map 1:1; supports index apply, transactions, snapshot reads, record annotations, incremental commit, and --changed-only apply. | | folder | A directory of files | Each sub-directory is a "collection"; each *.json is one record (id = the collection-relative path minus extension), each *.md / *.markdown becomes { frontmatter, body } (the raw frontmatter text is hashed for byte-stable round-trips). No database required. | | transform | A base driver + a bidirectional lens | Versions a nested document as many records without reshaping the database: explode on read, implode on write. See below. |

Transform driver — nested documents as records

When one document holds a nested pack of many logical entries (an "agent" document containing many (kind, id) entries, say), document-granular versioning is useless: one field change rewrites the whole document and diffs are opaque. The transform driver wraps a base driver with a lens that explodes each container document into one record per entry and imploded them back on write — leaving the stored shape untouched.

magit init --driver transform --base mongo --url-env MONGO_URI --db mydb --name my-repo

{ "driver": { "kind": "transform",
    "base": { "kind": "mongo", "urlEnv": "MONGO_URI", "db": "mydb" },
    "lens": { "agents": {
      "docKey": "agentId",
      "explode": { "path": "packs", "recordId": "{agentId}/{kind}/{id}",
                   "recordFields": ["kind","id"], "carry": ["agentId"] },
      "preserve": "passthrough" } },
    "encodingProfile": "current" } }

• Round-trip is the guarantee: implode(explode(doc)) is BSON-equal to doc. Agent-level (non-pack) fields ride in a hidden container envelope so the document is reconstructed exactly; the pack is replaced in place so field order — and therefore BSON layout — survives.
• Atomic container writes: every record of one docKey implodes and writes in a single upsert, and --batch-size counts containers for this driver. A sparse single-entry write re-reads, splices and writes under compare-and-swap, retrying on conflict — it never clobbers a sibling entry.
• Encoding: run magit driver audit-encoding --collection <c> --pack-path packs (read-only; reports BSON types and field paths, never values). A current verdict means the driver ships as-is; needs-shim sets encodingProfile: "ejson-fields", a narrow per-field EJSON shim over entry fields only — not a repo format change and not a history migration. magit doctor encoding surfaces the active profile.
• Escape hatch: --lens-module ./lens.mjs exporting explode / implode / recordId / parseRecordId + capabilities, loaded like integrity .mjs rules, for shapes the declarative lens cannot express.
• Record slices apply post-explode, so a kindIn-style selector works over nested storage. Details: specs §7.8.

# Mongo (DB-as-repo)
magit init --driver mongo --url-env MONGO_URI --db mydb --name my-repo

# Folder of JSON / Markdown (no database)
magit init --driver folder --path ./data --name my-repo

commit / push / pull / clone / diff and the pipeline commands work the same across drivers; capabilities a driver lacks (e.g. index apply on the folder driver) are skipped automatically. The default magit diff compares the working state to HEAD.

CLI overview

npx magit --help
npx magit --version

Machine-readable output (two channels)

| Flag | Stream | Purpose | |------|--------|---------| | --json | stdout | Final command result as JSON where the command supports it (e.g. magit --json diff --live, magit --json commit -m "…"). | | --json-progress | stderr | Newline-delimited JSON progress events (schemaVersion: "metagit.progress.v1") for long-running work. Independent of --json; you may use both together. |

Progress on stderr (human or NDJSON)

Long-running commands emit progress on stderr so stdout stays pipe-friendly when using --json.

| Root flag | Behavior | |-----------|----------| | (default, auto) | Human-readable lines when stderr or stdout is a TTY, or when FORCE_COLOR is enabled — otherwise quiet (typical CI with both streams piped) | | --progress | Force human progress even when stderr is not a TTY | | --no-progress | Disable human progress | | --quiet | Same as --no-progress (still overridden by --json-progress) | | --json-progress | NDJSON events on stderr (stable schema; see specs §38a) | | --progress-interval-ms <n> | Coalesce phase-tick events (default 1000). Use 0 to emit every tick. | | --verbose | Extra meta on human progress lines | | METAGIT_PROGRESS | When the CLI would otherwise use auto progress: human, json, or off (same as quiet). Overrides TTY / FORCE_COLOR heuristics. CLI flags above still win. |

npm / subprocess wrappers: many spawn Node with a non-TTY stderr; auto still prints human progress when stdout is a TTY (interactive terminal). For fully piped runs, use --progress, --json-progress, or METAGIT_PROGRESS. To force silence when stdout is a TTY, use --quiet or METAGIT_PROGRESS=off.

Commands that honor progress and safety flags: commit, diff (--live or --from/--to), push, pull, clone, checkout, promote, deploy plan, deploy apply, rollback plan, rollback apply, release-bundle plan, release-bundle apply.

Integrity gates: Declarative rulesets + optional read-only .mjs checks in .magit/pipeline.json (integrity.rulesets, per-env integrityRuleset, optional requiresIntegrity). Built-ins include source-validation rules for JSON/Markdown/folder repos (fileJsonSchema, markdownFrontmatterSchema, fileNamingConvention — run store-side or against a live source) and an auth-aware requiredScopes rule that gates deploys on the caller's Authix identity scopes (with allowMissingIdentity: fail|warn|skip); pair it with env-level requiredScopes in pipeline.json. Gates run automatically around release create, deploy/promote/release-bundle plan, and apply (rollback apply also runs preRollbackApply / postRollbackApply). magit integrity (validate-config, explain, run) is for inspection and manual runs. Reports live under .magit/store/integrity-reports/. Apply-style commands support --skip-integrity unless the environment sets requiresIntegrity: true. Rules are scoped to the records this repo owns (record-level .magitignore ∩ the record slice), through the same seam commit/diff/apply use — so a rule never sees, reports, or fails on another team's records in a shared collection, and noExtraneousRecords means extraneous within your slice; opt out per rule with includeRecordsOutsideRepoScope: true. Details: specs §6.1b (integrity subsection).

Grouped and remote validation (SDK-4, opt-in): The built-in rules judge one record at a time, which cannot express two real questions: is this pack valid as a whole (SDK-2 explodes it into records only so MaGit can version it), and does this record's reference resolve in the same version? metagit/recordSet builds the group and hands it to a validator — { "type": "recordSet", "id": "pack-valid", "groupBy": "agentId", "view": "logical", "module": "@scope/pkg/validate-pack", "severity": "fail" } — where view: "logical" reconstructs the container document through the repo's own driver.lens (the same implode a deploy writes through), view: "records" hands over the group's raw records, and groupBy: "*" is one group over everything. Findings normalize to { code, path, message, group }. metagit/externalValidator does the same against a running service: HMAC-signed (node:crypto), timeout-bounded so a hung validator never hangs a deploy, onUnreachable: fail|warn|skip mirroring allowMissingIdentity, and a payload bounded to the deploy selection ∩ your slice — validating what is being deployed, never shipping the rest of the repo to a third party. @x12i/magit-integrity-packlens ships ready validators (pack + generic cross-reference resolver) so enabling this is a config change, not code. Details: specs §6.1b.

Apply-style flags: deploy/promote/rollback/release-bundle apply accept --confirm-large-target when the target environment sets requireConfirmAboveEstimatedDocs and the preflight estimate exceeds it (see magit env add). They also accept --skip-integrity where allowed (see integrity gates above).

Core commands (v1)

| Command | Purpose | |--------|---------| | login / auth | Save a storage profile (hosted usage); auth status / verify / revoke / logout manage the token | | init | Create .magit/, config, local store; --driver mongo\|folder; optional GCS remote/* bootstrap | | commit | Snapshot current Mongo state into a commit | | push / pull | Sync object store with .magit/store | | clone | init + pull from a gs://… URL (use --gcs-credentials-base64-env when using SA-in-env, same as init) | | checkout | Apply a commit (indexes optional) | | diff --live | Compare HEAD to live Mongo | | diff --from <h> --to <h> | Compare two object-store commits | | log | Print commits from HEAD (first-parent chain); -n / --max-count; --json | | lock | Optional advisory soft locks: acquire <selector> --ttl / list / release <id> — coordination only, never access; an embedding platform should usually own locks itself (see Soft locks above) |

Initialize (Mongo + GCS; env vars hold secrets, not the config file):

magit init \
  --driver mongo \
  --url-env MONGO_URI \
  --db mydb \
  --name my-repo \
  --gcs-bucket my-bucket \
  --gcs-base-path my-prefix \
  --gcs-credentials-base64-env GOOGLE_SERVICE_ACCOUNT_BASE64 \
  --env local

Hosted usage (Authix-authenticated)

With the hosted storage API, customers keep their source (Mongo/folder) credentials locally/CI-side; MaGit objects go to the storage service, authenticated per-request by an Authix token with server-side workspace/repo/environment scope enforcement and revocation.

1. Get a token — open the dashboard https://magit.x12i.com, sign in (SSO), and generate a token under Tokens (choose a permission preset — viewer / developer / release-manager — plus workspace, repo, and environment scope). CI/service tokens can also be minted headlessly with the Authix token-cli. The token is shown once.

2. Log in and initialize:

export MAGIT_TOKEN="<token from the dashboard>"

magit login \
  --storage-url https://magit-storage.x12i.com \
  --token-env  MAGIT_TOKEN \
  --tenant     my-workspace          # must match the token's workspace

magit auth status                    # identity, scopes, workspaces, expiry
magit auth verify                    # live check against the storage API

magit init \
  --driver folder --path ./data \    # or: --driver mongo --url-env MONGO_URI --db mydb
  --name my-repo \
  --storage-provider x12i-http \
  --env dev

Manage the token from the CLI: magit auth status / verify inspect the saved profile; magit auth revoke revokes the current bearer token (storage pass-through, or Authix directly via --authix-url); magit auth logout deletes the local profile. Access is enforced by the token's scope — a request outside its workspace/repo/environment, or a mutating action beyond its features, is rejected 403; a revoked/expired token is 401 (within a short introspection-cache window).

The command workflow stays the same after initialization:

magit commit -m "update metadata"
magit push
magit pull
magit release create catalog-v3 --commit HEAD --from dev
magit deploy plan --release catalog-v3 --to production

The HTTP storage API is expected to expose repo object operations:

GET    /v1/repos/:repoId/objects?prefix=<prefix>  -> { "keys": ["..."] }
HEAD   /v1/repos/:repoId/objects/:objectPath
GET    /v1/repos/:repoId/objects/:objectPath
PUT    /v1/repos/:repoId/objects/:objectPath
DELETE /v1/repos/:repoId/objects/:objectPath

When --storage-token-env is set, MaGit sends Authorization: Bearer <token>. When --storage-tenant is set, MaGit sends x-metagit-tenant.

For local development, this repo includes a filesystem-backed reference storage API:

export PORT=4000
export X12I_MAGIT_TOKEN="dev-token"
export X12I_MAGIT_TENANT="demo-workspace"
export X12I_MAGIT_STORAGE_ROOT="/private/tmp/x12i-magit-storage"

npm run storage:dev

Useful init options (all optional):

• --gcs-root-folder <segment> — first path segment under the bucket for keys (default: metagit)
• --strict-remote — fail if remote/* cannot be written to remote storage (default: warn and continue)
• --remote-secrets-json <path> — upload wrapped secrets to GCS (high risk; strict IAM)
• --repo-metadata-json <path> — non-secret operator metadata (tier, purpose, hosting, …); see specs §6.1

After init, remote/repo.v1.json and remote/client-hints.v1.json are written under .magit/store/remote/ and uploaded when GCS is reachable. Each push uploads missing immutable objects, overwrites mutable refs and workflow manifests, and refreshes lastPushedAt on the remote copy. Before overwriting refs, push rejects if the remote ref is not already contained in the local commit history; run magit pull first or use magit push --force when intentionally replacing remote state. Layout and behavior are documented in specs §5.3–5.4.

Optional repo metadata

You can document deployment tier, purpose, related systems, hosting, exposure, access notes, etc. (never secrets) via --repo-metadata-json or the repo.metadata field in .magit/config.json. The same block is mirrored on remote/repo.v1.json when present. Details: specs §6.1.

Optional record annotations

Opt-in Mongo metadata under _<name> (default _metagit; name is configurable). It is written only when applyCommit runs—i.e. pipeline deploy / promote / rollback / release-bundle apply, not when you commit alone. Enable at magit init (--record-annotation, --record-annotation-name, …) or via recordAnnotation in .magit/config.json, with optional overrides on pipeline environments. Behavior (merge rules, diff stripping, fast path): specs §2.1b.

Pipeline layer (environments, releases, deploy, rollback, merge, promote, release-bundle)

The pipeline layer moves a selected MaGit commit across named MongoDB environments in a controlled, auditable, reversible way.

Command groups

magit env
magit release
magit release-bundle
magit deploy
magit rollback
magit merge
magit promote
magit integrity

Configure environments (stored in .magit/pipeline.json; Mongo URIs are referenced by env var name, not stored inline)

magit env add dev \
  --mongo-url-env MONGO_DEV_URI \
  --db mydb

magit env add production \
  --mongo-url-env MONGO_PROD_URI \
  --db mydb \
  --protected \
  --requires-plan \
  --requires-confirmation \
  --requires-pre-deploy-snapshot \
  --hosting cloud \
  --preflight-warn-estimated-docs 500000 \
  --require-confirm-above-estimated-docs 2000000

# Optional: attach a named integrity ruleset and require gates (see specs §6.1b)
# magit env add production ... --integrity-ruleset production-safe --requires-integrity

Optional --deploy-metagitignore / --deploy-metagitignore-extra: repo-relative paths to additional ignore files merged after .metagitignore when planning/applying to that environment only.

If two environment names share the same mongoUrlEnv + db, MaGit emits a warning (often accidental aliases).

Create a version in dev and deploy it to production

# Create a commit from dev (whatever `.magit/config.json` points at)
magit commit -m "prepare catalog v3"
magit push

# Name it as a release
magit release create catalog-v3 --commit HEAD --from dev

# Plan (read-only)
magit deploy plan --release catalog-v3 --to production

# Apply
magit deploy apply --release catalog-v3 --to production --plan <planHash>

Mandatory pre-deploy snapshot (rollback point)

Before deploy apply modifies the target environment, MaGit will:

• create a pre-deploy snapshot commit of the target environment (e.g. production)
• push it successfully
• store it as beforeCommit in deployments/<deploymentId>.json

That beforeCommit is the rollback point.

Rollback

magit rollback plan --deployment <deploymentId>
magit rollback apply --deployment <deploymentId> --mode exact

Merge before deploy

Merge creates a new MaGit commit first (it does not modify any Mongo environment).

magit merge plan --left env:dev --right env:production
magit merge resolve --merge <mergeId> --conflict users/64f... --take right
magit merge create --merge <mergeId> -m "merge production hotfixes into dev"

Promote shortcut

# Plan only
magit promote dev production --plan

# Apply (requires confirmation for protected env)
magit promote dev production --apply

Release bundle (monorepo)

# From a repo that will store the bundle manifest (cwd has .magit):
magit release-bundle create catalog-2026 --root /path/to/workspace --component packages/svc-a --component packages/svc-b

magit release-bundle plan --bundle catalog-2026 --to staging
# Save JSON map label -> planHash for environments with --requires-plan
magit release-bundle apply --bundle catalog-2026 --to production --plans-json ./plans-by-label.json

Large databases: progress, preflight, safety

For large MongoDB deployments and slow networks, MaGit reports how big the job is before heavy work and lets you bound batching and concurrency.

Preflight

Before the main loops, applicable commands emit a preflight progress phase whose phase-end summarizes scope (e.g. collection counts, sums of Mongo estimatedDocumentCount, record counts from manifests, file/object counts for sync). Human stderr prints a Preparing line with summary …; --json-progress emits phase-end with the same fields under meta. Details and keys: specs §38a.2.1.

Examples

# Human progress on stderr (default in auto mode when stderr or stdout is a TTY, or FORCE_COLOR)
magit promote dev production --apply

# NDJSON on stderr for CI / wrappers (stdout unchanged)
magit --json-progress promote dev production --apply 2>events.ndjson

# Final JSON on stdout + progress NDJSON on stderr
magit --json --json-progress deploy apply --release catalog-v3 --to production --plan <hash> \
  2>events.ndjson | jq .

# Silence progress
magit --quiet commit -m "snapshot"

# Throttle tick spam (or use 0 for every tick)
magit --progress-interval-ms 250 commit -m "snapshot"

Safety flags (on relevant subcommands)

Defaults are conservative (e.g. --max-collection-concurrency default 1). Full tables and tuning guidance: docs/large-databases.md.

| Flag | Role | |------|------| | --batch-size <n> | Records per bulkWrite / deleteMany chunk in apply (default 1000) | | --max-collection-concurrency <n> | Parallel collections during apply (default 1) | | --throttle-ms <n> | Pause between apply batches (default 0) | | --cursor-batch-size <n> | Mongo read cursor batch size (default 1000) | | --max-time-ms <n> | Mongo maxTimeMS on batched operations | | --mongo-max-pool-size <n> | Driver pool (also MONGO_MAX_POOL_SIZE) | | --mongo-socket-timeout-ms <n> | Also MONGO_SOCKET_TIMEOUT_MS | | --mongo-server-selection-timeout-ms <n> | Also MONGO_SERVER_SELECTION_TIMEOUT_MS | | --push-concurrency <n> | Parallel uploads in push (default 4) | | --force | Push only: allow overwriting divergent remote refs | | --pull-concurrency <n> | Parallel downloads in pull / clone (default 4) | | --dry-run | Apply paths: compute progress, no Mongo writes | | --transaction | Apply paths: one Mongo transaction across eligible record writes; requires --indexes skip | | --snapshot-read | Commit only: read records through one Mongo snapshot transaction when supported | | --resume <deploymentId> | Skip collections already listed in .magit/store/checkpoints/<id>.json | | --confirm-large-target | Acknowledge large target when env requireConfirmAboveEstimatedDocs would otherwise abort (deploy apply, rollback apply, promote --apply, release-bundle apply) | | --commit-write-concurrency <n> | Commit only: parallelize per-object exists-check + gzip + write (default 1). Hashing stays in cursor order, so commit hashes are unchanged. | | --incremental / --verify-full | Commit only (default off): reuse parent record hashes via record annotations to skip re-hashing unchanged records. --verify-full recomputes every reused hash and fails on a stale annotation. | | --changed-only | deploy apply only (default off): skip upserts whose live record hash already matches the commit; deletes still compare the full id set. Pair with a convergesToCommit integrity gate. | | --allow-slice-drift | deploy plan / apply-style commands (default off): proceed when the commit's sliceQueryHash differs from the target environment's compiled record slice. Aborts with code slice-drift otherwise, because exact-mode deletes would then be scoped by a different boundary. Logs loudly when used. See specs §7.5. | | --no-delta / --negotiation <auto\|list\|exists> | Push only: control bulk missing-object negotiation before upload (default auto). --no-delta forces per-object existence checks. | | --no-bundle / --bundle-max-bytes <n> / --bundle-max-entries <n> | Push only: control transport-only compressed upload bundles (defaults ~8 MB / 512 entries). The stored object layout is byte-identical either way. |

Apply-style commands (deploy apply, rollback apply, promote --apply, release-bundle apply) pass the active deploymentId (or rollback id) as the resume key automatically; use --resume to target another checkpoint explicitly.

Checkpoints and --records upsert-only|exact: see docs/large-databases.md. Progress schema and phase names: specs §38a. Failures include an error progress event with a magit rollback apply --deployment <id> hint where applicable.

Library

import {
  magit,
  createProgressReporter,
  noopProgressReporter,
  resolveSafetyOptions,
  type MetagitRepoMetadataV1,
  type DiffBetweenCommitsResult,
  type ProgressEventV1,
  type ProgressReporter,
  type SafetyOptions
} from "@x12i/magit";

const repo = await magit.open();
const diff = await repo.diffLive();
console.log(diff);

// Optional: pass a reporter + safety into repo methods (CLI does this for you).
const reporter = createProgressReporter({ mode: "json", intervalMs: 1000 });
reporter.start("commit", {});
try {
  await repo.commit({
    message: "snapshot",
    reporter,
    safety: resolveSafetyOptions({ batchSize: 500, cursorBatchSize: 1000 })
  });
} finally {
  reporter.end();
}

// Compare two object-store commits (no Mongo). Result includes `schemaVersion: "metagit.diffBetweenCommits.v1"`.
const between: DiffBetweenCommitsResult = await repo.diffBetweenCommits({
  from: "<commit-hash-a>",
  to: "<commit-hash-b>",
  recordFull: false
});
console.log(between);

CLI equivalents: magit diff --from <hash-a> --to <hash-b> (add --json and/or --record-full as needed). Default magit diff / magit diff --live compare HEAD to live MongoDB.

Also exported: humanProgressReporter, jsonProgressReporter, PhaseHandle, ProgressMode, ProgressEventType, ResolvedSafetyOptions, safetyDefaults, ApplyCheckpointV1, DiffBetweenCommitsOptions, DiffCommitToLiveOptions, DiffCollectionDelta, DiffLiveOptions, DiffLiveResult, DeployIgnoreLoadOptions, mergeMetagitIgnoreRules, loadEffectiveDeployIgnoreRules, eligibleCommitCollections, recordIncludeFilterForCollection, deployIgnoreLoadOptionsFromPipelineEnv, pipeline helpers (resolvePipelineConfigPath, duplicateMongoTargetWarningLines, …), release-bundle types, and CreateProgressReporterOptions.

Use repo.listCommitsFromHead({ maxCount }) for commit history; repo.diffCommitToLive / repo.applyCommit accept deployIgnore and optional deployPreflightContext for library-driven deploy tooling.

MetagitRepoMetadataV1 is optional metadata you can attach at init or in config; see specs §6.1. Diff shapes are documented in specs §24.5–24.5.1.

Hidden typed global package stores

The 2.9 package catalog is one shared, flat, untyped tree ({kind, id, version}). A typed global store adds a capability boundary and a structure-language boundary on top of it:

| Factor | Role | |--------|------| | MaGit token | Who you are + op scope | | storeSelector | Opens one store. Shared out-of-band, redacted in logs, no "list all stores" API | | packageTypeId | The store's structure language + validation policy. Wrong type → rejected even with the right selector | | publisher | Durable attribution on list/fetch/index. Not the hide boundary |

Hide-from-others = do not share the selector.

const store = await repo.createGlobalStore({
  packageTypeId: "memorix.agents.v1",
  allowedKinds: ["agent", "connector"]
});
// store.storeSelector is returned EXACTLY ONCE — MaGit keeps only a one-way hash

await repo.packPackageFR({
  storeSelector: store.storeSelector,
  packageTypeId: "memorix.agents.v1",
  kind: "agent",
  id: "plnr",
  message: "planner agent",
  payload: { agents: { plnr: { role: "planner" } } },
  publisher: { id: "memorix", appId: "app-1" }
});

const rows = await repo.listPackagesFR({
  storeSelector: store.storeSelector,
  packageTypeId: "memorix.agents.v1",
  kind: "agent"
});                                  // every row carries the durable publisher

await repoB.pullStore({ storeSelector: store.storeSelector });  // selector-scoped sync

packPackageFR / listPackagesFR / fetchPackageFR are overloaded: pass a storeSelector and the call is store-scoped, omit it and you get the 2.9 flat catalog byte-for-byte as before. Store objects live under pkgstores/, disjoint from packages/, and a plain pull() never transfers them — only pullStore(selector) does.

Also exported: rotateStoreSelector, revokeStoreSelector, openGlobalStore, and a metagit.packages Catalox index whose rows carry storeId (never a selector) and no payload.

Full reference, including the exact scope of the hiding guarantee and what is deliberately out of scope: docs/global-package-stores.md.

Develop / test

From the package repository root, typical env vars are MONGO_URI, STORAGE_BUCKET, and optional GOOGLE_SERVICE_ACCOUNT_BASE64 (see scripts/live-smoke.mjs and tests/scenarios/*/.env patterns). Scenario tests are env-aware: tests/run-all.mjs runs scenarios whose required external env vars are present and reports skipped scenarios explicitly.

| Script | What it runs | |--------|----------------| | npm test | typecheck → build → tests/run-all.mjs → npm pack --dry-run | | npm run test:cli | build + scenario tests only (no pack:check) | | npm run typecheck | TypeScript --noEmit | | npm run build | tsup → dist/ | | npm run dev | tsup --watch | | npm run pack:check | npm pack --dry-run with repo-local npm cache/logs (publish tarball dry run) | | npm run live:smoke | Build + end-to-end Mongo + GCS smoke (scripts/live-smoke.mjs) | | npm run pipeline:smoke | Build + pipeline smoke (scripts/pipeline-smoke.mjs; expects an initialized .magit or legacy .metagit at repo root and MONGO_DEV_URI / MONGO_PROD_URI) | | npm run cleanup:test-dbs | Drop disposable test Mongo DBs: test-metagit_cli_*, test-metagit_live_test_*, and legacy unprefixed names (uses repo .env MONGO_URI). Add --dry-run on the script to list only. |

Integration scenarios use disposable DB names prefixed with test- (e.g. test-metagit_cli_<scenario>_<timestamp>). Each scenario drops its Mongo database after a successful run. Use npm run cleanup:test-dbs for leftovers from failed runs, old unprefixed names, or scripts like npm run live:smoke (which does not auto-drop).

npm test

Shell completions for deploy / rollback subcommands ship under scripts/completions/ (magit.bash, magit.zsh; legacy metagit.* names are kept as aliases). Source the appropriate file from your shell rc after the usual completion setup (compinit on zsh).

Publishing notes

• Rebrand release: publish @x12i/magit as the primary package and keep the metagit binary alias during the transition. If @x12i/metagit remains published, make the next release a deprecation/redirect package or clearly document the last supported version there.

• 2.12.0 makes credentials store-shaped and both packaging modes explicit. A token may carry scope.custom.magit.stores = { storeIds, write }, and when it does, objects under pkgstores/ authorize by that grant instead of by repoIds — so a consumer environment can hold read-only access to one store without holding rights to the repo it lives in. write is opt-in (a store grant defaults to read-only), a grant never crosses a workspace, and it never manufactures a capability the token lacks. Safety rule: for a plain repo token an absent repos.repoIds means unrestricted, so a token carrying a store grant must name its repos explicitly — otherwise a "store-only consumer" credential would silently be granted every repo in the workspace. Authix needs no change; scope.custom is already free-form. Adds repo.listStores() for owner recovery — metadata only, never a selector, and requires repo-level access, so a lost selector no longer orphans a store (listStores → rotateStoreSelector({ storeId }) → pushStore). A 403 from hosted storage now carries the server's own reason plus the workspace, repo and storage it attempted, instead of discarding the body for generic advice. Documents Mode A (typed global store) and Mode B (flat repo catalog) as coexisting first-class paths, with tests asserting neither writes into the other's tree and neither requires the other.

• 2.11.0 closes the publish half of typed global stores. repo.pushStore({ storeSelector }) publishes exactly one store's objects — that selector's handle, the store's prefix, and any revocation tombstones naming its storeId — so publishing one package no longer means pushing an entire repository, and it is safe to retry because pkgstores/ is write-once. A store-scoped packPackageFR now publishes by default and its result carries published: in 2.10 a pack wrote locally only, so a version could exist on one machine's disk and be invisible to every other environment while reporting success. Pass publish: false to stage locally and publish later. StoreSelectorUnknownError now distinguishes wrong selector from not synced locally from not published centrally, three root causes that previously shared one message and have three different fixes. Behaviour change: a store-scoped pack that cannot publish now throws instead of silently succeeding.

• 2.10.0 adds hidden typed global package stores on top of the 2.9 catalog: repo.createGlobalStore() mints a store with a frozen packageTypeId (its structure language + enforced validation policy) and a high-entropy storeSelector — a bearer capability, returned exactly once, that opens one store. packPackageFR / listPackagesFR / fetchPackageFR are overloaded: pass a storeSelector and the call is store-scoped; omit it and the 2.9 flat catalog behaves byte-for-byte as before. A MaGit token alone reaches nothing in any store — there is no list-all-stores API, store objects live under pkgstores/ (disjoint from packages/), and a plain pull() never transfers them; only pullStore(selector) does. Every pack carries a durable publisher (explicit, or derived from actor, else fail-closed) that is part of the identity version hashes, so a reuse can never silently re-attribute a package. rotateStoreSelector() revokes the old capability without moving a byte. Ships a store-scoped Catalox package index (metagit.packages) whose rows carry storeId, never a selector, and never a payload. Hiding is an API-level capability boundary, not encryption at rest. Details, limits and the security notes: docs/global-package-stores.md.

• 2.9.0 adds a central package catalog: repo.packPackage() / listPackages() / fetchPackage() / restorePackage() write and read immutable, content-addressed packages (agent | connector | service | agent-bundle) under packages/, so one environment can pack an agent/connector/service (or a full-mode bundle pinning several) and another environment can fetch and install it with no shared database. version is the commit hash (lineage); checksum is content-only and is deliberately a different value from version, even for a byte-identical re-pack. Idempotent across environments — packing identical content (checksum + mode + pins + associations) reuses the existing version instead of minting a new one. Packages are immutable and not yet yankable. Details and limits: docs/sdk-integration.md §6–8.

• 2.8.0 adds advisory soft locks (magit lock acquire/list/release) — optional, and most embedding platforms should not use them: a platform with its own database and per-session identity is better off owning locks itself. MaGit's are repo-local (.magit/locks.v1.json, outside the object store, never pushed), advisory by default with an opt-in block mode and audited --steal, TTL-expiring, and overlap is detected with the same selector math as slices and slots. Locks never grant access — Authix scopes remain the sole authority, acquiring requires the same write capability a write requires, and holding a lock changes no authorization decision.

• 2.7.0 adds proposals: repo.proposePatchset() and magit import --patchset ./p.json turn a computed change (metagit.patchset.v1 — put/delete ops by id or selector, with provenance) into a reviewable overlay commit, returning its hash. Proposals are content-addressed — no branch, no ref, no HEAD movement — and they never touch the source: a proposal opens no driver session at all. Ops are validated against the record slice and rejected before any object is written, so a proposal cannot smuggle in another team's records; submissions are idempotent by idempotencyKey. magit import --from-diff <a> <b> converts a diff back into a patchset. Installing remains the separate gated deploy/sparse-apply path.

• 2.6.0 adds pack-level and external integrity: a recordSet rule that hands a validator a grouped view — including the imploded logical pack reconstructed through the transform lens — instead of one record at a time, so cross-referential checks ("this persona references a tool that must exist in this version") become a gate; and an externalValidator rule for validators that are a running service (HMAC-SHA256 signed, timeout-bounded, onUnreachable: fail|warn|skip, payload bounded to slice + selection so a validator only ever sees what is being committed or deployed). Ships packages/magit-integrity-packlens with ready rules and a ~3-line pipeline.json snippet. Existing rulesets and .mjs custom rules are unchanged.

• 2.5.0 adds overlay (partial) commits and sparse apply: magit commit --only <selector> [--base <ref>] produces a commit that is base plus the selected records replaced — still a complete full snapshot, with unselected collection hashes reused by reference and no re-read of unselected records; deploy plan/apply --only <selector> [--sparse-delete] applies just the selection, container-aware on the transform driver (one CAS'd container write, siblings intact). Deletes are opt-in and scoped to selector ∩ slice, never a slice-wide sweep. The plan's selection block is part of the hashed plan body, so a sparse plan cannot be replayed as a full apply. Additive: without --only, commit and apply are byte-identical.

• 2.4.0 adds the transform source driver (--driver transform): version a nested pack document as one record per entry without reshaping the database — declarative or .mjs module lens, container envelope preserving non-pack fields byte-exactly, atomic per-container writes, sparse single-entry writes under compare-and-swap, and current / ejson-fields encoding profiles selected by magit driver audit-encoding. Record slices apply post-explode. Mongo and folder repos are byte-identical.

• 2.3.0 adds the SDK track's operator surface: release slots (metagit.release-slots.v1 — deploy, roll back and diff a product version by component, slot selectors ANDed onto the record slice so they can only narrow) and labeled semantic diff (metagit.diffLabels.v1 — --by kind / --group / --by slot, a pure projection so CLI, library and any future UI agree). Both additive: releases without slots and repos without diffLabels are byte-identical to before.

• 2.2.0 scopes integrity rule evaluation to the records the repo owns (record-level .magitignore ∩ record slice), using the same seam as commit / diff / exact apply. Behavior change: in a repo that uses record-level ignore rules or a slice, noExtraneousRecords, jsonSchema and requiredFields previously read the whole collection and could flag — or fail a deploy over — records the repo does not own; they now see only owned records. This makes them agree with --records exact, which already refused to treat an out-of-slice record as a deletion candidate. Repos with neither ignore rules nor a slice are bit-identical. The old whole-collection view is available per rule via params.includeRecordsOutsideRepoScope: true, and is recorded in the check evidence.

• 2.1.1 fixes hosted-storage audit delivery: repoScope is sent as an array (Authix's schema rejected the bare string with 400, silently dropping every event), magit.token.used now fires for any served authorized request rather than only 2xx, permission_denied fires on every 403, and non-2xx audit responses are surfaced through onError instead of discarded.

• 2.1.0 adds record-scoped managed slices (metagit.record-slice.v1 in config, mirrored to remote/repo.v1.json and adopted by clone; Mongo push-down; slice-safe exact apply guarded by sliceQueryHash + --allow-slice-drift, with sliceScopedDeletes in plan output) and magit driver audit-encoding (read-only BSON-fidelity audit; reports types and field paths only, never values). Both are additive — a repo without a slice behaves byte-identically, and commit hashes are unchanged.

• 2.1.0 also adds multi-source drivers (--driver folder for JSON/Markdown alongside mongo, behind a unified SourceDriver interface), the Authix-authenticated hosted storage flow (magit login / auth status / verify / revoke / logout, server-side scope enforcement + revocation, self-service dashboard), faster sync (bulk missing-object delta negotiation + transport-only compressed upload bundles), faster commit/deploy (bounded write concurrency, gated --incremental commit, --changed-only apply, plus a scripts/bench/ harness), and source-validation + auth-aware integrity gates. Risky knobs default off; commit hashes and stored object layout are unchanged.

• 1.8.0 aligns deploy / diff-to-live / exact apply with .metagitignore (plus optional per-env deploy ignore merges), adds METAGIT_PIPELINE_CONFIG, duplicate-env warnings, hosting / preflight thresholds / --confirm-large-target, magit log, magit release-bundle, and expanded library exports. Breaking vs older previews: deploy semantics on targets previously scanned whole collections for live diffs.

• 1.6.0 adds repo.diffBetweenCommits, magit diff --from / --to, and named TypeScript exports for diff-related types (DiffBetweenCommitsResult, DiffLiveResult, DiffCollectionDelta, …).

• Later 1.6.x adds stderr progress (metagit.progress.v1), preflight scope summaries, safety knobs (batching, concurrency, throttles, checkpoints, --dry-run / --resume), GCS upload/download concurrency, and Mongo driver options via flags and env vars; see specs §38a and docs/large-databases.md.

• Auto progress: besides stderr TTY, auto enables human lines when stdout is a TTY or FORCE_COLOR is truthy; METAGIT_PROGRESS overrides when the CLI stays on auto. Scripts that require a silent stderr should pass --quiet or METAGIT_PROGRESS=off.

• The package is configured as public via publishConfig.access = "public".

• Published files: dist/, docs/specs.md, docs/large-databases.md, docs/open-feature-requests.md, scripts/completions/, README.md, LICENSE (see package.json files).

• Do not commit a real .npmrc token file; use .npmrc.example if you document registry setup.

• prepublishOnly runs build, typecheck, and pack:check (no live Mongo required).

• Run npm test before tagging a release when you change behavior that touches Mongo or GCS.