@x12i/metagit-storage

Metagit storage tier: repo-scoped blob operations on Google Cloud Storage, exposed as a TypeScript library (same operations as functions) and as a versioned HTTP API for deployment. Normative contract: docs/specs.md. Spec vs implementation: docs/spec-coverage.md.

Overview

• Library: Import @x12i/metagit-storage and call createGcsMetagitStorage / createGcsMetagitStorageFromEnv, then storage.forRepo(repoId) for exists, readBytes, writeBytes, list, delete (plus listPage and statBytes on the GCS adapter for pagination and HEAD sizing).
• HTTP service: Import @x12i/metagit-storage/server or run metagit-storage-server so clients use only URLs and auth; only this process needs GCS credentials (see spec §1).
• Backend: Google Cloud Storage via @google-cloud/storage.
• Runtime: Node.js ≥ 18, ESM.

Install

npm install @x12i/metagit-storage

Package entry points

| Import path | Use | |-------------|-----| | @x12i/metagit-storage | Types, GCS factory, env helper, credential decoding | | @x12i/metagit-storage/server | createMetagitStorageApp, createMetagitStorageAppFromEnv, startMetagitStorageServer | | metagit-storage-server (binary) | Standalone server after npm run build or global/local install |

Path and key layout

• Repo-relative paths in the API: no leading slash; segments use / (e.g. HEAD, refs/heads/main, commits/…). Same rules as docs/specs.md §2.
• GCS object key (internal): {STORAGE_OBJECT_PREFIX?}{repoId}/{objectPath}
  Optional STORAGE_OBJECT_PREFIX is a single tenant or root prefix (no trailing slash in env); do not expose a different path shape to HTTP clients.

Programmatic API (library)

Environment variables

| Variable | Required | Description | |----------|----------|-------------| | STORAGE_BUCKET | Yes | GCS bucket name (as shown in Cloud Console, not an https:// URL) | | GOOGLE_SERVICE_ACCOUNT_BASE64 | No | Base64-encoded Google service account JSON key. When omitted, the library uses Application Default Credentials (same idea as gcloud auth application-default login, Workload Identity, GOOGLE_APPLICATION_CREDENTIALS, etc.). | | FIREBASE_PROJECT_ID | No | GCP project id when using ADC, or overrides project_id from the JSON key when a base64 key is set | | GOOGLE_CLOUD_PROJECT / GCLOUD_PROJECT | No | Alternative env vars for default GCP project (ADC or alongside a key) | | STORAGE_OBJECT_PREFIX | No | Prefix before {repoId}/… inside the bucket |

From environment

With a base64 key in env (typical for CI or explicit keys), or without GOOGLE_SERVICE_ACCOUNT_BASE64 when the host already has ADC (GKE Workload Identity, local gcloud auth application-default login, etc.):

import { createGcsMetagitStorageFromEnv } from "@x12i/metagit-storage";

const storage = createGcsMetagitStorageFromEnv();
const repo = storage.forRepo("my-repo");

await repo.writeBytes("HEAD", Buffer.from("ref: refs/heads/main\n"), {
  contentType: "text/plain",
});
const head = await repo.readBytes("HEAD");
const keys = await repo.list("refs/");
await repo.delete("HEAD");

Explicit options

import {
  createGcsMetagitStorage,
  decodeServiceAccountJsonFromBase64,
} from "@x12i/metagit-storage";

const credentials = decodeServiceAccountJsonFromBase64(
  process.env.GOOGLE_SERVICE_ACCOUNT_BASE64!,
);

const storage = createGcsMetagitStorage({
  bucket: process.env.STORAGE_BUCKET!,
  credentials,
  projectId: process.env.FIREBASE_PROJECT_ID,
  objectKeyPrefix: "tenant-a",
});

const repo = storage.forRepo("my-repo");

Application Default Credentials (omit credentials so @google-cloud/storage resolves identity like @x12i/helpers/gcs without inline JSON):

import { createGcsMetagitStorage } from "@x12i/metagit-storage";

const storage = createGcsMetagitStorage({
  bucket: process.env.STORAGE_BUCKET!,
  // optional: pin project; otherwise the client uses env / metadata defaults
  projectId: process.env.GOOGLE_CLOUD_PROJECT,
  objectKeyPrefix: "tenant-a",
});
const repo = storage.forRepo("my-repo");

MetagitStorageAdapter (per repo)

| Method | Behavior | |--------|----------| | exists(path) | Whether an object exists at repo-relative path. | | readBytes(path, { maxBytes? }) | Full object bytes; throws if size > maxBytes when maxBytes is set. | | writeBytes(path, data, { contentType?, contentEncoding? }) | Create or overwrite the object. | | list(prefix) | All repo-relative keys under prefix (GCS pagination drained internally). | | delete(path) | Delete if present; idempotent (missing object is OK). |

GCS-specific helpers on forRepo(...)

| Method | Use | |--------|-----| | listPage(prefix, pageToken?) | One page; returns { keys, nextPageToken? } — mirrors HTTP list for custom clients. | | statBytes(path) | Size in bytes, or null if missing — used for HTTP HEAD / Content-Length. | | readBytesRange(path, start, end) | Inclusive byte range read (for HTTP Range); avoids loading the full object. |

Credentials helper

import {
  decodeServiceAccountJsonFromBase64,
  projectIdFromCredentials,
} from "@x12i/metagit-storage";

HTTP API (deployed service)

Start the server

Programmatic:

import { startMetagitStorageServer } from "@x12i/metagit-storage/server";

startMetagitStorageServer({ port: 8787 });

CLI (requires built dist/, e.g. after npm run build, or from a published install):

npx metagit-storage-server

Custom app (e.g. mount behind another server):

import { createMetagitStorageAppFromEnv } from "@x12i/metagit-storage/server";

const { app } = createMetagitStorageAppFromEnv();
// app.fetch(request) — Web Fetch API

Server environment variables

| Variable | Description | |----------|-------------| | Same as library | STORAGE_BUCKET; optional GOOGLE_SERVICE_ACCOUNT_BASE64 (omit for ADC); optional FIREBASE_PROJECT_ID, GOOGLE_CLOUD_PROJECT, GCLOUD_PROJECT, STORAGE_OBJECT_PREFIX | | PORT | Listen port (default 8787) | | METAGIT_STORAGE_AUTH_TYPE | none (default), basic, or bearer | | METAGIT_STORAGE_BASIC | For basic: literal user:password (metagit sends Authorization: Basic …) | | METAGIT_STORAGE_TOKEN | For bearer: shared secret (metagit sends Authorization: Bearer …) | | METAGIT_STORAGE_MAX_BODY_BYTES | Max PUT body size (default 64 MiB); larger bodies → 413 | | METAGIT_STORAGE_MAX_READ_BYTES | Optional; if set, GET/HEAD return 413 when the object is larger (checked before download). | | METAGIT_STORAGE_ALLOWED_REPO_IDS | Optional comma-separated repoId allowlist. When set, other repos → 403 (after auth). | | METAGIT_STORAGE_RATE_LIMIT_RPM | Optional positive integer: max requests per client per calendar minute (in-memory). Exceeded → 429 with Retry-After. Client key = first X-Forwarded-For hop, else X-Real-IP, else shared "global". Not suitable for multi-instance without a shared limiter or sticky routing. | | METAGIT_STORAGE_LOG_PATH_SHA256 | Set to 1 / true to log path_sha256 instead of raw path on access lines. |

Use none only behind a trusted network (e.g. VPC). Prefer TLS (https://) and Basic/Bearer in production per docs/specs.md §4.

HTTP routes (base /v1)

| Method | Path | Purpose | |--------|------|---------| | GET | /v1/repos/{repoId}/objects?prefix=…&pageToken=… | List keys under prefix; JSON { "keys": string[], "nextPageToken"?: string }. | | HEAD | /v1/repos/{repoId}/objects/{objectPath} | Exists + Content-Length when present (404 if missing). | | GET | /v1/repos/{repoId}/objects/{objectPath} | Raw body (404 if missing). Single Range: bytes=… supported (206 + Content-Range; unsatisfiable range → 416). Multiple ranges or invalid syntax → 200 full body. | | PUT | /v1/repos/{repoId}/objects/{objectPath} | Raw body; headers Content-Type (default application/octet-stream), optional Content-Encoding. 413 if body over max. | | DELETE | /v1/repos/{repoId}/objects/{objectPath} | 204 always (idempotent delete). |

{objectPath} is URL path after objects/ (percent-encoded segments, no leading slash in the logical key).

HTTP status codes (implemented)

| Code | When | |------|------| | 200 | Successful GET body, PUT, HEAD (no body), list JSON | | 204 | Successful DELETE | | 400 | Bad repoId or object path | | 401 | Auth required or wrong Basic/Bearer | | 403 | Authenticated but repoId not in METAGIT_STORAGE_ALLOWED_REPO_IDS (when that env is set). | | 404 | Missing object (GET / HEAD) | | 206 | GET with satisfiable single bytes= range (partial content). | | 413 | PUT body over METAGIT_STORAGE_MAX_BODY_BYTES, or object over METAGIT_STORAGE_MAX_READ_BYTES on GET/HEAD, or library readBytes maxBytes exceeded. | | 416 | GET with unsatisfiable Range (includes Content-Range: bytes */size). | | 429 | In-memory rate limit exceeded (METAGIT_STORAGE_RATE_LIMIT_RPM); includes Retry-After. | | 500 | Misconfigured auth or unexpected server error |

Observability

• Structured JSON access logs to stdout: ts, method, path (or path_sha256 when METAGIT_STORAGE_LOG_PATH_SHA256 is set), status, latency_ms, repo_id, bytes (response/body size where applicable; list uses 0), optional request_id from X-Request-Id.
• X-Request-Id: If the client sends this header, the same value is set on the response for correlation.
• Auditing: There is no separate audit sink; operators can forward stdout to their log stack. Sensitive path redaction: use METAGIT_STORAGE_LOG_PATH_SHA256.

GCS and IAM

• Grant the service account roles such as Storage Object Admin (or tighter custom role) on the target bucket.
• STORAGE_BUCKET must be the bucket name (e.g. my-project.appspot.com). If the client reports “Not Found” for a valid bucket, confirm the name in Google Cloud Console rather than a CDN or Firebase host string.

Development

git clone [email protected]:x12i/metagit-storage.git
cd metagit-storage
npm install
npm run build
npm test

• Unit tests: In-memory storage + HTTP behavior (src/server.test.ts).
• Live GCS tests: Run when STORAGE_BUCKET and GOOGLE_SERVICE_ACCOUNT_BASE64 are set (e.g. .env loaded via dotenv in src/gcs.integration.test.ts). Never commit .env or keys.

Publish

Scoped public package:

npm publish --access public

Requires npm login with permission to publish @x12i/*.

License

MIT — see LICENSE.