@volter/twin-gcs

Local Google Cloud Storage twin, built on the shared @volter/world-core kernel. One handler serves GCS's real wire surfaces, so real clients work unmodified:

• @google-cloud/storage (Node, v7) under STORAGE_EMULATOR_HOST — JSON API under /storage/v1/..., uploads under /upload/storage/v1/b/<bucket>/o (uploadType=media|multipart|resumable, including the SDK's real single-shot streaming Content-Range: bytes 0-*/* finalize semantics), and the SDK's root-path download quirk: file.download() requests /b/<bucket>/o/<object>?alt=media with no /storage/v1 or /download prefix — this twin serves that path too, deliberately more compatible than fake-gcs-server.
• google-cloud-storage (Python) under STORAGE_EMULATOR_HOST with anonymous credentials — same JSON surface plus /download/storage/v1/b/<bucket>/o/<object>?alt=media for media. (Documented parity: this repo's CI has no Python, so the claim is exercised via raw-path tests over the identical wire surface, not a Python process.)
• Plain HTTP fetchers (curl, headless Chrome, <video> elements) hitting V4 signed URLs — http://<host>/<bucket>/<object>?X-Goog-Algorithm=...&X-Goog-Signature=.... Signatures are accepted without cryptographic verification (the repo's fake-auth-locally idiom — the real check needs the service account's private key, which never exists here), but expiry is enforced vendor-faithfully: past X-Goog-Date + X-Goog-Expires → 400 with Google's ExpiredToken XML error. Range requests are honored on every media read (206 + Content-Range + Accept-Ranges) — video players seek.

Uploads return real md5Hash (base64) and crc32c (base64, big-endian Castagnoli) — the Node SDK's default integrity validation checks these and rejects an emulator that gets them wrong. Composite objects (compose) omit md5Hash, like real GCS.

State model: kernel control plane + a content-addressed byte plane (PROPOSED CONVENTION)

Control-plane state — buckets, object generations (with metagenerations, preconditions, versioning), resumable upload sessions — lives in the kernel's append-only action log and is folded into the projection, as in every pack.

Object BYTES do not. The repo's only prior object-storage twin, aws's S3 area, stores object bodies inline in the kernel action fields as base64 (data_b64, see aws-s3.ts's header) — workable for test fixtures, pathological for the video-sized media a GCS twin must hold (a 500 MB clip would become a ~667 MB base64 JSON line re-parsed on every projection fold). This pack instead stores bytes content-addressed through the kernel's BLOB SEAM (runtime contract R11, packages/world-core/src/blob-store.ts) under the key <root>/.volter/world/gcs/blobs/sha256/<aa>/<hash> — immutable, deduplicated, write-once, referenced from the projection by sha256 + size (gcs-blobs.ts). Resumable uploads stage chunks under blobs/staging/ until finalize, then promote into the store.

The seam is the filesystem by default (FsBlobStore uses a key verbatim as a path, so an existing world dir's tree is unchanged) and an object-storage adapter inside a hosted namespace, so the same handler code serves and accepts bytes at the edge whether the world is a local directory or a Durable Object with R2 behind it — gcs-serverless.test.ts drives the whole byte-carrying walk against MemoryWorldStore + MemoryBlobStore with a world root that never comes into existence on disk.

⚠ Splitting bulk bytes out of the log is still a per-pack choice, not a repo-wide one. S3 keeps them inline; this pack (with slack's file uploads) is the precedent for the seam.

Usage

world-gcs serve --port 4443            # or: --data-dir <dir> (alias of --root, fake-gcs-server parity)
# then point clients at it:
#   STORAGE_EMULATOR_HOST=http://127.0.0.1:4443           (Node + Python SDKs)
#   curl 'http://127.0.0.1:4443/<bucket>/<object>?X-Goog-Algorithm=...&X-Goog-Signature=...'
world-gcs conformance                  # self-referential endpoint/resource inventory check

--read-only serves a pure mirror: reads work, every write is rejected with 405. --cors <origin>[,<origin>...] is the operator-level CORS fallback for signed-URL consumers — see below.

CORS (browser consumers: fetch() and crossOrigin media loads)

Real GCS has two CORS regimes (cross-origin docs), and this twin models both:

• JSON API endpoints allow any origin, always — a bucket's CORS configuration does not apply to them. The twin answers JSON-API preflights and adds Access-Control-Allow-Origin (+ an expose list covering the media headers: content-range, x-goog-hash, …) to any JSON-API response whose request carries an Origin header. No configuration needed — a browser can fetch() a /storage/v1 or /download/storage/v1 media URL out of the box.
• The XML/signed-URL surface obeys the bucket's cors configuration — the real bucket resource field: a list of { origin, method, responseHeader, maxAgeSeconds } rules, set via bucket insert or patch, echoed on get (omitted when unset, like the vendor). Preflight OPTIONS is answered per the matching rule (Allow-Origin/Methods/Headers/Max-Age); matching simple requests get Allow-Origin + Expose-Headers. Origins and methods match exactly or via '*' (a '*' entry answers the literal *); an exact match echoes the origin and adds Vary: Origin. A non-matching origin gets the response without CORS headers — the vendor serves the bytes and the browser blocks, never a fake allow.

Nothing engages without an Origin request header, so every SDK/curl flow is byte-identical to the pre-CORS twin. An OPTIONS that is not a genuine preflight (no Origin + Access-Control-Request-Method) keeps the historical 405.

Setting a bucket's CORS config (what terraform does to the real bucket in prod):

curl -X PATCH "http://127.0.0.1:4443/storage/v1/b/<bucket>" \
  -H 'content-type: application/json' \
  -d '{"cors":[{"origin":["http://localhost:3000"],"method":["GET","HEAD"],"responseHeader":["Content-Type","Range"],"maxAgeSeconds":3600}]}'

Many worlds seed buckets out-of-band (in prod, terraform sets the CORS config and the app under test never does) — for those, world-gcs serve --cors http://localhost:3000 (or --cors '*') applies an operator fallback rule ({ origin: [<flags>], method: ['*'], responseHeader: ['*'], maxAgeSeconds: 3600 }) to every bucket that carries no cors config of its own; bucket-level config always wins. This is an operator lever in the spirit of --read-only and the --data-dir alias (fake-gcs-server, which this CLI claims drop-in parity with, answers CORS permissively) — the vendor-faithful path is the bucket config above.

Upstream mirror mode (hybrid worlds — OFF by default)

Hybrid eval worlds unseal only the AI vendors while storage stays on this local twin. Real media-understanding vendors (e.g. Vertex Gemini) fetch gs:// objects themselves, server-side — a vendor-to-vendor edge the app under test never sees and a local emulator can never serve. Upstream mirror mode makes the twin able to satisfy that edge: objects written to ONE configured local bucket are synchronously written through to the same-named bucket in real GCS at the same paths, so the exact gs://bucket/name URIs the app hands a real vendor resolve upstream — and the twin cleans up after itself. This is a general twin capability (any app, any world); there is no consumer-specific logic.

world-gcs serve --root <root> \
  --mirror-upstream <bucket>            # ONLY this local bucket write-throughs (same-named upstream bucket)
  --mirror-credentials <sa.json>        # service-account key (fallback: $GOOGLE_APPLICATION_CREDENTIALS)
  --mirror-strict | --no-mirror-strict  # strict is the DEFAULT: an upstream write failure fails the
                                        # originating twin request loudly (502 twinMirrorUpstreamFailed /
                                        # XML InternalError — the hybrid world is broken, the twin says so);
                                        # --no-mirror-strict downgrades to a logged warning
world-gcs mirror-cleanup --root <root> [--mirror-credentials <sa.json>]   # world teardown

Semantics (all enforced by failable verifies in the manifest's mirror area + gcs-mirror.test.ts):

• Write-through — on every successful local object.insert in the configured bucket (media/multipart upload, resumable finalize, XML/signed-URL PUT, copy, compose — they all funnel through one insert path) the same bytes land upstream at the same path before the twin responds. The local API surface is unchanged — clients still talk to the twin exactly as without mirroring.
• Mirror manifest — every upstream write is recorded in an append-only JSONL ledger (<root>/.volter/world/gcs/mirror-manifest.jsonl, same durability approach as the kernel action log: one append syscall per record, VOLTER_DURABLE=1 adds fsync): object path + upstream generation + timestamp. A write-ahead upstream-intent record brackets each upstream call so a crash between the upstream write landing and its success record cannot silently leak an object — cleanup reports such unresolved intents instead of guessing.
• Delete-through — a local delete that removes the live generation of a mirrored object deletes upstream too, guarded by ifGenerationMatch on the recorded upstream generation.
• mirror-cleanup — deletes every upstream object the manifest says this twin created and still owns. Generation-guarded: an object someone else overwrote is reported + disowned, never deleted. Idempotent (a second run makes zero upstream calls). The twin never touches upstream objects it did not create and never lists/reads/scans the upstream bucket beyond its own manifest entries.
• Real client — SDK-free (fetch + node:crypto RS256 service-account JWT; the vendor SDK stays a test-only devDependency), and fail-closed rate-guarded through the kernel's shared RateBudget under this pack's own declaration (gcs-budget.ts: 300 calls/min, 0.5% of Google's documented ~1,000 object writes/second per bucket — see the module header and the roster in scripts/rate-budget-isolation.test.ts). Credentials and minted tokens are never logged and never put in error messages or the manifest — mirror failure messages flow into both the manifest and the 502 body, so they carry only HTTP status + trimmed upstream response body.

Honest edges (say them rather than imply them away):

• Strict is loud, not atomic. The local write commits before the mirror runs (append-only log — no rollback), so a strict 502 means "local has the new bytes; the upstream copy is missing or stale". @google-cloud/storage retries 5xx by default, and each retry re-inserts locally and re-attempts the mirror — a transient upstream failure self-heals; a persistent one surfaces once the SDK's retries are exhausted. In --no-mirror-strict that divergence is only a logged warning — a real vendor reading the gs:// URI meanwhile gets stale bytes.
• A corrupt interior manifest line stops the mirror and cleanup cold (delete-through fails, mirror-cleanup refuses with the file:line). Deliberate: guessing about upstream ownership is how someone else's objects get deleted. Repair the named line, re-run. There is no --force.
• One serving process per state root. Mirror operations are serialized per serve context so manifest order equals upstream completion order; two processes appending to one root are not coordinated.
• mirror-cleanup is manifest-scoped, not bucket-scoped — it tears down every owned entry the root's manifest holds, whichever bucket recorded it.
• Connector push is a separate egress path (the kernel invokes performGcsAction): objects it uploads are NOT mirror-tracked and mirror-cleanup will not remove them. Don't mix push and --mirror-upstream on the same bucket unless you want that.

Coverage

Partial and honest — the manifest (gcs-capabilities.ts) is the real vendor surface as the denominator (113 capabilities: 68 done / 45 todo), authored top-down from the storage/v1 JSON API reference (the vendor publishes a first-party machine-readable Discovery document — see spec-sources.json), the XML API, and the V4 signed-URL surface. Modeled and proven by failable verifies: bucket CRUD + IAM policy get/set, the three upload flows (media/multipart/resumable incl. chunked 308 + status query + carried preconditions), all four media-read paths with Range, object metadata get/patch/delete, list with prefix/delimiter/pagination, copy/rewrite/compose, generations + versioning + noncurrent fetches, ifGenerationMatch preconditions (incl. the =0 create-guard), signed-URL accept/expiry/malformed/Range, CORS (bucket cors config on the XML/signed-URL surface + the JSON API's any-origin regime — see above), XML PUT/GET/DELETE, the vendor JSON error envelope, read-only mode, connector pull/push, and the upstream mirror (write-through, bucket scoping, strict / non-strict failure propagation, delete-through, cleanup idempotency

• generation guard — all over injected fakes). The larger todo backlog (ACLs, notifications config, HMAC keys, batch, XML multipart, lifecycle/retention config, HNS folders, …) is enumerated honestly in the manifest rather than implied complete. Unmodeled operations fail with an honest vendor-shaped error envelope (501 notImplemented) — never a fake success.

What this twin does not run

One local disk has no geo-redundancy, so location/storageClass/rpo are stored and echoed vendor-faithfully but no replication physics exist here. V4 signed URLs are not cryptographically verified — that needs the service account's real key pair — but parameter presence and expiry ARE enforced (400 ExpiredToken), which is the behaviour integrations depend on. Auth is faked, the repo idiom: IAM policies are stored and served through get/setIamPolicy rather than enforced by a real Google identity system. There is no Cloud KMS offline (CMEK/CSEK metadata acceptance is the separate todo gcs.upload.kms_key_metadata) and no network egress, so Pub/Sub notifications are never published (the config CRUD is gcs.notifications.config_crud).

Applying lifecycle rules, soft-delete retention and retention-policy expiry to stored objects is a filed gap (gcs.buckets.lifecycle_execution), alongside the config-acceptance todos.

No UI mirror

GCS is an API-first vendor: when someone does this vendor's core job they write code that calls Cloud Storage — the Cloud Console's object browser is incidental admin tooling for inspection and billing, not where the work happens. Per the contributor recipe's "Does this vendor get a mirror?" rule ("an integrator calls the API; the vendor's console is incidental → the API is the product → omit the mirror"), this pack ships no mirror and no UI capabilities; coverage is API + connector. Note: aws's S3 bucket-browser mirror is a documented legacy outlier, not precedent (the contributor recipe lists it as retro-fitted debugging affordance) — it does not make object stores a mirror category.

Observation and execution

• pull — syncGcsFromReal(client, { root, occurredAt, withBytes }) over an injected client (GcsLikeClient: the real @google-cloud/storage wrapped in prod, a fake in tests) folds real buckets + objects into the observed log via the kernel observation path (shadow-diff dedup — identical re-pull appends 0 deltas). Pulled bytes are promoted into the byte plane.
• write — the default: every API write is a local kernel action; no real GCS is ever called from the request path. (The one deliberate, opt-in exception is upstream mirror mode above — write-through over the injected GcsUpstreamClient, manifest-tracked and torn down by mirror-cleanup; without --mirror-upstream nothing changes.)
• deploy — the kernel invokes performGcsAction for supported bucket/object entries, with bytes read from BlobStore, and records the receipt. Push between Worlds and real vendor execution are separate operations; see the model.