swarmkit-infra

Shared cloud-execution infrastructure for the swarmkit ecosystem — the layer that answers "may this run use that capacity, and how does capacity get provisioned, shared, and wired together?" Design + decision log (I1–I22): swarmkit-infra-design.md.

Three layers, all shipped and live-validated:

• L0 — resource broker. Cross-process lease admission over shared account caps (Modal GPUs, E2B sandboxes) so a training run and an eval sweep coordinate instead of crashing each other on the provider cap. TTL leases, priority queue, atomic multi-pool co-allocation, live-quota refresh.
• L1 — capacity. A warm SandboxPool over placement providers (E2B, EC2 fleet) behind swarmkit-eval's ExecutionBackend port; a pluggable ArtifactStore (local + S3) for artifacts that outlive a run.
• L2 — topology. Declarative orchestration of an RL run's services (serving + proxy
  • envs + memory + trainer) — atomic co-allocation, dependency ordering, endpoint wiring, teardown, pinned manifest. See topology/README.md.

Packaging & how it's consumed

swarmkit-infra is a standalone package (name: swarmkit-infra, version: 0.0.1), not currently published to npm — and it does not need to be for the current single-machine, monorepo workflow. It's consumed three ways, none requiring a registry:

| Consumer | Mechanism | Needs publish? | |---|---|---| | swarmkit-eval (TS) | file:../infra dev-dep symlink + optional peer dep (swarmkit-infra ^0.0.1), lazy-imported — fail-open when absent | No, in-repo. Yes if swarmkit-eval is published to npm and external consumers want the lease/pool/artifact features (the optional peer must then resolve). | | chorus (Python) | vendored stdlib client (client/python/swarmkit_infra_client.py copied into chorus/infra/) — I19: never pip-published, deliberately | No — vendoring is the design. | | the CLI (broker, topology) | run the built dist/cli.js by path (the launchd plist and chorus both point at it) | No. Publish/npm link only if you want swarmkit-infra as a global command instead of node …/dist/cli.js. |

When to publish: only two triggers — (a) shipping swarmkit-eval to npm for external users who want the coordination features, or (b) wanting the CLI installed globally. npm publish is wired (the package is private: false with bin/exports/files set); until a trigger arrives, the in-repo symlink + vendored Python client + CLI-by-path is the intended posture (design I7: incubate in-repo, extract/publish when a second consumer needs it standalone).

Zero runtime dependencies (cloud SDKs are shelled via injectable exec, never imported).

The interface

CLI (dist/cli.js — the language-neutral surface, I20)

swarmkit-infra broker start [--config <path>] [--port <n>]   # the loopback daemon
swarmkit-infra status [--url <base>]                         # who holds what, queues
swarmkit-infra revoke <leaseId>                              # manually free a stuck lease
swarmkit-infra topology up <spec.json> [--dry-run] [--detach]  # co-allocate → launch → wire → run → teardown
swarmkit-infra topology status [<name>]                      # live/orphaned runs, held pools, urls
swarmkit-infra topology down <name>                          # tear down a run (clean, or force if orphaned)

topology up blocks until the job(s) finish, then tears everything down. --detach backgrounds a supervisor that holds+renews the leases and runs the job(s), returning immediately — the shape for a long training run. Either way a run record (~/.swarmkit/topologies/<name>.json) is written once services are up, so status/down work from any terminal. down signals the supervisor to unwind its own teardown; if the supervisor died (SIGKILL/reboot → DEAD(orphaned)), down force-tears-down from the record (re-runs shutdowns, releases leases by id). Spec needs use fully-qualified pool names (e2b-sandboxes/account-A); an unknown pool is a loud error, never a silent unmetered run.

The broker runs persistently via launchd (~/Library/LaunchAgents/com.swarmkit.infra-broker.plist).

Library (TypeScript, import … from "swarmkit-infra")

// L0 — leases
import { LeaseClient } from "swarmkit-infra";
const leases = new LeaseClient({ holder: "swarmkit-eval:run-42" });
await leases.withLease("e2b-sandboxes/account-A", { priority: "eval" }, async () => { /* create sandbox */ });
await leases.acquireAll([{ pool: "skypilot-gpu/aws", count: 2 }, { pool: "ec2-fleet/default", count: 4 }]); // atomic (I16)

// L1 — capacity
import { SandboxPool, Ec2FleetProvider, createArtifactStore } from "swarmkit-infra";
const pool = new SandboxPool(new Ec2FleetProvider({ nodes: 2 }), { leaser, warmTarget: 12 });
const store = createArtifactStore();                       // ArtifactRouter from ~/.swarmkit/infra.json
await store.put("runs/r1/results.json", data, { kind: "run-evidence" });

// L2 — topology
import { runTopology, realTopoExec } from "swarmkit-infra";
await runTopology(spec, { exec: realTopoExec, leaser, artifacts: store, signal });

// broker (for embedding a daemon in-process, e.g. tests)
import { startBroker, loadConfig } from "swarmkit-infra";

Full export surface: broker (LeaseClient, startBroker, config, refreshCapacities), sandbox (SandboxPool, Ec2FleetProvider), artifacts (ArtifactStore, LocalArtifactStore, S3ArtifactStore, ArtifactRouter, createArtifactStore), topology (runTopology, topoOrder, TopologyError). See index.ts.

Python (vendored, stdlib-only)

from swarmkit_infra_client import LeaseClient          # copied into the consuming repo
client = LeaseClient(holder="chorus:verl_b2")
with client.lease("modal-gpu/sudocode-ai", count=8, priority="training"):
    ...  # modal run

Topologies are driven from Python via the CLI (swarmkit-infra topology up), not a Python engine (I19/I20). Single-pool acquire is the only lease call chorus needs; multi-pool acquireAll lives in the CLI's topology path.

Configuration — ~/.swarmkit/infra.json

Caps are data, never code (I3). Pools declare a capacity (concurrent-hold level), an optional ratePerSec (create-rate pacing), an optional capacityCmd (a shell command whose stdout is the live capacity — AWS quota grants self-apply, I14), and a note. The artifacts section selects ArtifactStore backends + placement (I13). Example:

{
  "pools": {
    "modal-gpu/sudocode-ai":    { "capacity": 10, "note": "Modal workspace cap" },
    "e2b-sandboxes/account-A":  { "capacity": 20, "ratePerSec": 5 },
    "ec2-fleet/default":        { "capacity": 24 },
    "skypilot-gpu/aws":         { "capacity": 2, "capacityCmd": "aws service-quotas …" }
  },
  "priorities": { "training": 100, "eval": 50, "batch": 10 },
  "artifacts": {
    "backends": { "local": { "dir": "~/.swarmkit/artifacts" }, "s3": { "bucket": "…", "region": "us-west-2" } },
    "placement": { "byKind": { "run-evidence": "s3", "trajectory": "s3" }, "default": "local" }
  }
}

Semantics (the load-bearing rules)

• Lease, not lock — every grant has a TTL; clients auto-renew; a crashed holder's capacity returns at expiry. Priority = queue order (training > eval > batch); high priority jumps the queue, never evicts. Atomic multi-pool (acquireAll, I16) — all-or-nothing, so a topology never idle-holds GPUs while blocked on another pool.
• Fail-open covers transport, not config (I4, I23) — broker unreachable ⇒ callers proceed on their own backoff + one warning (SWARMKIT_INFRA_REQUIRE=1 for claim runs that must not race). But a reachable broker answering "unknown pool" throws UnknownPoolError — silently unmetering a typo'd pool would reintroduce the exact oversubscription the broker exists to prevent.
• Held vs cycled (I22) — the topology co-allocates resources held for the run (GPU/fleet nodes); cycled per-episode sandboxes stay lazily leased by their consumer.
• Signals are safe (Ctrl-C / topology down) — topology up traps SIGINT/SIGTERM into a teardown that stops launched services (reverse order) and releases every lease; a running job is interrupted via the forwarded abort signal (no hang). A detached run survives its launching terminal; down (or a second Ctrl-C) unwinds it.

Development

npm run typecheck && npm test && npm run build   # 62 tests, zero runtime deps

Incubated in the swarmkit repo (the swarmkit-eval pattern). Dependency arrow: swarmkit-eval → swarmkit-infra, never the reverse.