cv-orchestrate

Fleet orchestration daemon for CV-Agent. Deploy, supervise, and monitor AI agents across machines.

The Problem

You have agents. You don't have a fleet. Every cva agent session is a manual SSH + terminal process that dies when the shell closes. No auto-restart, no health monitoring, no central visibility.

cv-orchestrate solves this with two components:

1. cvod — A daemon that runs on each machine as a systemd service, spawning and supervising agent processes per a YAML manifest, and heartbeating health to CV-Hub.
2. Fleet API — CV-Hub endpoints for machine enrollment, heartbeat, and fleet visibility. (Remote command delivery, agent deploy, and task routing are planned — see Fleet API.)

Architecture

┌─────────────────────────────────────────────────────────┐
│                   CV-Hub (Control Plane)                  │
│                                                          │
│  ┌────────────────┐  ┌────────────┐  ┌────────────────┐ │
│  │ Enroll tokens  │  │  Machine   │  │ Fleet view     │ │
│  │ + machine      │  │  registry  │  │ (status, last  │ │
│  │ identity       │  │            │  │  heartbeat,    │ │
│  │ (fleet:*)      │  │            │  │  agents)       │ │
│  └───────┬────────┘  └─────┬──────┘  └────────────────┘ │
└──────────┼─────────────────┼─────────────────────────────┘
     enroll │        heartbeat│  (per-machine machine_token)
     ┌──────┴───┐     ┌───────┴──┐     ┌──────────┐
     │ primary  │     │  aux-1   │     │   gpu-1  │
     │  cvod    │     │  cvod    │     │  cvod    │
     │ agent-a  │     │ agent-c  │     │ trainer  │
     │ agent-b  │     │ mcp-svc  │     │          │
     └──────────┘     └──────────┘     └──────────┘

Auth model — enrollment (read this first)

Machines authenticate with a per-machine identity token, not a shared org PAT.

1. An org owner/admin mints a single-use fleet:enroll bootstrap token: POST /api/v1/fleet/enroll-tokens (PAT auth).
2. Put it in the environment as CV_HUB_ENROLLMENT_TOKEN (see enrollment_token_env).
3. On first boot cvod exchanges it once at POST /api/v1/fleet/enroll for a per-machine machine_token (scope fleet:heartbeat), and persists it 0600 at credential_path. The enroll token is single-use and revoked on success.
4. Every subsequent boot loads the persisted credential and heartbeats with it — no network exchange, no bootstrap token needed.

Deprecated: hub_token_env (a bare org PAT, default CV_HUB_TOKEN) is a transition fallback only. The deployed hub now rejects a bare org PAT at /heartbeat with 403 — it requires a fleet:heartbeat machine identity. Enroll instead.

Quick Start

# Install (npm global / npm link — the CLI is exposed as `cvod`)
npm install -g @controlvector/cv-orchestrate

# Initialize on a machine (writes ~/.config/cvod/agents.yaml)
# (role is a manifest field — default `auxiliary`; edit machine.role in agents.yaml to change it)
cvod init --name aux-1

# Mint a single-use enroll token in CV-Hub (org owner/admin), then export it:
export CV_HUB_ENROLLMENT_TOKEN=<fleet:enroll token>

# Edit the manifest to add your agents (safe at any point in this order)
vim ~/.config/cvod/agents.yaml

# Enroll this machine (one-shot). This UPDATES the manifest's identity keys
# (machine.name / hub_url / enrollment_token_env) and PRESERVES everything else you
# edited — your agents, machine.role, tags, paths, capabilities and any custom keys are
# kept (merge, not rewrite). `cvod start` never enrolls implicitly.
cvod enroll --hub https://api.hub.controlvector.io

# Run foreground for testing (heartbeats; spawns only agents with `enabled: true`)
cvod start --foreground

# Or install + run as a systemd service (renders the unit for THIS host). The unit +
# installer ship in the npm package, so this works straight after `npm i -g`:
INSTALL_SERVICE=true bash "$(npm root -g)/@controlvector/cv-orchestrate/scripts/install.sh"
sudo systemctl start cvod

The manifest is searched (first match wins): /etc/cvod/agents.yaml, ~/.config/cvod/agents.yaml, ./agents.yaml. cvod init writes the second.

Agent Manifest

# ~/.config/cvod/agents.yaml
machine:
  name: aux-1
  role: auxiliary
  # NOTE: machine.tags are NOT sent to CV-Hub (not in GET /fleet/machines) — they are not
  # fleet-targeting metadata yet. Any tags you add are preserved in the file but unused.

  # --- Fleet identity (see "Auth model" above) ---
  hub_url: https://api.hub.controlvector.io      # default
  enrollment_token_env: CV_HUB_ENROLLMENT_TOKEN  # default; env var holding the enroll token
  credential_path: /var/lib/cvod/credential.json # default; persisted 0600 machine_token
  hub_token_env: CV_HUB_TOKEN                     # default; DEPRECATED org-PAT fallback

  # --- Local paths (must be writable by the service user) ---
  log_dir: /var/log/cvod                         # default; per-agent log files
  pid_file: /run/cvod/cvod.pid                   # default (systemd unit provisions /run/cvod)

agents:
  # Agents FAIL CLOSED: an agent runs only with `enabled: true`, and auto-approves tasks
  # only with `auto_approve: true`. Omitting either keeps it OFF (safe default). Unknown
  # keys you add are preserved (round-tripped), not stripped.
  - name: example-mcp
    repo: myorg/example-mcp
    workspace: /home/user/projects/example-mcp
    auto_approve: true
    restart: always        # always | on-failure | never
    max_restarts: 10       # per hour
    backoff_base_ms: 1000
    env_file: ~/.env.example-mcp
    enabled: true

  # NOTE: `schedule:` is NOT yet implemented. The supervisor only logs the schedule and
  # tells you to register it with system cron — a scheduled agent will NOT run as a
  # managed process. Omit it (run continuously) until cron support lands.
  - name: scheduled-job
    repo: myorg/scheduled-job
    workspace: /home/user/projects/scheduled-job
    schedule: "0 */6 * * *"  # cron-style — see NOTE above (not implemented)
    enabled: true

Defaults note: the shipped credential_path (/var/lib/cvod), log_dir (/var/log/cvod), and pid_file (/run/cvod) require writable directories. The bundled systemd unit provisions all three for the service user via StateDirectory=/LogsDirectory=/RuntimeDirectory= (see Systemd). If you run cvod outside systemd as a non-root user, point these at a home path (e.g. ~/.local/share/cvod/…) — the supervisor mkdirs log_dir with no fallback and will exit if it isn't writable.

CLI Commands

cvod init           # Initialize machine, write manifest template
cvod enroll --hub … # Enroll with a hub (one-shot); merges identity into the manifest
cvod start          # Start daemon (spawns agents, heartbeats) — does NOT enroll
cvod status         # Show agents, PIDs, and daemon state
cvod logs <agent>   # Tail an agent's logs
cvod deploy <repo>  # Clone repo + add agent to the manifest

cvod start --foreground runs in the foreground (used by the systemd unit). Graceful shutdown is handled via SIGTERM/SIGINT; SIGHUP reloads the manifest.

Commands such as stop, restart <agent>, remove, register, and unregister are not implemented — the daemon is stopped via systemd/signals, and machine identity is established by enrollment (above), not a register command.

Systemd

deploy/cvod.service is a template; scripts/install.sh renders it for the current host (INSTALL_SERVICE=true bash "$(npm root -g)/@controlvector/cv-orchestrate/scripts/install.sh", or bash scripts/install.sh from a source checkout). It:

• resolves the real cvod path with command -v cvod (npm-global/npm link installs are not in /usr/local/bin) for ExecStart;
• puts the install bin dir on the unit's PATH so the daemon can spawn cva;
• keeps NoNewPrivileges=true + ProtectSystem=strict, and uses StateDirectory=cvod (/var/lib/cvod), LogsDirectory=cvod (/var/log/cvod), and RuntimeDirectory=cvod (/run/cvod) so the default credential_path/log_dir/pid_file are writable without pre-creating anything as root;
• grants ReadWritePaths=<home> /tmp for agent workspaces and temp files;
• loads EnvironmentFile=-<home>/.env.cvod (for CV_HUB_ENROLLMENT_TOKEN on first boot).

Fleet API (CV-Hub)

Implemented:

POST   /api/v1/fleet/enroll-tokens     Mint a single-use fleet:enroll token (PAT, org owner/admin)
POST   /api/v1/fleet/enroll            Exchange an enroll token for a per-machine machine_token
POST   /api/v1/fleet/heartbeat         Report health/agents (machine_token, fleet:heartbeat)
GET    /api/v1/fleet/machines          List machines: status, last heartbeat, agents (PAT, org-scoped)

Planned — not yet implemented:

GET    /api/v1/fleet/machines/:id      Machine detail
POST   /api/v1/fleet/machines/:id/cmd  Command delivery (needs a cvod inbound channel)
POST   /api/v1/fleet/deploy            Remote agent deploy
GET    /api/v1/fleet/agents            Fleet-wide agent list
DELETE /api/v1/fleet/agents/:id        Stop + deregister an agent
GET    /api/v1/fleet/health            Fleet-wide health summary

Task/affinity routing (capability-based dispatch across machines) is also planned and not part of the current daemon or hub.

License

Apache-2.0. See LICENSE. Copyright (c) Control Vector LLC.