@assemblyline-agents/vps

Official supported Assembly Line deployment plugin for Hetzner VPS hosts.

The public gateway kind is hetzner; @assemblyline-agents/vps is the reusable host publisher implementation behind it. Existing Hetzner hosts can be registered directly or securely created or adopted with the package's exported bootstrapHetznerHost() operator API, which configures the provider firewall, UFW, fail2ban, unattended upgrades, Docker, host protection, and pinned SSH trust before writing inventory. The plugin supports AMD64 Ubuntu 24.04/26.04 and Debian 12 hosts with Docker Engine, Docker Compose, flock, curl, ss, AppArmor, seccomp, and either root SSH or passwordless sudo. Host Postgres mode also requires OpenSSL and systemd.

Configure

deploy:
  hetzner:
    host: production-eu
    resources:
      cpus: 1
      memory: 1g
      pids: 256
    database:
      mode: external
state: postgres
blob: r2
sandbox: e2b

Use deploy: hetzner when ASSEMBLY_LINE_HETZNER_HOST supplies the host, or set deploy.hetzner.host declaratively. The inventory owns the address, SSH settings, pinned host key, provider resource ID, region, and ingress namespace.

Register the existing server in assembly-line.hosts.json:

{
  "version": 2,
  "hosts": {
    "production-eu": {
      "address": "203.0.113.10",
      "ingress": {
        "baseDomain": "agents.example.com",
        "defaultVisibility": "public"
      },
      "ssh": {
        "user": "deploy",
        "port": 22,
        "identityFileEnv": "ASSEMBLY_LINE_PRODUCTION_EU_SSH_KEY",
        "hostKeySha256": "SHA256:replace-with-ssh-keygen-fingerprint"
      },
      "provider": {
        "kind": "hetzner",
        "resourceId": "optional-provider-server-id"
      }
    }
  }
}

identityFileEnv names a local environment variable whose value is the SSH private-key path. The private key is never copied into the runtime or receipt. The host-key fingerprint is mandatory and is verified before the first SSH session. On Hetzner hosts created by Assembly Line, the provider firewall is the authoritative source-IP gate for port 22 while UFW admits SSH only after traffic has passed that edge. If a native deploy cannot reach SSH, Assembly Line resolves the native process's current public IPv4 address, replaces stale SSH sources on its host firewall with that single /32, and retries the pinned host key once. A host-key mismatch never changes firewall access. This supports normal network and full-tunnel VPN changes without opening SSH globally; browser relay state is not used for the lookup.

Deploy

Point wildcard DNS for *.agents.example.com to the VPS once, put runtime credentials in an owner-only private file such as .env.deploy, then run:

assembly-line deploy agent --target hetzner --sync-secrets --secrets-from .env.deploy --env production

The first deploy requires ASSEMBLY_LINE_ADMIN_TOKEN, all model/channel/blob/ sandbox credentials, and DATABASE_URL when database.mode is external. Use --sync-secrets for that first deploy. Later deploys may reuse the complete remote 0600 secret set without keeping a second copy of runtime credentials on the operator machine; required keys are validated remotely before database setup, migrations, or activation. The host owns the public namespace. For the default deployment environment, the publisher derives <agent-id>.agents.example.com; non-default environments derive <agent-id>-<environment>.agents.example.com. Each agent still receives dedicated runtime containers, ingress/data networks, storage, secrets, and a hostname route. Health-checked blue/green activation uses a shared managed Caddy edge container without placing agents on one shared application network. Activation records the outgoing release as the rollback target only after the new release passes its health and readiness gates, so a failed deploy leaves both the live runtime and the recorded previous release untouched. deploy --rollback refuses with a clear error when no distinct previous release exists instead of stopping the live container.

Every post-preflight deploy attempt ends with managed VPS cleanup, including failed builds, successful activation, --prepare-only, explicit activation, rollback, and ingress reconciliation. The cleanup is serialized with image builds and removes only Assembly Line-labelled stopped containers, dangling deployment-labelled images across the host, obsolete tagged images for the current deployment, and artifact-upload directories abandoned for more than 24 hours. Active, rollback, prepared, and container-referenced images are protected. Container cleanup is limited to exited or dead runtime containers for the current deployment, excluding the active-slot container when an operator has stopped it intentionally; it ignores newly created containers and shared edge/database containers. It never prunes volumes or databases, stops active containers, or removes tagged active and rollback images. A cleanup problem is reported as a degraded cleanup receipt without replacing the deployment's real outcome.

Set deploy.hetzner.ingress.visibility: private in agent.md to create no public hostname, Caddy route, certificate, or ASSEMBLY_LINE_PUBLIC_URL. Switching an existing deployment between public and private ingress is transactional and can be applied without rebuilding with deploy --ingress-only.

database.mode: "host" creates a private shared Postgres container but gives each agent its own database and login role. Migrations reach that otherwise private cluster through a temporary fingerprint-pinned SSH tunnel and close the tunnel when the migration command exits. Host mode also requires these S3-compatible backup settings. Put bucket, region, and optional endpoint in config.production.ts; keep the access key and secret access key in the configured secret store or host deployment secrets:

ASSEMBLY_LINE_VPS_BACKUP_BUCKET=
ASSEMBLY_LINE_VPS_BACKUP_REGION=
ASSEMBLY_LINE_VPS_BACKUP_ACCESS_KEY_ID=
ASSEMBLY_LINE_VPS_BACKUP_SECRET_ACCESS_KEY=
ASSEMBLY_LINE_VPS_BACKUP_ENDPOINT=

The endpoint is optional. Deployment verifies backup access with a pre-migration dump, then a daily systemd timer writes and verifies compressed database dumps below <bucket>/<deployment-id>/ and enforces retention. A weekly timer restores the latest backup into a scratch database. The deployment-scoped restore.sh requires explicit replacement confirmation, takes a fresh backup, and restores the pre-operation database automatically if the requested restore fails.

This is trusted-owner isolation, not hostile multi-tenant isolation. Put mutually untrusted tenants on separate VMs or microVMs.

The supported target schedules one AMD64 runtime replica per agent. ARM64, high availability, multi-replica scheduling, and automatic workload deletion are not currently supported.

Smoke tests

The default test suite uses a fake SSH transport and does not touch infrastructure. Opt-in live smoke coverage supports a local Linux-over-SSH host and Hetzner:

ASSEMBLY_LINE_VPS_LIVE_SMOKE=1 \
ASSEMBLY_LINE_VPS_LIVE_HOSTS_FILE=/absolute/path/assembly-line.hosts.json \
ASSEMBLY_LINE_VPS_LIVE_SECRETS_FILE=/absolute/path/smoke.env \
ASSEMBLY_LINE_VPS_LIVE_HETZNER_HOST=hetzner-smoke \
node --test tests/vps-live-smoke.test.mjs

Use the corresponding ASSEMBLY_LINE_VPS_LIVE_LOCAL_* or ASSEMBLY_LINE_VPS_LIVE_HOSTINGER_* variables for the other profiles. These tests perform a real deployment and therefore require the explicit enable flag.

License

MIT