Celilo E2E Test Environment

A Docker-based simulated internet for end-to-end testing of Celilo's full deployment flow: importing modules, configuring firewalls, registering DNS, deploying services via Ansible, and obtaining TLS certificates via ACME.

Why This Exists

Unit and integration tests validate individual components, but can't test the real interactions between SSH, Ansible, iptables, DNS resolution, and ACME certificate issuance. The E2E environment simulates a complete internet with real routing, a real DNS hierarchy, and service simulators that behave like Namecheap, the ISP router, and Let's Encrypt.

Celilo runs inside this simulated network and deploys to Docker containers as if they were real machines. From Celilo's perspective, it's deploying to a real home lab with a real ISP and real internet services.

Network Architecture

The network consists of 7 Docker bridge networks simulating a realistic home lab topology:

| Network | Subnet | Purpose | |---------|--------|---------| | internal | 10.226.1.0/24 | Home LAN (management, firewalls) | | dmz | 10.226.10.0/24 | Public-facing services | | app | 10.226.20.0/24 | Internal applications | | secure | 10.226.30.0/24 | Sensitive services | | isp-external | 100.100.0.0/24 | ISP network (between home and internet) | | internet-external | 100.64.0.0/24 | Simulated internet (DNS, ACME, etc.) | | real-internet | 172.30.0.0/24 | Bridge to actual internet (for apt, pip, etc.) |

All networks except real-internet have Docker IP masquerade disabled — traffic flows through explicit iptables routing, not Docker's NAT. The real-internet network has masquerade enabled to provide actual internet access for package downloads.

Why these addresses (#539)

Every simulated private zone lives inside SIM_PRIVATE_SUPERNET (10.226.0.0/16 — mnemonic 0xE2 = "e2"), declared once in src/types.ts and derived from there by the compose generator, the zone classifier and the harness's system init. Two reasons, both load-bearing:

1. A leaked stack must not be able to collide with a real fleet. The sim used to number its zones exactly like a production celilo deployment; on celilo's own forgejo-builder, which lives in the real dmz, a leaked bridge claimed the builder's own /24 and blackholed every containerized CI job's route to the forge. Teardown cannot be made reliable enough to prevent that (a SIGKILL runs no handler), so the sim moved instead.
2. Nothing may be magic about production's octets. A suite that only passes on one site's address plan is asserting an accident, not celilo's behaviour.

Don't hardcode an octet: use zoneIp(zone, host) / ZONE_SUBNETS / ZONE_GATEWAYS. src/address-plan.test.ts fails if a zone leaves the supernet, or if any retired fleet prefix reappears anywhere in the suite.

Machines

Fixed Infrastructure (always present)

| Machine | Role | Networks | IPs | |---------|------|----------|-----| | management | Celilo CLI, Ansible, SSH | internal | 10.226.1.100 | | fw-main | iptables firewall (managed by Celilo) | internal, dmz, app, secure | 10.226.1.254, 10.226.10.1, 10.226.20.1, 10.226.30.1 | | fw-isp | Greenwave router simulator | internal, isp-external | 10.226.1.1, 100.100.0.100 | | fw-ext | Edge router + transparent HTTPS proxy (Squid) | isp-external, internet-external, real-internet | 100.100.0.101, 100.64.0.1, 172.30.0.5 | | comcast-resolver | Unbound recursive DNS resolver | isp-external, real-internet | 100.100.0.1, 172.30.0.4 | | root-dns | Knot authoritative DNS (root zone) | internet-external | 100.64.0.53 | | tld-dns | Knot authoritative DNS (.com, .org TLDs) | internet-external | 100.64.0.54 | | namecheap-dns | Knot DNS + DDNS API simulator | internet-external | 100.64.0.55 | | letsencrypt | Pebble ACME test server | internet-external | 100.64.0.100 | | apt-cache | apt-cacher-ng package proxy | internet-external, real-internet | 100.64.0.2, 172.30.0.2 |

Dynamic Test Machines

Added per-test via the network() builder API. Each is an Ubuntu 22.04 container running systemd, with SSH server, placed on the appropriate zone network.

| Example | Network | IP | Purpose | |---------|---------|-----|---------| | caddy | dmz | 10.226.10.10 | Caddy reverse proxy deployment | | idp | app | 10.226.20.100 | Identity provider deployment | | db | secure | 10.226.30.50 | Database deployment |

Routing

Traffic flows through explicit routing chains, mimicking a real network:

| From | To | Path | |------|----|------| | management | dmz/app/secure | via fw-main (10.226.1.254) | | management | internet | via fw-isp (10.226.1.1) → fw-ext | | dmz machines | internet | via fw-main → fw-isp → fw-ext | | Pebble | caddy (HTTP-01 challenge) | via fw-ext → fw-isp DNAT → fw-main DNAT → caddy |

All firewall/router containers run ip_forward=1 and MASQUERADE on their outbound interfaces.

DNS

The DNS system is a hybrid of simulated and real resolution:

Simulated domains (handled by the E2E DNS hierarchy):

• iamtheinternet.org — stub zone to namecheap-dns (100.64.0.55)
• park-your-domain.com — stub zone to namecheap-dns (Namecheap DDNS API)
• acme-v02.api.letsencrypt.org — local-data in Unbound pointing to Pebble (100.64.0.100)

Real domains (forwarded to actual DNS):

• Everything else (e.g., dl.cloudsmith.io, deb.debian.org) is forwarded by Unbound to 8.8.8.8 and 1.1.1.1 via the real-internet bridge.

This means Celilo's modules install real packages from the internet while all domain-specific operations (DDNS, ACME) go through the simulation.

Transparent HTTPS Proxy

A Squid proxy with SSL bumping runs on fw-ext, transparently intercepting outbound HTTPS traffic from the simulated network. This lets target machines curl https://dl.cloudsmith.io (to install Caddy from its apt repo) without explicit proxy configuration.

How it works:

1. iptables REDIRECT rules on fw-ext's ISP interface capture port 80/443 traffic
2. Exception: traffic to 100.64.0.0/24 (simulated services) passes through directly
3. Squid does SSL bump (MITM) with a pre-shared CA trusted by all machines
4. Squid resolves DNS via real public DNS and fetches from the real internet

Simulators

Namecheap Dynamic DNS

Runs on namecheap-dns (100.64.0.55). A Bun HTTP server on port 8080 that implements the Namecheap DDNS API:

GET /update?host=<host>&domain=<domain>&password=<password>&ip=<ip>

When called, it updates the Knot DNS zone file and signals a zone reload. DNS propagates through the full chain (namecheap-dns → tld-dns → root-dns → comcast-resolver).

Greenwave Router

Runs on fw-isp (10.226.1.1). A Bun HTTPS server implementing the C4000XG REST API subset:

• POST /cgi/cgi_action — Login/logout
• GET /cgi/cgi_get — Read config (public IP, port mappings)
• POST /cgi/cgi_set — Add/remove port forwarding rules

Port forwarding rules are applied via actual iptables commands, making NAT functional in the simulated network. The management HTTPS interface binds to the internal interface only (10.226.1.1), not the external interface.

Pebble (Let's Encrypt)

Uses the official Pebble ACME test server, wrapped in a custom Dockerfile that adds routing. Pebble uses the comcast-resolver for DNS and validates HTTP-01 challenges through the full routing chain.

The Pebble TLS certificate includes acme-v02.api.letsencrypt.org as a SAN, so Caddy's default ACME configuration works with zero changes (DNS resolves the real Let's Encrypt hostname to Pebble).

fw-ext automatically fetches Pebble's runtime ACME root CA from its management API (https://pebble:15000/roots/0) at startup, so curl from fw-ext can verify Caddy's ACME-issued certificates.

Manual Usage

Starting the Environment

cd e2e

# Infrastructure only (no target machines)
./bin/e2e-up

# With a caddy machine in the DMZ
./bin/e2e-up --caddy

# Full stack (caddy + idp + db)
./bin/e2e-up --full-stack

# Custom machine spec
./bin/e2e-up --custom '{"dmz":{"caddy":"10.226.10.10","web":"10.226.10.20"}}'

After startup, you're dropped into the management machine shell with tab completion for celilo commands (aliased as c).

Accessing Machines

# Reconnect to management (default)
./bin/e2e-shell

# Shell into any container
./bin/e2e-shell caddy
./bin/e2e-shell fw-main
./bin/e2e-shell fw-ext
./bin/e2e-shell namecheap-dns
./bin/e2e-shell letsencrypt

Management uses zsh (with celilo aliases). All other containers use bash.

Running the Manual Test Script

From the management shell (./bin/e2e-shell or after ./bin/e2e-up):

cd /celilo/modules

# System init
c system init --accept-defaults \
    network.dmz.subnet=10.226.10.0/24 \
    network.app.subnet=10.226.20.0/24 \
    network.secure.subnet=10.226.30.0/24 \
    network.internal.subnet=10.226.1.0/24 \
    primary_domain=iamtheinternet.org \
    [email protected] \
    dns.primary=100.100.0.1 \
    dns.fallback=1.0.0.1,8.8.8.8

# Import all modules
c module import namecheap
c module import greenwave
c module import iptables
c module import caddy

# Add machines (--ssh-user root needed for non-interactive)
c machine add 10.226.1.1 --ssh-user root --earmark greenwave
c machine add 10.226.1.254 --ssh-user root --earmark iptables
c machine add 10.226.10.10 --ssh-user root

# Pre-configure secrets and non-derivable config
c module config set namecheap domains '["iamtheinternet.org"]'
c module secret set namecheap ddns_passwords '{"iamtheinternet.org":"test123"}'
c module config set greenwave router_ip 10.226.1.1
c module secret set greenwave router_username admin
c module secret set greenwave router_password admin
c module config set iptables nat_ip 10.226.1.253
c module config set caddy hostname www
c module config set caddy acme_ca https://acme-v02.api.letsencrypt.org/dir

# Deploy (iptables auto-derives config from earmarked machine)
c module deploy namecheap --no-interactive
c module deploy greenwave --no-interactive
c module deploy iptables
c module deploy caddy --no-interactive

Verifying the Deployment

From fw-ext (the "internet" side):

./bin/e2e-shell fw-ext
curl -s https://www.iamtheinternet.org
# Expected: "Caddy reverse proxy is running"

Checking Status

./bin/e2e-status     # Shows containers, DNS, connectivity

Tearing Down

./bin/e2e-down           # Stop and remove everything
./bin/e2e-down --keep    # Stop but keep volumes (faster restart)

Writing Automated Tests

Tests use Vitest and the NetworkBuilder API to start isolated networks, run Celilo commands, and verify results.

Test Structure

import { afterAll, describe, expect, it } from 'vitest';
import { CADDY_DEPLOYMENT } from '../src/fixtures';
import type { NetworkHandle } from '../src/types';

describe('my deployment test', () => {
  let net: NetworkHandle;

  afterAll(async () => {
    await net?.stop(); // Always clean up
  });

  it('deploys and verifies', async () => {
    // 1. Start the network with desired machines
    net = await CADDY_DEPLOYMENT().start();

    // 2. Add machines, import modules, configure
    await net.celilo('machine add 10.226.10.10 --ssh-user root');
    await net.celilo('module import /celilo/modules/caddy');
    await net.celilo('module config set caddy hostname www');

    // 3. Deploy
    const result = await net.celilo('module deploy caddy --no-interactive');
    expect(result.exitCode).toBe(0);

    // 4. Verify from any container
    const curl = await net.exec('fw-ext', 'curl -s https://www.iamtheinternet.org');
    expect(curl.stdout).toContain('Caddy');
  }, 300_000); // Per-test timeout
});

NetworkHandle API

| Method | Description | |--------|-------------| | celilo(cmd, timeout?) | Run a celilo CLI command on the management machine | | exec(container, cmd, timeout?) | Execute a command in any container | | dig(name) | Resolve a DNS name from management | | waitFor(check, timeout, label) | Poll until a condition is true | | socksProxy(opts?) | Spawn a SOCKS5 proxy attached to the test network (details) | | browser(opts) | Launch a host-side browser routed into the network (details) | | stop() | Tear down the entire network |

Browser-driven tests

net.browser() returns a Playwright Browser whose traffic enters the simulated network through a containerized SOCKS5 proxy, so a host-side browser sees the deployed app exactly as a real user would (DNS, TLS, ingress firewall, all in the loop).

Bring your own playwright. @celilo/e2e has zero playwright dependency — the consuming module imports its own chromium (or firefox/webkit) and passes it in. This keeps version alignment between the playwright runtime and the cached browser binaries entirely under the consumer's control.

import { chromium } from 'playwright-core';
// or: import { chromium } from '@playwright/test';

const handle = await net.browser({ vantage: 'isp-external', chromium });
const browser = handle.browser as import('playwright-core').Browser;

const ctx = await browser.newContext({ ignoreHTTPSErrors: true });
const page = await ctx.newPage();
await page.goto('https://www.iamtheinternet.org/');
// ... assertions ...

await handle.close();  // tears down browser + proxy

Vantage controls which simulated zone the proxy attaches to:

• 'isp-external' (default) — residential-user view via comcast-resolver. Exercises the full public ingress path (DNS publication, fw-ext DNAT, ACME, Caddy host routing).
• 'internal' — operator-on-LAN view. Uses the internal split-horizon resolver to reach caddy directly via the internal NAT IP. Useful for debugging.

For manual operator sessions (no browser, just SOCKS), use net.socksProxy(opts) directly. The published port is logged so you can configure Firefox/Chrome to use socks5://127.0.0.1:<port> with remote DNS.

Consumer-side setup. Each consuming module is responsible for:

1. Picking a playwright version. If your module already uses @playwright/test, add playwright-core as a devDependency at the same major.minor — both should track the same chromium build, otherwise the playwright runtime asks for a chromium build the cache doesn't have.
2. Installing the chromium binary. Once per host (or after a playwright bump):

   bunx playwright install chromium

The proxy container itself is built from docker/Dockerfile.socks-proxy in this package — eagerly by e2e-build, or lazily on first net.browser() call. No setup needed for that piece.

Fixtures

Pre-built network configurations in src/fixtures.ts:

CADDY_DEPLOYMENT()    // caddy machine in DMZ
FULL_STACK()          // caddy (dmz) + idp (app) + db (secure)
INFRASTRUCTURE_ONLY() // no dynamic machines

Custom Network Configurations

import { network } from '../src/network-builder';

const net = await network()
  .dmz({ caddy: '10.226.10.10', web: '10.226.10.20' })
  .app({ api: '10.226.20.100' })
  .secure({ db: '10.226.30.50' })
  .start();

Running Tests

cd e2e

# Run all E2E tests
bun run vitest run

# Run a specific test
bun run vitest run tests/caddy-deploy.test.ts

# Run with verbose output
bun run vitest run --reporter=verbose

Tests run sequentially (fixed CIDRs prevent parallelism). The container manager automatically cleans up leftover networks from previous failed runs.

Key Design Decisions

Systemd on target machines: Target machines run systemd as PID 1 (privileged mode) so Ansible's systemd module works naturally. Network setup runs as a oneshot systemd service at boot.

No test parallelism: Fixed network CIDRs mean only one test network can run at a time. Tests run sequentially via Vitest's singleFork config.

Bind-mounted source: The celilo source tree is bind-mounted at /celilo on the management container. Code changes take effect immediately without rebuilding images.

Docker image caching: Images are built once and cached. Only config/simulator changes require rebuilds. The management container uses the bind mount, so celilo code changes are free.

ACME via DNS interception: Instead of overriding Caddy's ACME URL, acme-v02.api.letsencrypt.org resolves to Pebble (100.64.0.100) in the simulated DNS. The caddy module's acme_ca variable allows pointing to Pebble's /dir endpoint (vs Let's Encrypt's /directory).

Directory Structure

e2e/
  bin/
    e2e-up              # Start interactive network
    e2e-down            # Tear down network
    e2e-shell           # Shell into containers
    e2e-status          # Show network status
  config/
    dns/                # Knot zone files and configs
    pebble/             # Pebble ACME config and TLS certs
    proxy/              # Squid transparent proxy config
    resolver/           # Unbound recursive resolver config
    routing/            # Per-container routing scripts
    ssh/                # SSH key generation
  docker/
    Dockerfile.*        # Container images
  docs/
    network-diagram.svg # Network topology diagram
  simulators/
    greenwave/          # C4000XG router REST API simulator
    namecheap-ddns/     # Namecheap DDNS API simulator
  src/
    container-manager.ts    # Docker compose orchestration
    docker-compose-generator.ts  # Generates compose YAML
    fixtures.ts             # Pre-built network configs
    network-builder.ts      # Fluent API for network setup
    types.ts                # Shared type definitions
  tests/
    caddy-deploy.test.ts    # Full caddy deployment with HTTPS
    smoke.test.ts           # Basic connectivity verification
    ...

Suite health policy (e2e-confidence)

The suite must never be tolerated as ambient-red — a regression hiding in chronic red is how the install-sh DNS-pollution bug survived for weeks. Two rules:

• No ambient red. A flaky or failing test is either fixed or quarantined with a tracking issue, never left standing-red. Quarantine by adding a cele2e-ci-unsafe marker comment to the test file (with the issue id); the nightly runs --ci-safe, which skips quarantined tests, so it stays green on the supported set. --complete / --all run everything including quarantined.

• Randomized-order lane. Order-dependent bugs (a test polluting shared infra for a later test) pass in fixed order and solo. The weekly cele2e-shuffle workflow runs --ci-safe --shuffle to flush them out. The shuffle is seeded and the seed is logged; replay a failing order exactly with:

  cele2e run --ci-safe --shuffle --seed=<n>

Supporting guards (so a future bug is caught at the cause, not 15 tests later): the shared-DNS reset verifies its own postcondition, the namecheap-ddns sim refuses overwrites of protected seeded records, the runner names any test that leaves shared DNS polluted, and readiness waits attach live-state diagnostics on timeout instead of speculating.