Celilo E2E Test Environment

A Docker-based simulated internet for end-to-end testing of Celilo's full deployment flow: importing modules, configuring firewalls, registering DNS, deploying services via Ansible, and obtaining TLS certificates via ACME.

Why This Exists

Unit and integration tests validate individual components, but can't test the real interactions between SSH, Ansible, iptables, DNS resolution, and ACME certificate issuance. The E2E environment simulates a complete internet with real routing, a real DNS hierarchy, and service simulators that behave like Namecheap, the ISP router, and Let's Encrypt.

Celilo runs inside this simulated network and deploys to Docker containers as if they were real machines. From Celilo's perspective, it's deploying to a real home lab with a real ISP and real internet services.

Network Architecture

The network consists of 7 Docker bridge networks simulating a realistic home lab topology:

| Network | Subnet | Purpose | |---------|--------|---------| | internal | 192.168.0.0/24 | Home LAN (management, firewalls) | | dmz | 10.0.10.0/24 | Public-facing services | | app | 10.0.20.0/24 | Internal applications | | secure | 10.0.30.0/24 | Sensitive services | | isp-external | 100.100.0.0/24 | ISP network (between home and internet) | | internet-external | 100.64.0.0/24 | Simulated internet (DNS, ACME, etc.) | | real-internet | 172.30.0.0/24 | Bridge to actual internet (for apt, pip, etc.) |

All networks except real-internet have Docker IP masquerade disabled — traffic flows through explicit iptables routing, not Docker's NAT. The real-internet network has masquerade enabled to provide actual internet access for package downloads.

Machines

Fixed Infrastructure (always present)

| Machine | Role | Networks | IPs | |---------|------|----------|-----| | management | Celilo CLI, Ansible, SSH | internal | 192.168.0.100 | | fw-main | iptables firewall (managed by Celilo) | internal, dmz, app, secure | 192.168.0.254, 10.0.10.1, 10.0.20.1, 10.0.30.1 | | fw-isp | Greenwave router simulator | internal, isp-external | 192.168.0.1, 100.100.0.100 | | fw-ext | Edge router + transparent HTTPS proxy (Squid) | isp-external, internet-external, real-internet | 100.100.0.101, 100.64.0.1, 172.30.0.5 | | comcast-resolver | Unbound recursive DNS resolver | isp-external, real-internet | 100.100.0.1, 172.30.0.4 | | root-dns | Knot authoritative DNS (root zone) | internet-external | 100.64.0.53 | | tld-dns | Knot authoritative DNS (.com, .org TLDs) | internet-external | 100.64.0.54 | | namecheap-dns | Knot DNS + DDNS API simulator | internet-external | 100.64.0.55 | | letsencrypt | Pebble ACME test server | internet-external | 100.64.0.100 | | apt-cache | apt-cacher-ng package proxy | internet-external, real-internet | 100.64.0.2, 172.30.0.2 |

Dynamic Test Machines

Added per-test via the network() builder API. Each is an Ubuntu 22.04 container running systemd, with SSH server, placed on the appropriate zone network.

| Example | Network | IP | Purpose | |---------|---------|-----|---------| | caddy | dmz | 10.0.10.10 | Caddy reverse proxy deployment | | idp | app | 10.0.20.100 | Identity provider deployment | | db | secure | 10.0.30.50 | Database deployment |

Routing

Traffic flows through explicit routing chains, mimicking a real network:

| From | To | Path | |------|----|------| | management | dmz/app/secure | via fw-main (192.168.0.254) | | management | internet | via fw-isp (192.168.0.1) → fw-ext | | dmz machines | internet | via fw-main → fw-isp → fw-ext | | Pebble | caddy (HTTP-01 challenge) | via fw-ext → fw-isp DNAT → fw-main DNAT → caddy |

All firewall/router containers run ip_forward=1 and MASQUERADE on their outbound interfaces.

DNS

The DNS system is a hybrid of simulated and real resolution:

Simulated domains (handled by the E2E DNS hierarchy):

• iamtheinternet.org — stub zone to namecheap-dns (100.64.0.55)
• park-your-domain.com — stub zone to namecheap-dns (Namecheap DDNS API)
• acme-v02.api.letsencrypt.org — local-data in Unbound pointing to Pebble (100.64.0.100)

Real domains (forwarded to actual DNS):

• Everything else (e.g., dl.cloudsmith.io, deb.debian.org) is forwarded by Unbound to 8.8.8.8 and 1.1.1.1 via the real-internet bridge.

This means Celilo's modules install real packages from the internet while all domain-specific operations (DDNS, ACME) go through the simulation.

Transparent HTTPS Proxy

A Squid proxy with SSL bumping runs on fw-ext, transparently intercepting outbound HTTPS traffic from the simulated network. This lets target machines curl https://dl.cloudsmith.io (to install Caddy from its apt repo) without explicit proxy configuration.

How it works:

1. iptables REDIRECT rules on fw-ext's ISP interface capture port 80/443 traffic
2. Exception: traffic to 100.64.0.0/24 (simulated services) passes through directly
3. Squid does SSL bump (MITM) with a pre-shared CA trusted by all machines
4. Squid resolves DNS via real public DNS and fetches from the real internet

Simulators

Namecheap Dynamic DNS

Runs on namecheap-dns (100.64.0.55). A Bun HTTP server on port 8080 that implements the Namecheap DDNS API:

GET /update?host=<host>&domain=<domain>&password=<password>&ip=<ip>

When called, it updates the Knot DNS zone file and signals a zone reload. DNS propagates through the full chain (namecheap-dns → tld-dns → root-dns → comcast-resolver).

Greenwave Router

Runs on fw-isp (192.168.0.1). A Bun HTTPS server implementing the C4000XG REST API subset:

• POST /cgi/cgi_action — Login/logout
• GET /cgi/cgi_get — Read config (public IP, port mappings)
• POST /cgi/cgi_set — Add/remove port forwarding rules

Port forwarding rules are applied via actual iptables commands, making NAT functional in the simulated network. The management HTTPS interface binds to the internal interface only (192.168.0.1), not the external interface.

Pebble (Let's Encrypt)

Uses the official Pebble ACME test server, wrapped in a custom Dockerfile that adds routing. Pebble uses the comcast-resolver for DNS and validates HTTP-01 challenges through the full routing chain.

The Pebble TLS certificate includes acme-v02.api.letsencrypt.org as a SAN, so Caddy's default ACME configuration works with zero changes (DNS resolves the real Let's Encrypt hostname to Pebble).

fw-ext automatically fetches Pebble's runtime ACME root CA from its management API (https://pebble:15000/roots/0) at startup, so curl from fw-ext can verify Caddy's ACME-issued certificates.

Manual Usage

Starting the Environment

cd e2e

# Infrastructure only (no target machines)
./bin/e2e-up

# With a caddy machine in the DMZ
./bin/e2e-up --caddy

# Full stack (caddy + idp + db)
./bin/e2e-up --full-stack

# Custom machine spec
./bin/e2e-up --custom '{"dmz":{"caddy":"10.0.10.10","web":"10.0.10.20"}}'

After startup, you're dropped into the management machine shell with tab completion for celilo commands (aliased as c).

Accessing Machines

# Reconnect to management (default)
./bin/e2e-shell

# Shell into any container
./bin/e2e-shell caddy
./bin/e2e-shell fw-main
./bin/e2e-shell fw-ext
./bin/e2e-shell namecheap-dns
./bin/e2e-shell letsencrypt

Management uses zsh (with celilo aliases). All other containers use bash.

Running the Manual Test Script

From the management shell (./bin/e2e-shell or after ./bin/e2e-up):

cd /celilo/modules

# System init
c system init --accept-defaults \
    network.dmz.subnet=10.0.10.0/24 \
    network.app.subnet=10.0.20.0/24 \
    network.secure.subnet=10.0.30.0/24 \
    network.internal.subnet=192.168.0.0/24 \
    primary_domain=iamtheinternet.org \
    [email protected] \
    dns.primary=100.100.0.1 \
    dns.fallback=1.0.0.1,8.8.8.8

# Import all modules
c module import namecheap
c module import greenwave
c module import iptables
c module import caddy

# Add machines (--ssh-user root needed for non-interactive)
c machine add 192.168.0.1 --ssh-user root --earmark greenwave
c machine add 192.168.0.254 --ssh-user root --earmark iptables
c machine add 10.0.10.10 --ssh-user root

# Pre-configure secrets and non-derivable config
c module secret set namecheap ddns_password test123
c module config set greenwave router_ip 192.168.0.1
c module secret set greenwave router_username admin
c module secret set greenwave router_password admin
c module config set iptables nat_ip 192.168.0.253
c module config set caddy hostname www
c module config set caddy acme_ca https://acme-v02.api.letsencrypt.org/dir

# Deploy (iptables auto-derives config from earmarked machine)
c module deploy namecheap --no-interactive
c module deploy greenwave --no-interactive
c module deploy iptables
c module deploy caddy --no-interactive

Verifying the Deployment

From fw-ext (the "internet" side):

./bin/e2e-shell fw-ext
curl -s https://www.iamtheinternet.org
# Expected: "Caddy reverse proxy is running"

Checking Status

./bin/e2e-status     # Shows containers, DNS, connectivity

Tearing Down

./bin/e2e-down           # Stop and remove everything
./bin/e2e-down --keep    # Stop but keep volumes (faster restart)

Writing Automated Tests

Tests use Vitest and the NetworkBuilder API to start isolated networks, run Celilo commands, and verify results.

Test Structure

import { afterAll, describe, expect, it } from 'vitest';
import { CADDY_DEPLOYMENT } from '../src/fixtures';
import type { NetworkHandle } from '../src/types';

describe('my deployment test', () => {
  let net: NetworkHandle;

  afterAll(async () => {
    await net?.stop(); // Always clean up
  });

  it('deploys and verifies', async () => {
    // 1. Start the network with desired machines
    net = await CADDY_DEPLOYMENT().start();

    // 2. Add machines, import modules, configure
    await net.celilo('machine add 10.0.10.10 --ssh-user root');
    await net.celilo('module import /celilo/modules/caddy');
    await net.celilo('module config set caddy hostname www');

    // 3. Deploy
    const result = await net.celilo('module deploy caddy --no-interactive');
    expect(result.exitCode).toBe(0);

    // 4. Verify from any container
    const curl = await net.exec('fw-ext', 'curl -s https://www.iamtheinternet.org');
    expect(curl.stdout).toContain('Caddy');
  }, 300_000); // Per-test timeout
});

NetworkHandle API

| Method | Description | |--------|-------------| | celilo(cmd, timeout?) | Run a celilo CLI command on the management machine | | exec(container, cmd, timeout?) | Execute a command in any container | | dig(name) | Resolve a DNS name from management | | waitFor(check, timeout, label) | Poll until a condition is true | | stop() | Tear down the entire network |

Fixtures

Pre-built network configurations in src/fixtures.ts:

CADDY_DEPLOYMENT()    // caddy machine in DMZ
FULL_STACK()          // caddy (dmz) + idp (app) + db (secure)
INFRASTRUCTURE_ONLY() // no dynamic machines

Custom Network Configurations

import { network } from '../src/network-builder';

const net = await network()
  .dmz({ caddy: '10.0.10.10', web: '10.0.10.20' })
  .app({ api: '10.0.20.100' })
  .secure({ db: '10.0.30.50' })
  .start();

Running Tests

cd e2e

# Run all E2E tests
bun run vitest run

# Run a specific test
bun run vitest run tests/caddy-deploy.test.ts

# Run with verbose output
bun run vitest run --reporter=verbose

Tests run sequentially (fixed CIDRs prevent parallelism). The container manager automatically cleans up leftover networks from previous failed runs.

Key Design Decisions

Systemd on target machines: Target machines run systemd as PID 1 (privileged mode) so Ansible's systemd module works naturally. Network setup runs as a oneshot systemd service at boot.

No test parallelism: Fixed network CIDRs mean only one test network can run at a time. Tests run sequentially via Vitest's singleFork config.

Bind-mounted source: The celilo source tree is bind-mounted at /celilo on the management container. Code changes take effect immediately without rebuilding images.

Docker image caching: Images are built once and cached. Only config/simulator changes require rebuilds. The management container uses the bind mount, so celilo code changes are free.

ACME via DNS interception: Instead of overriding Caddy's ACME URL, acme-v02.api.letsencrypt.org resolves to Pebble (100.64.0.100) in the simulated DNS. The caddy module's acme_ca variable allows pointing to Pebble's /dir endpoint (vs Let's Encrypt's /directory).

Directory Structure

e2e/
  bin/
    e2e-up              # Start interactive network
    e2e-down            # Tear down network
    e2e-shell           # Shell into containers
    e2e-status          # Show network status
  config/
    dns/                # Knot zone files and configs
    pebble/             # Pebble ACME config and TLS certs
    proxy/              # Squid transparent proxy config
    resolver/           # Unbound recursive resolver config
    routing/            # Per-container routing scripts
    ssh/                # SSH key generation
  docker/
    Dockerfile.*        # Container images
  docs/
    network-diagram.svg # Network topology diagram
  simulators/
    greenwave/          # C4000XG router REST API simulator
    namecheap-ddns/     # Namecheap DDNS API simulator
  src/
    container-manager.ts    # Docker compose orchestration
    docker-compose-generator.ts  # Generates compose YAML
    fixtures.ts             # Pre-built network configs
    network-builder.ts      # Fluent API for network setup
    types.ts                # Shared type definitions
  tests/
    caddy-deploy.test.ts    # Full caddy deployment with HTTPS
    smoke.test.ts           # Basic connectivity verification
    ...