Celilo Backend

Phase 0 implementation of Celilo's backend services.

Phase 0 Scope

CLI-only interface for:

• Module import and validation
• Manifest validation
• Variable resolution
• Template generation
• Secret management
• Zero-configuration modules (Phase 0 Part 4):
  • Automatic hostname/zone assignment from well-known capabilities
  • Automatic VMID/IP allocation (IPAM)
  • Automatic network config derivation from zones
  • Automatic VM resource defaults from manifests

Project Structure

celilo/backend/
├── src/
│   ├── db/
│   │   ├── schema.ts          # Drizzle ORM schema
│   │   ├── schema.test.ts     # Schema tests
│   │   ├── client.ts          # Database connection
│   │   └── migrate.ts         # Migration runner
│   ├── manifest/
│   │   ├── schema.ts          # Zod schemas for manifest validation
│   │   ├── validate.ts        # Manifest validation logic
│   │   └── validate.test.ts   # Validation tests
│   ├── module/
│   │   ├── import.ts          # Module import logic
│   │   └── import.test.ts     # Import tests
│   ├── variables/
│   │   ├── types.ts           # Variable resolution types
│   │   ├── parser.ts          # Variable parsing logic
│   │   ├── parser.test.ts     # Parser tests
│   │   ├── resolver.ts        # Variable resolution logic
│   │   ├── resolver.test.ts   # Resolver tests
│   │   ├── context.ts         # Resolution context builder
│   │   └── context.test.ts    # Context tests
│   ├── secrets/
│   │   ├── master-key.ts      # Master key generation and management
│   │   ├── master-key.test.ts # Master key tests
│   │   ├── encryption.ts      # AES-256-GCM encryption/decryption
│   │   └── encryption.test.ts # Encryption tests
│   ├── templates/
│   │   ├── types.ts           # Template generation types
│   │   ├── generator.ts       # Template generation logic
│   │   └── generator.test.ts  # Generation tests
│   └── index.ts               # Entry point
├── drizzle/                   # Generated migrations
├── package.json
├── tsconfig.json
├── biome.json                 # Linting/formatting config
└── drizzle.config.ts          # Drizzle Kit config

Database Schema

Tables

modules - Module metadata and manifest data

• id (TEXT, PK) - Module identifier
• name (TEXT) - Display name
• version (TEXT) - Semantic version
• description (TEXT) - Optional description
• state (TEXT) - Lifecycle state (IMPORTED, VALIDATED, CONFIGURED, etc.)
• manifest_data (JSON) - Full manifest content
• source_path (TEXT) - Original import path
• imported_at (TIMESTAMP) - Import timestamp
• updated_at (TIMESTAMP) - Last update timestamp
• error_message (TEXT) - Error details if state is ERROR

module_configs - User configuration key-value pairs

• id (INTEGER, PK, AUTO)
• module_id (TEXT, FK → modules.id) - Module reference
• key (TEXT) - Configuration key
• value (TEXT) - Configuration value
• created_at (TIMESTAMP)
• updated_at (TIMESTAMP)

capabilities - Registered capabilities from modules

• id (INTEGER, PK, AUTO)
• module_id (TEXT, FK → modules.id) - Provider module
• capability_name (TEXT) - Capability identifier (e.g., dns_external)
• version (TEXT) - Capability version
• data (JSON) - Capability data (nameserver, zone, etc.)
• registered_at (TIMESTAMP)

secrets - Encrypted secrets per module

• id (INTEGER, PK, AUTO)
• module_id (TEXT, FK → modules.id) - Module reference
• name (TEXT) - Secret name
• encrypted_value (TEXT) - AES-256-GCM encrypted value
• iv (TEXT) - Initialization vector
• auth_tag (TEXT) - Authentication tag
• created_at (TIMESTAMP)
• updated_at (TIMESTAMP)

All tables use CASCADE DELETE when module is removed.

Dependencies

Required

• Bun (v1.0+) - JavaScript runtime
• Ansible (v2.9+) - Required for Ansible Vault secret encryption
• Terraform (v1.0+) - Required for validating generated Terraform code

See ../SETUP.md for installation instructions.

Setup

# Install dependencies
bun install

# Generate migrations (if schema changed)
bunx drizzle-kit generate

# Run migrations
bun run db:migrate

# Run tests
bun test                    # Unit tests (216 tests)
bun run test:integration    # Integration tests (56 tests)

# Lint and format
bun run lint:fix
bun run format

Development

Common Development Workflows

Full development cycle:

# 1. Make code changes
vim src/module/import.ts

# 2. Run unit tests (< 1s)
bun test src/module/import.test.ts

# 3. Run all unit tests (includes zero-config integration tests)
bun test

# 4. Run CLI-based integration tests (10-20s)
bun run test:integration

# 5. Before commit: Run all tests
bun test && bun run test:integration

# 6. Before major release: Run slow tests (3-5min)
bun run test:integration-slow

Working with the CLI:

# Use the wrapper script (works from any directory)
../../celilo module list

# Or run directly from backend
bun run src/cli/index.ts module list

# Or use bun link for system-wide command
bun link
celilo module list  # From anywhere

Testing module changes:

# 1. Import test module
bun run src/cli/index.ts module import ../../modules/test-module

# 2. Configure
bun run src/cli/index.ts module config set test-module hostname test-app
bun run src/cli/index.ts module config set test-module container_ip 10.0.20.100

# 3. Generate
bun run src/cli/index.ts module generate test-module

# 4. Inspect generated files
ls -la /tmp/celilo/modules/test-module/generated/

# 5. Clean up
bun run src/cli/index.ts module remove test-module

Working with secrets:

# Set module secret
bun run src/cli/index.ts secret set homebridge api_key test-key-123

# Get Ansible Vault password (for inspecting secrets.yml)
bun run src/cli/index.ts system vault-password

# View encrypted Ansible secrets
ansible-vault view /tmp/celilo/modules/homebridge/generated/ansible/inventory/secrets.yml \
  --vault-password-file=<(bun run src/cli/index.ts system vault-password)

Zero-Configuration Workflow (Phase 0 Part 4):

# One-time system setup: Configure network zones
celilo system config set network.dmz.subnet "10.0.10.0/24"
celilo system config set network.dmz.gateway "10.0.10.1"
celilo system config set network.dmz.vlan "10"
celilo system config set network.dmz.bridge "vmbr0"

# Import module with well-known capability (e.g., Caddy with public_web)
celilo module import ./modules/caddy

# Configure ONLY app-specific settings (infrastructure auto-configured!)
celilo module config set caddy domain "example.com"

# Generate - everything else is automatic!
celilo module generate caddy

# Debug: Show all configuration (user + auto-derived)
celilo module show-config caddy

# Debug: Show zone and network settings
celilo module show-zone caddy

# IPAM Management: Reserve VMIDs/IPs for existing infrastructure
celilo ipam vmid reserve 2100 --reason "Existing Proxmox VM"
celilo ipam ip reserve 10.0.10.1-10.0.10.9 --zone dmz --reason "Infrastructure"

# IPAM Management: List allocations and reservations
celilo ipam list-allocations
celilo ipam vmid list-reservations
celilo ipam ip list-reservations

What Gets Auto-Configured:

• ✅ Hostname - From well-known capabilities (e.g., public_web → www)
• ✅ Zone - From well-known capabilities (e.g., public_web → dmz)
• ✅ VMID - Auto-allocated from 2100+ (IPAM)
• ✅ Container IP - Auto-allocated from zone subnet (IPAM)
• ✅ Gateway - From network.{zone}.gateway system config
• ✅ VLAN - From network.{zone}.vlan system config
• ✅ Subnet - From network.{zone}.subnet system config
• ✅ Bridge - From network.{zone}.bridge system config
• ✅ VM Resources - cores, memory, disk, storage from manifest defaults

See ../../reference/MODULE_DEVELOPMENT_GUIDE.md for complete zero-config documentation.

Database Location

Default locations:

• macOS: ~/Library/Application Support/celilo/celilo.db
• Linux: /var/lib/celilo/celilo.db
• Development (ENVIRONMENT=dev): ./celilo-data/celilo.db

Override: Set CELILO_DB_PATH environment variable to use a custom location.

Override base directory: Set CELILO_DATA_DIR to change where all Celilo data (including database) is stored.

Testing Strategy

Three test tiers (see TESTING_STRATEGY.md for details):

Tier 1: Unit Tests (< 1 second)

bun test                    # Run all unit tests (494 tests)
bun test schema.test.ts     # Run specific test file
bun test --watch            # Watch mode for active development

Tier 2: Fast Integration Tests (10-20 seconds)

bun run test:integration    # All integration tests (run via bun test)

# Coverage:
# - Module import/validation via CLI
# - Configuration management
# - Template generation
# - Ansible/Terraform generation
# - Well-known capability auto-assignment
# - IPAM auto-allocation
# - Zone-based networking
# - VM resource defaults

# What gets skipped:
# - Nix builds
# - Docker builds
# - Slow module builds (Caddy)

Tier 3: Slow Integration Tests (3-5 minutes)

bun run test:integration-slow  # Full builds + end-to-end

# Includes everything from fast tests PLUS:
# - Full Nix builds (Caddy with modules)
# - Docker builds
# - Complete module packaging

Before committing (MANDATORY per Rule 7.4):

bun test && bun run test:integration

Before releasing:

bun test && bun run test:integration && bun run test:integration-slow

Database Studio

View database contents with Drizzle Studio:

bun run db:studio

Opens web interface at http://localhost:4983

Debugging

Enable verbose output:

# Set log level
export CONDUCTOR_LOG_LEVEL=debug
bun run src/cli/index.ts module import ./path

# Or use Bun's debugger
bun --inspect run src/cli/index.ts module import ./path

Inspect database:

# Use Drizzle Studio
bun run db:studio

# Or sqlite3 directly
sqlite3 celilo.db
sqlite> .tables
sqlite> SELECT * FROM modules;
sqlite> SELECT * FROM module_configs WHERE module_id = 'homebridge';

Check generated files:

# Generated files location
ls -la /tmp/celilo/modules/<module-id>/generated/

# View Terraform
cat /tmp/celilo/modules/<module-id>/generated/terraform/main.tf

# View Ansible
cat /tmp/celilo/modules/<module-id>/generated/ansible/playbook.yml

# Check encrypted secrets
ansible-vault view /tmp/celilo/modules/<module-id>/generated/ansible/inventory/secrets.yml \
  --vault-password-file=<(bun run src/cli/index.ts system vault-password)

Architecture Decisions

Why Drizzle ORM?

• Type-safe schema-first approach
• Excellent TypeScript inference
• Lightweight (no heavy runtime)
• SQL-like API (easier for developers familiar with SQL)

Why SQLite?

• Single-file database (easy backup/restore)
• No separate server process needed
• Sufficient for home lab scale
• ACID compliant with WAL mode

Foreign Key Cascades

All child tables (configs, capabilities, secrets) cascade delete when parent module is removed. This ensures no orphaned data.

JSON Columns

manifest_data and capability data use JSON columns for flexibility. These are validated with Zod schemas in application code.

Manifest Validation

Schema (src/manifest/schema.ts)

Zod schemas for validating module manifest structure:

• ModuleManifestSchema - Complete manifest validation
• VariableDeclareSchema - Variable declarations
• VariableUseSchema - Variable usage from capabilities
• CapabilityRequirementSchema - Required capabilities
• CapabilityProviderSchema - Provided capabilities
• LifecycleHookSchema - Lifecycle hooks (on_install, health_check, etc.)

Validation Functions (src/manifest/validate.ts)

validateManifest(yamlContent: string): ValidationResult

• Parses YAML and validates against schema
• Returns success with typed manifest or errors with path/message
• Policy function - no side effects

validateCapabilityRequirements(manifest, availableCapabilities): ValidationError | null

• Checks that all required capabilities exist in system
• Caller provides list of available capabilities (from database)

validateVariableSources(manifest): ValidationError | null

• Validates that capability references in variables match required capabilities
• Ensures consistency between requires.capabilities and variables.uses

Example Usage

import { validateManifest } from './manifest/validate';

const yamlContent = await readFile('./modules/homebridge/manifest.yml', 'utf-8');
const result = validateManifest(yamlContent);

if (!result.success) {
  console.error('Validation errors:', result.errors);
  return;
}

const manifest = result.data;
console.log(`Module: ${manifest.name} v${manifest.version}`);

Module Import

Import Functions (src/module/import.ts)

Following Rule 10.1, functions are separated by responsibility:

Policy Functions (validation, no side effects):

• validateModuleDirectory(sourcePath) - Checks directory exists and has manifest.yml
• readModuleManifest(sourcePath) - Reads and validates manifest from YAML

Execution Functions (perform I/O):

• copyModuleFiles(sourcePath, targetPath) - Recursively copies all module files
• insertModuleToDb(manifest, targetPath, db?) - Inserts module record to database
• moduleExists(moduleId, db?) - Checks if module already exists

Orchestration Function:

• importModule(options) - Main entry point, coordinates all steps

Import Process

1. Validate directory structure and manifest
2. Check module doesn't already exist
3. Copy files to target location (/tmp/celilo/modules/{module-id}/)
4. Insert module record to database with state IMPORTED

Example Usage

import { importModule } from './module/import';

const result = await importModule({
  sourcePath: './modules/homebridge',
  targetBasePath: '/data/celilo/modules',
});

if (!result.success) {
  console.error('Import failed:', result.error);
  return;
}

console.log(`Imported module: ${result.moduleId}`);
console.log(`Files copied to: ${result.targetPath}`);

Dependency Injection

Functions accept optional db parameter for testing (Rule 2.3):

// Production - uses global singleton
const exists = moduleExists('homebridge');

// Testing - inject test database
const testDb = createDbClient({ path: './test.db' });
const exists = moduleExists('homebridge', testDb);

Variable Resolution

System Overview (src/variables/)

The variable resolution system follows Rule 10.1 with clear separation of concerns:

Parser (parser.ts) - Policy function:

• Extracts variable references from template strings
• Pattern: $type:path.to.value
• Validates variable format

Resolver (resolver.ts) - Policy + Orchestration:

• resolveVariable() - Resolves single variable from context
• resolveTemplate() - Resolves all variables in template

Context (context.ts) - Execution function:

• buildResolutionContext() - Fetches data from database
• buildContextFromData() - Creates context from explicit data (testing)

Variable Types

$self:path - Module's own configuration

• Example: $self:container_ip → 192.168.0.50
• Source: module_configs table

$system:path - System-wide configuration

• Example: $system:management.ip → 192.168.0.10
• Source: Hardcoded defaults (Phase 0), database (Phase 1+)

$secret:name - Encrypted secrets

• Example: $secret:api_key → decrypted_value
• Source: secrets table (Phase 0: plaintext, Phase 0.5: encrypted)

$capability:name.path - Capability provider data

• Example: $capability:dns_external.nameserver → ns1.example.com
• Source: capabilities table
• Format: capability_name.data.path

Example Usage

import { buildResolutionContext, resolveTemplate } from './variables';

// Build context from database
const context = await buildResolutionContext('homebridge');

// Resolve template
const template = `
hostname: $self:hostname
ip: $self:container_ip
gateway: $system:management.ip
dns: $capability:dns_external.nameserver
api_key: $secret:api_key
`;

const result = resolveTemplate(template, context);

if (result.success) {
  console.log(result.content);
  // hostname: homebridge
  // ip: 192.168.0.50
  // gateway: 192.168.0.10
  // dns: ns1.example.com
  // api_key: secret123
} else {
  console.error('Resolution errors:', result.errors);
}

Resolution Process

1. Parse - Extract all variable references from template
2. Resolve - For each variable:
   • Determine type (self, system, secret, capability)
   • Lookup value from appropriate data source in context
   • Handle nested paths for capabilities
3. Replace - Substitute all variables with resolved values
4. Return - Success with content or errors with details

Testing

All functions are testable without database:

import { buildContextFromData, resolveTemplate } from './variables';

const context = buildContextFromData('test-module', {
  selfConfig: { ip: '192.168.0.50' },
  secrets: { key: 'secret' },
  capabilities: { dns: { server: 'ns1' } },
});

const result = resolveTemplate('ip: $self:ip', context);
// result.success === true
// result.content === 'ip: 192.168.0.50'

Secret Encryption

System Overview (src/secrets/)

AES-256-GCM encryption for module secrets following Rule 10.1 separation:

Master Key (master-key.ts):

• generateMasterKey() - Policy: Generates 32-byte key
• isValidMasterKey() - Policy: Validates key format
• writeMasterKey() - Execution: Writes to file with 0600 permissions
• readMasterKey() - Execution: Reads from file
• getOrCreateMasterKey() - Orchestration: Ensures key exists
• masterKeyExists() - Check if key file exists

Encryption (encryption.ts):

• encryptSecret() - Policy: AES-256-GCM encryption with random IV
• decryptSecret() - Policy: Decrypts with key verification
• isValidEncryptedSecret() - Policy: Validates encrypted data format

Encryption Details

Algorithm: AES-256-GCM (Authenticated Encryption with Associated Data)

• Key: 256 bits (32 bytes)
• IV: 128 bits (16 bytes) - randomly generated per encryption
• Auth Tag: 128 bits (16 bytes) - ensures integrity

Storage Format:

{
  encryptedValue: string,  // Hex-encoded ciphertext
  iv: string,              // Hex-encoded initialization vector
  authTag: string          // Hex-encoded authentication tag
}

Master Key Management

Location:

• Development: /tmp/celilo/master.key (configurable via CELILO_MASTER_KEY_PATH)
• Production: /etc/celilo/master.key

Generation:

import { getOrCreateMasterKey } from './secrets/master-key';

// Automatically generates if missing
const masterKey = await getOrCreateMasterKey();

File Permissions: 0600 (owner read/write only)

Example Usage

import { getOrCreateMasterKey } from './secrets/master-key';
import { encryptSecret, decryptSecret } from './secrets/encryption';

// Get master key (generates if missing)
const masterKey = await getOrCreateMasterKey();

// Encrypt a secret
const encrypted = encryptSecret('my-api-key-12345', masterKey);
console.log(encrypted);
// {
//   encryptedValue: '4a3b2c1d...',
//   iv: 'f1e2d3c4...',
//   authTag: 'a9b8c7d6...'
// }

// Decrypt the secret
const plaintext = decryptSecret(encrypted, masterKey);
console.log(plaintext); // 'my-api-key-12345'

Security Features

1. Authenticated Encryption: GCM mode provides both confidentiality and integrity
2. Random IVs: Each encryption uses unique IV, same plaintext → different ciphertext
3. Key Validation: Master key length enforced (32 bytes)
4. Tamper Detection: Auth tag verification prevents corrupted/modified ciphertext
5. Fail-Fast: Invalid keys or corrupted data throw clear errors

Error Handling

try {
  const decrypted = decryptSecret(encrypted, masterKey);
} catch (error) {
  // Possible errors:
  // - "Master key must be 32 bytes"
  // - "Invalid encrypted secret: missing required fields"
  // - "Invalid IV length: expected 16, got N"
  // - "Failed to decrypt secret: [reason]" (wrong key, corrupted data)
}

Testing

All functions are pure (no global state):

import { generateMasterKey, encryptSecret, decryptSecret } from './secrets';

const masterKey = generateMasterKey();
const encrypted = encryptSecret('test-secret', masterKey);
const decrypted = decryptSecret(encrypted, masterKey);
// decrypted === 'test-secret'

Ansible Vault Integration

System Overview (src/ansible/, src/secrets/vault.ts)

Celilo uses Ansible Vault to encrypt secrets in generated Ansible configurations, ensuring no plaintext secrets are written to disk.

Key Components:

• vault.ts - Derives vault password from master key (deterministic)
• ansible/secrets.ts - Generates and encrypts secrets.yml file
• ansible-resolver.ts - Converts $secret: variables to Jinja2 {{ }} syntax

Encryption Flow

1. Decrypt secrets from database (using master key)
2. Format as YAML with all module secrets
3. Derive vault password from master key (SHA-256)
4. Encrypt with ansible-vault (AES-256)
5. Write encrypted file to ansible/inventory/secrets.yml

Vault Password Derivation

import { deriveVaultPassword, getVaultPassword } from './secrets/vault';

// Deterministic derivation from master key
const masterKey = await getOrCreateMasterKey();
const vaultPassword = deriveVaultPassword(masterKey);
// Same master key always produces same vault password

// Or use orchestration function
const vaultPassword = await getVaultPassword();

Algorithm: SHA-256(concat("ansible-vault:", masterKey))

Properties:

• Deterministic (same master key → same vault password)
• Domain-separated (prevents key reuse attacks)
• Standard hash output (64 hex characters)

Generated Ansible Structure

Playbook (playbook.yml):

---
- name: Deploy Module
  hosts: module-host
  become: true

  vars_files:
    - inventory/secrets.yml  # Encrypted with ansible-vault

  roles:
    - module-role

Secrets File (inventory/secrets.yml):

$ANSIBLE_VAULT;1.1;AES256
35646166616130633832363334383234306139626264373935623630393937313639623334356138
6561383035303565323364373934306632336461626562630a386237663365633039396631623639
...

Templates (e.g., templates/config.json):

{
  "api_key": "{{ api_key }}",
  "password": "{{ db_password }}"
}

Variable Resolution

Ansible templates use different resolution from Terraform:

Terraform (resolved at generation time):

• $self:hostname → "myhost"
• $system:dns.primary → "192.168.0.1"
• $secret:api_key → "actual_secret_value" (plaintext in file!)

Ansible (resolved at runtime):

• $self:hostname → "myhost" (resolved at generation)
• $system:dns.primary → "192.168.0.1" (resolved at generation)
• $secret:api_key → {{ api_key }} (Jinja2 variable, resolved from encrypted secrets.yml)

CLI Usage

Get vault password:

# Display vault password
celilo system vault-password

# Use with ansible-vault
ansible-vault view inventory/secrets.yml \
  --vault-password-file=<(celilo system vault-password)

# Edit encrypted secrets
ansible-vault edit inventory/secrets.yml \
  --vault-password-file=<(celilo system vault-password)

Run Ansible playbook:

cd /tmp/celilo/modules/homebridge/generated/ansible

# Option 1: Process substitution (recommended)
ansible-playbook playbook.yml \
  --vault-password-file=<(celilo system vault-password)

# Option 2: Environment variable
export ANSIBLE_VAULT_PASSWORD=$(celilo system vault-password)
echo "$ANSIBLE_VAULT_PASSWORD" | ansible-playbook playbook.yml --vault-password-file=/dev/stdin

Security Considerations

Benefits:

• ✅ No plaintext secrets on disk
• ✅ Industry-standard encryption (Ansible Vault AES-256)
• ✅ Deterministic (same master key → same vault password)
• ✅ Works with standard Ansible tooling
• ✅ Secrets encrypted independently from celilo database

Requirements:

• ⚠️ Ansible must be installed - Generation fails without ansible-vault
• ⚠️ Master key must be protected (file permissions 0600)
• ⚠️ Vault password derivation is deterministic (anyone with master key can decrypt)

Threat Model:

• Protects against: Accidental secret exposure (logs, commits, backups)
• Does NOT protect against: Attacker with master key access
• Assumption: Master key is stored securely with restricted file permissions

Implementation Details

Files:

• src/secrets/vault.ts - Vault password derivation
• src/ansible/secrets.ts - Secrets file generation and encryption
• src/variables/ansible-resolver.ts - Jinja2 variable conversion
• src/templates/generator.ts - Integration with template generation

Functions:

• deriveVaultPassword(masterKey) - Deterministic password derivation
• generateAnsibleSecrets(moduleId, outputPath, db) - Full generation pipeline
• encryptWithAnsibleVault(yamlContent, password) - Ansible Vault wrapper
• convertSecretsToJinja(content, context) - Variable conversion for Ansible

Testing

Unit Tests (13 tests):

bun test src/secrets/vault.test.ts
bun test src/variables/ansible-resolver.test.ts

Integration Tests:

bun run test:integration
# Validates:
# - Secrets encrypted with ansible-vault
# - Ansible templates use Jinja2 variables
# - Vault password command works
# - Generated secrets can be decrypted

Manual Verification:

# 1. Generate module
celilo module generate homebridge

# 2. Check secrets file is encrypted
head /tmp/celilo/modules/homebridge/generated/ansible/inventory/secrets.yml
# Should start with: $ANSIBLE_VAULT;1.1;AES256

# 3. Decrypt and verify
ansible-vault view /tmp/celilo/modules/homebridge/generated/ansible/inventory/secrets.yml \
  --vault-password-file=<(celilo system vault-password)
# Should show YAML with secret values

# 4. Check templates use Jinja2
cat /tmp/celilo/modules/homebridge/generated/ansible/roles/homebridge/templates/config.json
# Should contain {{ variable_name }}, not plaintext secrets

Template Generation

System Overview (src/templates/)

Template generation following Rule 10.1 separation:

Policy Functions:

• isTemplateFile(filename) - Checks if file has template extension (.tpl, .j2)
• getOutputFilename(template) - Removes template extension from filename

Execution Functions:

• discoverTemplateFiles(baseDir) - Recursively finds template files
• readTemplateFiles(modulePath, paths) - Reads template content from disk
• writeGeneratedFiles(outputPath, files) - Writes generated files to disk

Orchestration Function:

• generateTemplates(options) - Main entry point, coordinates entire generation process

Template Directories

Celilo looks for templates in:

• terraform/ - Terraform configuration templates
• ansible/ - Ansible playbook templates

Both directories are searched recursively.

Template Extensions

• .tpl - Generic template files (Terraform style)
• .j2 - Jinja2 template files (Ansible style)

Output files have extension removed:

• main.tf.tpl → main.tf
• playbook.yml.j2 → playbook.yml

Generation Process

1. Discover - Find all template files in terraform/ and ansible/ directories
2. Build Context - Load module config, secrets, and capabilities from database
3. Read - Load template content from files
4. Resolve - Replace variables using variable resolution system
5. Write - Save generated files to output directory
6. Return - Success with file list or detailed error messages

Example Usage

import { generateTemplates } from './templates/generator';

const result = await generateTemplates({
  moduleId: 'homebridge',
  modulePath: '/data/modules/homebridge',
  outputPath: '/data/modules/homebridge/generated',
});

if (result.success) {
  console.log(`Generated ${result.files.length} files:`);
  for (const file of result.files) {
    console.log(`  - ${file.path}`);
  }
} else {
  console.error('Generation failed:', result.error);
}

Module Structure

modules/homebridge/
├── manifest.yml
├── terraform/
│   ├── main.tf.tpl          # → generated/terraform/main.tf
│   ├── variables.tf.tpl     # → generated/terraform/variables.tf
│   └── outputs.tf.tpl       # → generated/terraform/outputs.tf
└── ansible/
    ├── playbook.yml.tpl     # → generated/ansible/playbook.yml
    └── roles/
        └── homebridge/
            └── tasks/
                └── main.yml.tpl  # → generated/ansible/roles/homebridge/tasks/main.yml

Template Example

Input (terraform/main.tf.tpl):

resource "proxmox_lxc" "container" {
  hostname = "$self:hostname"
  cores    = $self:cores
  memory   = $self:memory

  network {
    name   = "eth0"
    bridge = "vmbr0"
    ip     = "$self:container_ip/24"
    gw     = "$system:management.ip"
  }

  provisioner "ansible" {
    playbook = "./ansible/playbook.yml"
  }
}

Output (generated/terraform/main.tf):

resource "proxmox_lxc" "container" {
  hostname = "homebridge"
  cores    = 2
  memory   = 2048

  network {
    name   = "eth0"
    bridge = "vmbr0"
    ip     = "192.168.0.50/24"
    gw     = "192.168.0.10"
  }

  provisioner "ansible" {
    playbook = "./ansible/playbook.yml"
  }
}

Error Handling

Clear error messages for common failures:

// Module path doesn't exist
{ success: false, error: "Module path does not exist: /path" }

// No templates found
{ success: false, error: "No template files found in module" }

// Variable resolution failed
{
  success: false,
  error: "Failed to resolve variables in templates:\n" +
         "terraform/main.tf:\n" +
         "  $self:missing_var: Self variable 'missing_var' not found in module configuration"
}

// File write failed
{ success: false, error: "Failed to write generated files", details: Error }

Integration

Template generation integrates with:

• Variable Resolution - Resolves all $self:, $system:, $secret:, $capability: variables
• Database - Loads module configuration and capability data
• File System - Reads templates and writes generated code

Next Steps

Phase 0 Part 1.7:

1. ✅ Database layer
2. ✅ Manifest validation
3. ✅ Module import logic
4. ✅ Variable resolution system
5. ✅ Secret encryption (AES-256-GCM)
6. ✅ Template generation (File I/O + variable resolution)
7. CLI interface (Commands + user interaction)

CLI Interface

System Overview (src/cli/)

The CLI interface provides command-line tools for all Phase 0 operations following Rule 10.1 separation:

Parser (parser.ts) - Policy functions:

• parseArguments() - Parses command-line arguments into structured format
• validateRequiredArgs() - Validates argument count
• getArg(), getFlag(), hasFlag() - Type-safe accessors

Commands (commands/) - Orchestration functions:

• module-import.ts - Import modules from directory
• module-list.ts - List installed modules
• module-config.ts - Get/set module configuration
• module-generate.ts - Generate templates for modules
• secret-set.ts - Set encrypted secrets

Entry Point (index.ts) - Main orchestrator:

• runCli() - Routes commands to handlers
• main() - Entry point with error handling and exit codes

Commands

Module Import

celilo module import <path> [--target <path>]

# Example
celilo module import ./modules/homebridge
celilo module import ./modules/homebridge --target /data/celilo/modules

Module List

celilo module list

# Output
Installed modules:

homebridge (v1.0.0) - IMPORTED
  Bridge for HomeKit accessories

pihole (v2.1.0) - CONFIGURED

Module Configuration

# Set configuration value
celilo module config set <module-id> <key> <value>

# Get specific value
celilo module config get <module-id> [key]

# Examples
celilo module config set homebridge hostname mybridge
celilo module config set homebridge container_ip 192.168.0.50
celilo module config get homebridge hostname
celilo module config get homebridge  # Get all config

Module Generate

celilo module generate <module-id> [--output <path>]

# Examples
celilo module generate homebridge
celilo module generate homebridge --output ./generated

Secret Management

celilo secret set <module-id> <name> <value>

# Example
celilo secret set homebridge api_key mykey123

System Configuration

# Set system-wide config
celilo system config set <key> <value>
celilo system config get [key]

# Get Ansible Vault password
celilo system vault-password

# Examples
celilo system config set dns.primary 192.168.0.1
celilo system config get dns.primary
ansible-vault view secrets.yml --vault-password-file=<(celilo system vault-password)

Help

celilo help
celilo --help
celilo -h
celilo module --help  # Command-specific help

Argument Parsing

The CLI uses a simple argument parser without external dependencies:

Format: celilo <command> [subcommand] [args...] [--flags]

Parsing Rules:

• First argument is the command (module, secret, help)
• Second argument is the subcommand if it doesn't start with --
• Remaining arguments are positional args or flags
• Flags start with -- and can be boolean (--verbose) or string (--target /path)

Examples:

# Command only
celilo help

# Command + subcommand
celilo module list

# Command + subcommand + args
celilo module import ./path

# Command + subcommand + args + flags
celilo module generate homebridge --output ./out

# Nested subcommand
celilo module config set homebridge hostname myhost

Error Handling

All commands return structured results:

interface CommandSuccess {
  success: true;
  message: string;
  data?: unknown;
}

interface CommandError {
  success: false;
  error: string;
  details?: unknown;
}

Exit Codes:

• 0 - Success
• 1 - Error (validation, not found, etc.)

Error Messages include usage hints:

Error: Missing required arguments. Expected 1, got 0

Usage: celilo module import <path> [--target <path>]

Testing

All CLI components have comprehensive tests:

• parser.test.ts - Argument parsing (28 tests)
• cli.test.ts - Integration tests (23 tests)

Tests use environment variables to configure database and master key paths for isolation.

Entry Point

The main entry point is src/cli/index.ts, which exports main():

export async function main(): Promise<void> {
  const result = await runCli(process.argv);
  // Handle output and exit codes
}

Recommended: Use the top-level wrapper script (from anywhere):

/path/to/celilo/celilo module list
/path/to/celilo/celilo system config get

Or run directly from backend directory:

bun run src/cli/index.ts module list
bun run src/cli/index.ts module import ./path

Or build a binary:

bun build src/cli/index.ts --compile --outfile celilo
./celilo module list

Architecture

CLI Orchestration:

User Input
   ↓
parseArguments()   [Policy - parse and validate]
   ↓
runCli()          [Orchestration - route to handler]
   ↓
handleModuleXxx()  [Orchestration - coordinate operations]
   ↓
importModule()     [Execution - database, filesystem]
generateTemplates()
etc.

Separation of Concerns:

• Parser - Pure policy functions, no side effects
• Handlers - Orchestrate multiple operations
• Core modules - Execute actual work (import, generate, etc.)
• Entry point - Handle errors and exit codes

All commands integrate with existing modules:

• Database - Module metadata, configuration, secrets
• Manifest validation - Parse and validate manifests
• Module import - Copy files and insert to database
• Variable resolution - Resolve template variables
• Secret encryption - Encrypt/decrypt with master key
• Template generation - Generate Terraform/Ansible code

Troubleshooting

Common Errors and Solutions

"Module ID must use kebab-case"

Error:

Error: Manifest validation failed
Module ID must use kebab-case (lowercase letters, numbers, hyphens between segments)

Cause: Module ID uses underscores, uppercase, or invalid characters.

Solution: Use kebab-case for all module IDs:

# ❌ WRONG
metadata:
  name: dns_external   # Underscores

# ✅ CORRECT
metadata:
  name: dns-external   # Kebab-case

Pattern: /^[a-z0-9]+(-[a-z0-9]+)*$/

"Variable resolution failed: Self variable not found"

Error:

Failed to resolve variables in templates:
terraform/main.tf:
  $self:container_ip: Self variable 'container_ip' not found in module configuration

Cause: Template references variable that isn't configured.

Solution: Set the missing variable:

celilo module config set <module-id> container_ip 10.0.20.100

Or check for typos in template:

# Check spelling
ip = "$self:container_ip"  # Not 'containerIP', 'container-ip', etc.

"ENOENT: no such file or directory" (path with spaces)

Error:

Error: ENOENT: no such file or directory, open '/Users/user/Application Support/celilo/...'

Cause: Unquoted path in shell command with spaces.

Solution: Always quote paths in shell commands:

// ❌ WRONG
execSync(`cd ${modulePath} && make build`);

// ✅ CORRECT
execSync(`cd "${modulePath}" && make build`);

// ✅ BEST
import { shellEscape } from '@/utils/shell';
execSync(`cd ${shellEscape(modulePath)} && make build`);

"Drizzle where clause returns all rows"

Error: Query returns all modules instead of filtering by ID.

Cause: Using JavaScript operators instead of Drizzle operators.

Solution: Use Drizzle operator functions:

import { eq } from 'drizzle-orm';

// ❌ WRONG - returns ALL rows!
db.select().from(modules).where(modules.id === 'homebridge');

// ✅ CORRECT - filters correctly
db.select().from(modules).where(eq(modules.id, 'homebridge'));

Always use: eq(), ne(), gt(), lt(), and(), or(), etc.

"ansible-vault: command not found"

Error:

Error: Failed to encrypt secrets with ansible-vault
ansible-vault: command not found

Cause: Ansible is not installed or not in PATH.

Solution: Install Ansible:

# macOS
brew install ansible

# Ubuntu/Debian
sudo apt install ansible

# Verify
ansible-vault --version

"Test passes alone, fails in suite"

Error: Test passes when run individually, fails when run with others.

Cause: State leakage between tests (shared database, temp files, environment).

Solution: Add proper setup/teardown:

import { describe, test, beforeEach, afterEach } from 'bun:test';

describe('Feature Tests', () => {
  let testDb: Database;
  let tempDir: string;

  beforeEach(async () => {
    testDb = await setupTestDatabase();      // Fresh database
    tempDir = await createTempDirectory();   // Fresh directory
  });

  afterEach(async () => {
    await cleanupTestDatabase(testDb);       // MUST await!
    await removeTempDirectory(tempDir);      // MUST await!
  });

  test('isolated test', async () => {
    // Test runs with clean state
  });
});

"Cannot find module '@/something'"

Error:

Error: Cannot find module '@/module/import'

Cause: TypeScript path aliases not resolved.

Solution: Check tsconfig.json has correct paths:

{
  "compilerOptions": {
    "baseUrl": ".",
    "paths": {
      "@/*": ["src/*"]
    }
  }
}

And run with Bun (which respects tsconfig paths):

bun run src/cli/index.ts  # ✅ Works
node src/cli/index.ts     # ❌ Doesn't resolve @/ aliases

"Master key permission denied"

Error:

Error: EACCES: permission denied, open '/etc/celilo/master.key'

Cause: Master key file has wrong permissions or wrong owner.

Solution: Fix permissions:

# Check current permissions
ls -la /tmp/celilo/master.key

# Fix permissions (owner read/write only)
chmod 600 /tmp/celilo/master.key

# Or regenerate (WARNING: invalidates all encrypted secrets!)
rm /tmp/celilo/master.key
celilo system vault-password  # Generates new key

"Migration already exists"

Error:

Error: Migration 0005_add_capability_secrets.sql already exists

Cause: Schema changed but migration not generated or conflicting migration number.

Solution: Check for duplicates:

# List migrations
ls drizzle/

# Remove duplicate or conflicting migration
rm drizzle/0005_duplicate.sql

# Regenerate
bunx drizzle-kit generate

Debug Checklist

When debugging issues, check:

1. Test isolation:

   • [ ] beforeEach/afterEach properly clean up
   • [ ] Tests can run in any order
   • [ ] Tests use isolated database/filesystem

2. Path handling:

   • [ ] All shell commands quote paths
   • [ ] Tested with paths containing spaces
   • [ ] No hardcoded absolute paths

3. Drizzle queries:

   • [ ] Using operator functions (eq(), not ===)
   • [ ] Importing operators from 'drizzle-orm'
   • [ ] Not using callback form in where()

4. Identifier naming:

   • [ ] Module IDs use kebab-case
   • [ ] No underscores or uppercase
   • [ ] Pattern: /^[a-z0-9]+(-[a-z0-9]+)*$/

5. Variable resolution:

   • [ ] Using colon syntax ($self:var, not $self.var)
   • [ ] Simple syntax for standalone values
   • [ ] Braced syntax for concatenation (${self:disk}G)

6. Secrets:

   • [ ] Master key exists and has 0600 permissions
   • [ ] Ansible installed for vault encryption
   • [ ] Secrets encrypted in database

Getting Help

Documentation:

• CLAUDE.md - Engineering standards
• TESTING_STRATEGY.md - Testing approach
• IDENTIFIER_NAMING_CONVENTIONS.md - Naming rules
• TEMPLATE_VARIABLE_SYNTAX.md - Variable syntax

Debugging Tools:

• Drizzle Studio: bun run db:studio
• Test watch mode: bun test --watch
• Verbose logs: export CONDUCTOR_LOG_LEVEL=debug

Next Steps

Phase 0 Part 1.7:

1. ✅ Database layer
2. ✅ Manifest validation
3. ✅ Module import logic
4. ✅ Variable resolution system
5. ✅ Secret encryption (AES-256-GCM)
6. ✅ Template generation (File I/O + variable resolution)
7. ✅ CLI interface (Commands + user interaction)

Phase 0 Part 1 is complete!

Next: Phase 0 Part 2 - Web UI (React + tRPC)