@c4a/daemon

A lightweight daemon framework for building persistent background agents that communicate over WebSocket using JSON-RPC 2.0. Originally built for C4A (Context For AI) to bridge remote servers with local filesystems and Git repositories, but designed as a general-purpose daemon toolkit.

What It Does

@c4a/daemon provides two standalone executables:

• c4a-daemon — A long-running agent process that connects to a remote server via WebSocket, receives RPC commands, and executes them locally. Built-in handlers cover filesystem operations, Git repository inspection, and remote Git clone with progress reporting. Supports path-based access control, automatic reconnection with exponential backoff, and heartbeat-based liveness detection.

• c4a-daemon-scheduler — A process pool manager that spawns, monitors, and recycles daemon instances on demand. Exposes a simple HTTP API for requesting daemons by logical ID, with automatic idle eviction (LRU), orphan cleanup on restart, and capacity limits.

Architecture

                           ┌─────────────────────────────────────────────┐
                           │              C4A API Server                 │
                           │                                             │
                           │  DaemonManager          (WebSocket hub)     │
                           │    ├─ handshake/heartbeat management        │
                           │    ├─ bidirectional JSON-RPC dispatch       │
                           │    └─ online/offline broadcast              │
                           │                                             │
                           │  DaemonRpc               (RPC routing)      │
                           │    ├─ resolveMachineId (daemon / cloud)     │
                           │    ├─ callLibraryRpc → sendRpcRequest       │
                           │    └─ auto-retry on transient disconnect    │
                           └─────────┬──────────────────┬────────────────┘
                                     │ ws://            │ HTTP
                    ┌────────────────┘                  └──────────────────┐
                    │                                                      │
          ┌─────────▼──────────┐                            ┌──────────────▼──────────┐
          │    c4a-daemon       │                            │  c4a-daemon-scheduler   │
          │   (user machine)    │                            │  (cloud / CI host)      │
          │                     │                            │                          │
          │  WebSocketClient    │                            │  DaemonPool              │
          │   ├─ auto-reconnect │                            │   ├─ spawn c4a-daemon    │
          │   ├─ heartbeat      │                            │   ├─ PID file tracking   │
          │   └─ RPC dispatch   │                            │   ├─ idle sweep (LRU)    │
          │                     │                            │   ├─ orphan cleanup       │
          │  RpcHandler         │                            │   └─ capacity limit       │
          │   └─ 15 handlers    │                            │                          │
          │     ├─ fs/*         │         ┌─────────────┐    │  HTTP API                │
          │     ├─ index/*      │         │ c4a-daemon   │◄───┤   POST /daemons         │
          │     ├─ vcs/*        │         │ (child proc) │    │   GET  /daemons/:id     │
          │     └─ config/*     │
          └────────────────────┘         │  ...         │    │   DELETE /daemons/:id   │
                    │                     └──────┬──────┘    │   GET  /health          │
                    │                            │ ws://      └─────────────────────────┘
                    ▼                            ▼
            Local filesystem             Remote Git repos
            & Git repositories           (shallow clone)

Two Operating Modes

The same c4a-daemon binary runs in two modes, determined by startup arguments:

| | User Daemon | Cloud Daemon | |---|-------------|--------------| | Launched by | User (manual) | Scheduler (automatic) | | machine_id | os.hostname() (e.g., MacBook-Pro) | cloud_<lib_id> (e.g., cloud_lib_abc123) | | Filesystem | User's local disk | Scheduler workspace directory | | Lifecycle | Long-running, user manages | On-demand spawn, idle eviction | | --work-dir | Optional | Required (set by Scheduler) | | Duplicate connect | Server rejects new connection | Server replaces old mapping |

Both modes use the same WebSocket + JSON-RPC protocol. The server detects cloud daemons by the cloud_ prefix and routes accordingly.

Communication Flow

1. Daemon starts and opens a WebSocket connection to the server (ws://<host>/ws/daemon).
2. Handshake: daemon sends daemon/handshake with machine info; server responds with accepted: true and optional lib_id binding.
3. Heartbeat: daemon sends periodic daemon/heartbeat notifications. Server marks the daemon offline if heartbeats stop arriving within the threshold.
4. RPC: server sends JSON-RPC requests (e.g., fs/repoScan) to the daemon; daemon executes locally and returns results.
5. Progress: during long-running operations (e.g., vcs/clone), the daemon sends task/progress notifications with percent updates back to the server.
6. Reconnect: if the WebSocket drops, the client automatically reconnects with exponential backoff (1s to 30s, up to 64 attempts).

Dead Process Prevention

| Mechanism | Component | Description | |-----------|-----------|-------------| | Heartbeat timeout | Server (DaemonManager) | Marks daemon offline when heartbeats exceed DAEMON_OFFLINE_THRESHOLD. Broadcasts daemon/status event. | | Auto-reconnect | Daemon (WebSocketClient) | Exponential backoff reconnect (1s~30s), stops after maxReconnectAttempts (default 64). | | PID file tracking | Scheduler (DaemonPool) | Writes daemon.pid on spawn; on restart, kills orphan PIDs (SIGTERM, wait 2s, SIGKILL). | | Idle sweep | Scheduler (DaemonPool) | Periodically kills daemons that haven't been used within CLOUD_DAEMON_IDLE_TIMEOUT. LRU eviction when pool is full. | | Workspace directory reuse | Scheduler (DaemonPool) | When a daemon is evicted (idle or LRU), its work directory is preserved. Next spawn reuses existing clone data, avoiding redundant downloads. Directory is only deleted on explicit DELETE /daemons/:lib_id. | | Graceful shutdown | Both | SIGINT/SIGTERM handlers: daemon closes WebSocket; scheduler kills all child processes and waits for exit. | | Duplicate rejection | Server (DaemonManager) | User daemons: rejects handshake if same machine_id is already connected. Cloud daemons: silently replaces the old mapping. |

Installation

npm install @c4a/daemon
# or
bun add @c4a/daemon

Usage

Running a Daemon

# Connect to a local server
c4a-daemon --server 127.0.0.1:5100

# Connect to a remote server (auto-detects wss://)
c4a-daemon --server example.com

# Restrict filesystem access to specific paths
c4a-daemon --server 127.0.0.1:5100 --allow /home/user/projects,/opt/repos

# Override machine identity
c4a-daemon --server 127.0.0.1:5100 --machine-id my-build-agent

# Set a working directory for VCS clone operations
c4a-daemon --server 127.0.0.1:5100 --work-dir /var/lib/c4a-daemon

Git Credential Persistence

When --work-dir is set, the config/setAuth RPC writes Git credentials to the work directory using Git's native credential store:

<work-dir>/
├── .git-daemon.config        # [credential] helper = store --file ...
├── .git-daemon.credentials   # https://oauth2:[email protected] (chmod 600)
└── repos/
    └── my-repo/              # cloned repositories

All git commands run with GIT_CONFIG_GLOBAL pointing to .git-daemon.config, so authentication is automatic. Credentials persist across daemon restarts — no need for the server to re-inject auth after a respawn.

Incremental Clone

vcs/clone detects existing repositories and avoids redundant downloads:

• Same remote URL: git fetch --depth=1 + checkout (incremental update)
• Different remote URL: removes old repo, clones fresh
• URL comparison strips auth credentials before matching (https://token@host/repo and https://host/repo are treated as the same remote)

Running the Scheduler

# Start with defaults
c4a-daemon-scheduler --server 127.0.0.1:5100

# Custom port, workspace, and daemon limit
c4a-daemon-scheduler \
  --server 127.0.0.1:5100 \
  --port 5110 \
  --workspace /var/lib/daemon-workspace \
  --max-daemons 20

Programmatic Usage

import { startScheduler, createSchedulerApp, DaemonPool } from "@c4a/daemon/scheduler";

// Option 1: Start the full HTTP scheduler server
await startScheduler({
  serverUrl: "http://127.0.0.1:5100",
  port: 5110,
  workspace: "./data/daemon-workspace",
  maxDaemons: 20,
});

// Option 2: Use the pool directly (e.g., embed in your own server)
const pool = new DaemonPool({
  serverUrl: "http://127.0.0.1:5100",
  workspace: "./data/daemon-workspace",
  maxDaemons: 10,
});
await pool.initializeFromWorkspace();
pool.startIdleSweep();

const entry = await pool.ensureDaemon("my-library-id");
console.log(entry.machine_id, entry.status); // "cloud_my-library-id" "ready"

// Cleanup
await pool.destroyAll();

Module Overview

| Module | Export | Description | |--------|--------|-------------| | @c4a/daemon | CLI entry (c4a-daemon) | Daemon process — WebSocket client + RPC handler + built-in handlers | | @c4a/daemon/scheduler | CLI entry (c4a-daemon-scheduler) | Scheduler process — HTTP API + DaemonPool | | @c4a/daemon/scheduler | DaemonPool | Process pool: spawn, evict, orphan cleanup | | @c4a/daemon/scheduler | createSchedulerApp | Returns { handler, pool } for embedding | | @c4a/daemon/scheduler | startScheduler | Starts the HTTP server with graceful shutdown |

Internal Modules (not exported, for reference)

| File | Responsibility | |------|----------------| | wsClient.ts | WebSocket connection with auto-reconnect and pending-request tracking | | rpcHandler.ts | JSON-RPC 2.0 request dispatcher with error handling | | handshake.ts | Builds handshake payload (OS, version, allowed paths) | | config.ts | Server URL resolution and path parsing | | errors.ts | JSON-RPC error codes and RpcError class | | handlers/ | 15 built-in RPC method handlers (see REFERENCE.md) | | scheduler/daemonPool.ts | Process lifecycle, PID tracking, idle sweep | | scheduler/health.ts | Disk usage reporting |

API Reference

Complete RPC method signatures, Scheduler HTTP endpoints, and type definitions are documented in REFERENCE.md.

License

MIT