ZTC — Zerg Terminal Client

An autonomous AI agent for the terminal. ZTC combines Claude-style conversational coding with native computer control, multi-agent orchestration, and a Voyager-inspired routine system for autonomous operation in graphical applications.

  ╭─ ZTC v0.1.57 ─────────────────────────────────────────────────────╮
  │ Provider: Anthropic  Model: claude-sonnet-4-20250514             │
  │ Working directory: ~/projects/my-app  Branch: main                │
  ╰───────────────────────────────────────────────────────────────────╯

❯ Build a house in Luanti using the saved routines

◆ I'll use the Voyager routine system to build the house.
  ⚙ computer_list_routines [0.1s]
    → 10 saved routines
  ⚙ computer_run_routine [216s]
    → build_house completed: 7x5x3 cobblestone house

  House built successfully — cobblestone walls, wood roof, door and window.

● Ready                                          [████████░░] 42%

Quick Start

# Install dependencies and build
npm install && npm run build

# Build native Rust binaries (screen capture, input injection)
npm run build:rust

# Run in development mode
npm run dev

# Or run the built version
npm start

Requirements

• Node.js >= 18
• Rust toolchain (for native binaries — cargo)
• Anthropic API key (set via ANTHROPIC_API_KEY env var or /config key)
• macOS, Linux, or Windows

Features

Terminal AI Agent

• Agentic tool use — file I/O, shell commands, code search, LSP integration
• Streaming responses with real-time tool execution traces
• Context management — automatic compaction at configurable thresholds, manual /compact
• Session persistence — auto-save, resume with --resume, per-project session storage
• Sub-agent delegation — spawn isolated agents for context-heavy research
• Plan mode — read-only codebase exploration, then structured implementation

Computer Use

• Native input injection — keyboard, mouse, and relative mouse movement at the hardware level
• Screen capture — native PNG screenshots via the ztc-computer Rust binary
• Routine system — save, version, and replay JavaScript routines in a sandboxed subprocess
• Application harnesses — app-specific configs, reference docs, and automated setup (Luanti built-in)
• Voyager loop — autonomous multi-round task completion with auto-continuation

Multi-Agent Orchestration (zmux)

• Terminal multiplexer — fullscreen split-pane environment for parallel agent instances
• Swarm mode — conductor agent delegates tasks across child ZTC panes
• RPG mode — generative agent simulation with 2D world visualization
• Message bus — inter-agent communication across panes

Voice & Media

• Voice dictation — native Whisper transcription (Ctrl+R)
• TTS synthesis — multiple backends (macOS say, Parler, OpenAI, ElevenLabs)
• Image support — paste from clipboard (Ctrl+I), screenshot compression

Computer Use

ZTC can autonomously control graphical applications through native keyboard/mouse injection and screen capture. The system follows a Voyager-style approach: observe the screen, plan an action, execute it, verify the result, and save successful routines for reuse.

Enabling Computer Use

# Interactive mode with computer control
npm run dev -- --computer-use

# Headless mode with Voyager loop
node dist/cli_main.js --headless --computer-use \
  --voyager-loop --voyager-task "Build a house in Luanti" \
  --voyager-max-rounds 15 --timeout 1800000

Computer Use Tools

| Tool | Description | |------|-------------| | computer_screenshot | Native PNG screenshot of the entire screen | | computer_key | Inject keyboard press/release events | | computer_type_text | Type a string with keystroke simulation | | computer_mouse_move | Move mouse to absolute coordinates | | computer_mouse_move_relative | Relative mouse delta (works with FPS games) | | computer_mouse_click | Click at current or specified position |

Routine System

Routines are JavaScript scripts executed in a sandboxed subprocess with access to native input APIs. They run at native speed — no LLM round-trip per keystroke.

# The agent can save and reuse routines:
computer_save_routine    # Save a verified routine
computer_list_routines   # List saved routines
computer_run_routine     # Execute a routine (with optional timeout)
computer_delete_routine  # Remove a routine

Routine sandbox globals:

// Mouse control
mouse.move(x, y)              // Absolute move
mouse.move_relative(dx, dy)   // Relative move (camera rotation)
mouse.click("left"|"right")   // Click
mouse.click("left", x, y)     // Click at position
mouse.down("left") / mouse.up("left")

// Keyboard control
key.press("W")                // Press and release
key.down("W") / key.up("W")  // Hold/release
key.type("hello", 20)         // Type string (20ms between keys)
key.combo("Ctrl", "C")        // Key combination

// Screen
screen.dimensions()           // { width, height }
screen.shot()                 // Take screenshot (returns base64 PNG)

// Utilities
sleep(ms)                     // Wait
log("message")                // Log to routine output
runRoutine("name", args)      // Call another saved routine

Routines are stored in ~/.ztc/computer_use/routines/ and persist across sessions.

Application Harnesses

Harnesses provide app-specific configuration, reference documentation, and setup automation. They're loaded with computer_load_harness.

Built-in harness: Luanti (Minetest)

• Flat world with frozen noon, creative mode, all privileges
• Comprehensive reference doc with controls, building strategies, item aliases
• Auto-setup: writes minetest.conf, creates world, configures fullscreen

# Create custom harnesses in:
~/.ztc/computer_use/harnesses/<name>/
  config.json    # Launch config, window size, env vars
  harness.md     # Reference doc injected into agent context
  setup.ts       # Optional setup automation

Voyager Loop Mode

For long-running autonomous tasks, the Voyager loop auto-continues across multiple agent rounds. Between rounds, the agent is prompted to screenshot current state, review saved routines, and proceed — naturally recovering from context compaction.

node dist/cli_main.js --headless --computer-use \
  --voyager-loop \
  --voyager-task "Build a cobblestone house with a roof" \
  --voyager-max-rounds 15 \
  --timeout 1800000

| Flag | Description | Default | |------|-------------|---------| | --voyager-loop | Enable auto-continuation loop | — | | --voyager-task <text> | Task description (alternative to --prompt) | — | | --voyager-max-rounds <n> | Maximum continuation rounds | 20 | | --timeout <ms> | Wall-clock timeout | 1800000 (30 min) |

The loop exits when the agent says "Task completed" (detected via regex patterns) or the round/time budget is exhausted.

Native Input Injection (macOS)

ZTC uses hardware-level input injection for compatibility with games and fullscreen apps:

• Keyboard: CGEvent injection — works with all applications including SDL/game engines
• Mouse clicks: CGEvent with embedded position — avoids async cursor race conditions
• Relative mouse: IOHIDPostEvent (IOKit layer) — works with SDL native relative mouse mode, indistinguishable from real hardware input

zmux — Terminal Multiplexer

zmux provides a fullscreen split-pane environment for running multiple terminal processes and ZTC agent instances simultaneously.

# Launch zmux
npm run dev -- --mux

# With auto-created agent panes (3 columns)
npm run dev -- --mux --num-zergs 3

# Grid layout (2 columns x 3 rows)
npm run dev -- --mux --num-zergs 2x3

Pane Tools

When running in zmux, the agent gains access to terminal pane management:

| Tool | Description | |------|-------------| | create_terminal_pane | Spawn a process in a new pane (shell, server, child ZTC) | | read_terminal_pane | Read current terminal buffer contents | | write_terminal_pane | Send keystrokes to a pane | | close_terminal_pane | Close a pane and kill its process | | list_terminal_panes | List all panes with status |

Swarm Mode

Conductor mode for orchestrating multiple agents in parallel:

/swarm    — Enter conductor mode
/swarm-status  — Check agent progress
/swarm-stop    — Exit conductor mode

The conductor agent delegates tasks to child ZTC panes, monitors progress, and coordinates results.

RPG Mode

Generative agent simulation inspired by "Generative Agents" (Park et al., 2023):

/rpg         — Start simulation (creates character agents)
/rpg-event   — Inject a world event
/rpg-stop    — End simulation

Features a 2D world visualization at http://localhost:3941/ with character positions, locations, and real-time state updates.

Mirror Mode

Browser-based UI that mirrors the terminal session:

/mirror       — Start mirror at http://localhost:3940
--mirror      — Enable on startup
--mirror-port — Custom port

Modes

Modes modify ZTC's behavior by restricting tools, injecting system prompts, and producing artifacts.

| Mode | Command | Description | |------|---------|-------------| | Normal | (default) | Full tool access with approval prompts | | Auto-accept | Shift+Tab | Skip tool approval — fast execution | | Plan | /plan | Read-only exploration, structured planning | | Swarm | /swarm | Multi-agent conductor (zmux) | | RPG | /rpg | Generative agent simulation (zmux) | | Manifest | /manifest | YAML manifest creation with validation | | Strict | — | Minimal output, exact constraint following | | Zerg | — | Cloud agent delegation via Zerg API |

Custom Modes

Create YAML mode definitions in ~/.ztc/modes/:

name: my_mode
version: "1.0"
label: My Mode
description: Custom operational mode
auto_accept: false
tools:
  blocked: [run_command]    # Tools to disable
system_prompt: |
  Additional instructions for the agent...
artifact:
  enabled: true
  type: markdown
  path_template: "./output-{timestamp}.md"
commands:
  - name: my_cmd
    description: Custom command
on_enter_questions:
  - name: target
    question: "What should we work on?"
    type: text

Headless Mode

Run ZTC without a TUI for automation, CI, and scripting:

# Single prompt, exit when done
node dist/cli_main.js --headless --prompt "Refactor auth module"

# JSON protocol (structured I/O)
node dist/cli_main.js --headless --json

# With timestamps and file output
node dist/cli_main.js --headless --prompt "Fix the tests" \
  --timestamps --output result.txt --timeout 300000

| Flag | Description | Default | |------|-------------|---------| | --headless | Enable headless mode | — | | --prompt <text> | Single prompt to execute | — | | --json | JSON protocol for I/O | — | | --max-iterations <n> | Max agent iterations | 50 | | --timeout <ms> | Operation timeout | 300000 | | --output <file> | Save response to file | — | | --timestamps | Show timestamps in output | — | | --quiet / -q | Suppress build output | — | | --session-id <id> | Use specific session ID | — |

CLI Flags Reference

General

--help, -h              Show help
--version, -v           Show version
--cwd <path>            Start in directory

Sessions

--resume [id], -c       Resume session (latest or by ID)
--list-sessions         List saved sessions

Modes & Features

--plan                  Start in plan mode
--mode <path>           Load mode from YAML file
--mux                   Launch zmux multiplexer
--pane                  Pane mode (for zmux children)
--computer-use          Enable computer control tools
--ml                    Enable ML inference/training tools
--mirror                Enable browser mirror
--mirror-port <n>       Mirror port (default: 3940)

zmux Options

--new-window            Open new terminal window
--fullscreen            Maximize terminal window
--num-zergs <n|CxR>     Auto-create agent panes

Commands

Configuration

| Command | Description | |---------|-------------| | /config show | View configuration | | /config key <key> | Set API key | | /config model <name> | Set model | | /model <name> | Alias for config model | | /models | List available models | | /permissions | View/update tool permissions |

Session & Context

| Command | Description | |---------|-------------| | /clear | Clear message history | | /sessions | List/manage saved sessions | | /status | Show current status | | /context | Detailed context window usage | | /compact [%] | Compact conversation (default: 50%) | | /retry | Retry last request |

Modes

| Command | Description | |---------|-------------| | /mode [name] | Show or enter a mode | | /modes | List available modes | | /exit-mode | Return to normal | | /plan [desc] | Enter plan mode | | /execute | Execute plan |

Shell & Files

| Command | Description | |---------|-------------| | /shell <cmd> or !<cmd> | Run shell command | | /cd <path> | Change directory | | /clipboard | Copy conversation |

Voice

| Command | Description | |---------|-------------| | /dictation | Manage dictation (devices, model) | | /voice | Manage TTS synthesis |

zmux & Orchestration

| Command | Description | |---------|-------------| | /swarm | Enter conductor mode | | /swarm-status | Check agent progress | | /swarm-stop | Exit conductor mode | | /rpg | Start RPG simulation | | /rpg-event [desc] | Inject world event | | /rpg-stop | Stop simulation | | /mirror [port] | Start browser mirror |

Meta

| Command | Description | |---------|-------------| | /help | Show commands | | /update | Update ZTC | | /debug | Toggle debug mode | | /keybindings | Show keyboard shortcuts | | /emulation | Prompt emulation profiles | | /exit | Exit ZTC |

Keyboard Shortcuts

| Shortcut | Action | |----------|--------| | Enter | Submit message | | Shift+Enter | Insert newline | | Ctrl+C | Exit | | Ctrl+L | Clear screen | | Ctrl+R | Toggle voice dictation | | Ctrl+I | Paste image from clipboard | | Ctrl+O | Expand/collapse last tool output | | Ctrl+Shift+O | Toggle all tool outputs | | Ctrl+G | Open plan file in editor | | Ctrl+T | Toggle task list | | Shift+Tab | Cycle modes | | Tab | Path completion | | Up/Down | Command history | | Ctrl+A / Ctrl+E | Start / end of line | | Ctrl+U | Clear input line | | Ctrl+W | Delete word before cursor | | Ctrl+K | Kill to end of line |

Agent Tools

File System

• read_file — Read file contents
• write_file — Write/modify files (with diff display)
• list_directory — List directory contents
• search — Regex code search (ripgrep-style)

Shell & Execution

• run_command — Execute shell commands with timeout
• run_and_monitor — Long-running command monitoring

Development (LSP)

• goto_definition — Jump to symbol definition
• find_references — Find all references to a symbol
• hover — Get type info and documentation
• diagnostics — Compiler errors and warnings
• lsp_status — Check language server status

Supported: Rust (rust-analyzer), TypeScript, Python (pyright), C/C++ (clangd), Go (gopls)

Agent Orchestration

• delegate_task — Spawn sub-agent for isolated research
• plan_mode / submit_plan / ask_user — Planning workflow tools
• todo_write — Task list management
• ztc_command — Execute slash commands programmatically

ML Tools (--ml)

• ml_load_model — Load model from safetensors weights
• ml_infer — Run inference on images
• ml_train — Train on session data
• ml_status — GPU device and model status

Architecture

Rendering

ZTC uses a Rust/Ratatui terminal renderer communicating with the TypeScript agent via Unix domain socket IPC. The renderer operates in inline viewport mode — content scrolls above while the input area stays fixed at the bottom.

Agent Loop

User Input → Agent → Claude API → Tool Use? → Execute → Continue
                ↑                      │            │
                └──────────────────────┴────────────┘

Standard ReAct loop with streaming, automatic retry (429/529/5xx), and configurable max iterations.

Native Binaries (Rust)

• ztc-computer — Screen capture (CoreGraphics/X11/Win32), hardware-level input injection (IOHIDPostEvent/CGEvent), process management
• ztc-ml — GPU-accelerated ML inference and training (candle framework)

Built with npm run build:rust — cross-compiled per platform.

Project Structure

ztc/
├── src/
│   ├── cli_ratatui.ts         # Main CLI entry point
│   ├── headless.ts            # Headless/automation mode
│   ├── agent/
│   │   ├── factory.ts         # Agent creation (model, tools, capabilities)
│   │   ├── runtime/           # Agent execution engine
│   │   ├── tools/             # Tool definitions (file, shell, computer, ml, ...)
│   │   └── commands/          # Slash command handlers
│   ├── modes/
│   │   ├── types.ts           # Mode definition types
│   │   ├── registry.ts        # Mode discovery and loading
│   │   └── builtin/           # Built-in mode YAML definitions
│   ├── computer_use/
│   │   ├── routine_runner.ts  # Sandboxed routine subprocess executor
│   │   └── harnesses/         # Application-specific harness configs
│   │       └── luanti/        # Luanti (Minetest) harness
│   ├── utils/
│   │   ├── computer_native.ts # ztc-computer binary bridge
│   │   ├── ml_native.ts       # ztc-ml binary bridge
│   │   ├── compaction.ts      # Context compaction
│   │   └── shell.ts           # Shell utilities
│   └── web/                   # Mirror mode web UI
├── packages/
│   ├── ztc-computer/          # Rust: screen capture + input injection
│   └── ztc-ml/                # Rust: ML inference + training
├── tbench/                    # Terminal-Bench evaluation harness
└── scripts/
    └── build-rust.sh          # Native binary build script

Data Directories

~/.ztc/
├── config.json                # User configuration
├── sessions/                  # Saved conversation sessions
├── plans/                     # Plan mode artifacts
├── modes/                     # User custom modes
├── models/                    # Downloaded ML/TTS models
├── logs/                      # Session logs
└── computer_use/
    ├── routines/              # Saved Voyager routines
    └── harnesses/             # User application harnesses

Development

npm run build          # Compile TypeScript + copy assets
npm run build:rust     # Build native Rust binaries
npm run dev            # Development mode (tsx + rust build)
npm run dev:ink        # Legacy Ink renderer
npm run web            # Web UI at http://localhost:3000
npm start              # Production entry point

Environment Variables

| Variable | Description | |----------|-------------| | ANTHROPIC_API_KEY | Anthropic API key | | ZTC_DEBUG | Enable debug logging | | ZTC_SUBAGENT_DEBUG | Log sub-agent execution | | ZTC_DISABLE_AUTO_COMPACT | Disable automatic compaction | | ZTC_DICTATION_FILTER | Disable filler word filter (set to 0) | | ZTC_DOCKER_CONTAINER | Route file tools through Docker | | ZTC_WEB_MIRROR | Enable web mirror mode | | SDL_MOUSE_RELATIVE_MODE_WARP | Force SDL warp mode (computer use fallback) |

License

MIT

Part of the Zerg autonomous AI framework.