helixcoder

The command-line surface for the Helix engine: the primary dogfooding driver and the eval-harness runner. It opens a session, submits a prompt, renders the SSE stream (a live $/task cost meter plus tool, verify, retrieval, and permission-gate transparency), and answers human-in-the-loop gates. It speaks the frozen @helix/protocol contract to @helix/engine over the same RPC + SSE path any client uses, so the CLI path is the production path.

Build

pnpm -C helix build      # tsc -b across the workspace
pnpm -C helix test       # vitest

The executable is helix (the bin of this package); from a build you can also run node packages/cli/dist/bin.js.

helix run: run the agent on a task

helix run [options] <prompt>

Without --workspace it runs an offline demo host (a scripted run, no API key, no repo) so the transparency surface is visible end to end. With --workspace it operates on a real repository.

| Option | Default | Meaning | |---|---|---| | --workspace <dir> | (demo) | Operate on a real repo. Requires the selected provider's API key (BYOK); aborts otherwise. | | --provider <name> | anthropic | Model vendor (ADR-005): anthropic (ANTHROPIC_API_KEY, default claude-opus-4-8), google (GEMINI_API_KEY, default gemini-2.5-flash), or openai (OPENAI_API_KEY, default gpt-4o-mini). Pair with --model for a specific model. | | --base-url <url> | (openai.com) | OpenAI-compatible base URL (--provider openai only; else OPENAI_BASE_URL). One adapter, many hosts: point it at Groq (https://api.groq.com/openai/v1), OpenRouter, Cerebras, or a local Ollama (http://localhost:11434/v1) for FREE strong models. | | --test <cmd> | pnpm test | The verify command, run with shell:false (split on whitespace). It is the loop's real green signal. | | --model <id> | claude-opus-4-8 | Model id. | | --max-iters <n> | 4 | Iteration budget, an integer 1-6. | | --allow-exec | off | Offer the run_command tool (run tests/builds/linters in the workspace). Opt-in because arbitrary execution is the most dangerous capability. Run shell-free (no pipes/redirects/globs), policy-gated (exec → ask), with the destructive-command floor still denying (rm -rf, force-push, ...). WARNING: the floor is a name/flag denylist, not a sandbox. With eval --autonomy, exec gates auto-approve, so --allow-exec --autonomy lets the agent run arbitrary non-floored commands unattended — including network egress (curl/wget, which can exfiltrate repo contents) and non-rm mutation (mv, find -delete, interpreter one-liners). Use it only on repos/inputs you trust; container isolation is the planned backstop for untrusted runs (ADR-004). Because of this, eval --workspace --autonomy --allow-exec refuses to start (exit 2) unless you set HELIX_SANDBOX=1 to assert a disposable/isolated environment (we do not auto-detect the sandbox: only the operator knows). A real env var is preferred; HELIX_SANDBOX can also be satisfied by a <cwd>/.env, in which case Helix warns (a committed .env should not silently re-enable unattended exec). |

Example:

ANTHROPIC_API_KEY=sk-... helix run --workspace . --test "pnpm test" "add a null check to the parser"

Providing the API key

The BYOK key can come from the environment directly, or from a .env file in the directory you run helix from. A .env file is read at startup; a real environment variable always wins over the file (so an explicit export overrides .env). The file is gitignored. Copy .env.example to .env and fill in your key:

# .env  (in your working directory)
ANTHROPIC_API_KEY=sk-...
GEMINI_API_KEY=...

Then just: helix run --workspace . --provider google "...".

FREE strong models via the OpenAI-compatible provider. Groq, OpenRouter, Cerebras, and local Ollama all speak the OpenAI Chat Completions API, so one adapter reaches them via a base-URL override. Example (Groq free tier):

OPENAI_API_KEY=<groq key> OPENAI_BASE_URL=https://api.groq.com/openai/v1 \
  helix run --workspace . --provider openai --model llama-3.3-70b-versatile "..."

What you see per run: run.started, each tool.requested/tool.completed (with its side-effect class and redacted args), the verify result per iteration, the live cost meter, any permission gate, and a final result block (status, summary, cost, retrieved files, open risks).

The $/task meter uses a dated default price table (prices.ts, captured 2026-06-20, USD per 1M tokens for the known Anthropic and Gemini models). Prices are config, not fact: treat them as a snapshot to override, not ground truth. A model with no entry prices to $0.00 rather than failing the run.

Permission gates

When the policy classifies a tool call as ask (writes ask by default; reads allow; destructive commands deny), the run pauses for you:

  ⛔ permission: write_file (proposed deny, timeout 0ms) {path=src/x.ts}
     allow write_file? [y]es / [a]lways (session) / [N]o (default deny):

• y / yes: allow this one call.
• a / always: allow this action (same tool + args) for the rest of the session, no re-prompt.
• anything else (including empty / Ctrl-D): deny. The default is fail-safe: only an explicit yes or always permits.

A non-interactive invocation (piped stdin, CI) has no responder, so every gate auto-applies the terminal-safe deny and the run never hangs.

helix eval: run a suite and report the metric tuple

helix eval [--suite <file>] [--min-pass <0..1>]

Drives the engine headlessly over a task suite and prints the gate tuple: pass@1 with its Wilson 95% CI, mean $/task, latency (p50/p95), edits-to-green, and context precision/recall. Without --suite it runs a built-in demo suite.

| Option | Meaning | |---|---| | --suite <file> | A JSON suite file (see below). Omit for the built-in demo suite. | | --min-pass <0..1> | CI gate: exit 1 if pass@1 is below the threshold, else 0. Without it, eval is report-only (always exit 0). | | --json | Emit the report ({summary, tasks}) as a single JSON line to stdout instead of the human render. The exit code still reflects --min-pass; the human gate line is suppressed so stdout stays pure JSON for CI to parse. | | --workspace <dir> | Run the suite against a real repo (each task's prompt drives the real runner: model + filesystem tools + verify command). Requires ANTHROPIC_API_KEY. Omit for the offline demo host. | | --autonomy | Auto-approve ask gates for unattended eval (there is no human to answer). The destructive-command floor and role allowlist still deny (a remembered/auto allow can never lift a floor deny). Without it, write/exec gates auto-deny and those tasks fail. | | --isolate | Reset the workspace to its git HEAD baseline (git reset --hard && git clean -fd) before each task, so tasks are independent (task N does not inherit N-1's edits). Requires a git --workspace; destructive (discards uncommitted changes), so point it at a throwaway checkout. | | --max-tasks <n> | Run at most the first n tasks of the suite (CI cost control for large suites). Reports how many were skipped (the summary's task count reflects the cap). | | --samples <n> | Run each task n times (1-20) and report pass@k for k in {3,5} (k <= n) via the unbiased estimator (benchmarks.md 1.1) — the run-to-run variance signal. pass@1 stays the first-attempt rate (the gated metric, with a CI); pass@k is a diagnostic and ships without a CI, so don't gate on it. Default 1 (single-shot); multiplies cost by n. |

Workspace eval (--workspace)

Each task runs the real agent against <dir> end to end, so this is how you measure pass@1, $/task, and latency on actual repositories.

Two things to know:

• Unattended: eval has no interactive responder, so pass --autonomy for tasks that need writes/exec. It auto-approves ask gates but never lifts a floor deny (so rm -rf, force-push, out-of-allowlist tools are still blocked). Run it against a throwaway checkout, not a repo with work you care about.
• Isolation: tasks run sequentially against the same working tree, so task N sees task N-1's edits unless you pass --isolate, which git reset --hard && git clean -fds the workspace to its HEAD baseline before each task (requires a git workspace; destructive, so use a throwaway checkout).

ANTHROPIC_API_KEY=sk-... helix eval --suite evals/example.suite.json --workspace /tmp/repo-checkout --autonomy --min-pass 0.7

Suite file format

{
  "tasks": [
    { "id": "add-null-check", "prompt": "add a null check to the parser", "grade": { "type": "status_ok" } },
    { "id": "rename-foo",     "prompt": "rename foo to bar",             "grade": { "type": "verify_passed" }, "gold": { "files": ["src/parser.ts"] } }
  ]
}

Each task has an id, a prompt, a declarative grade spec, and an optional gold context set (for context precision/recall). Context precision/recall compares the files the agent actually read (read_file calls, recorded as the run's retrieval evidence, the ADR-002 agentic-search floor) against the task's gold.files. A task with no gold is not measured. Graders are deterministic (no LLM judge for pass@1):

| grade.type | Passes when | |---|---| | status_ok | the run's result status is ok | | status_is (value) | the result status equals value | | summary_contains (value) | the result summary contains value | | verify_passed | any verify.result event reported a pass |

A malformed suite fails loudly (a clear error, exit 2) rather than silently skipping tasks. Suite files are parsed with prototype-pollution-safe JSON handling.

A copy-paste starter suite lives at evals/example.suite.json.

CI example:

helix eval --suite evals/example.suite.json --min-pass 0.9            # non-zero exit fails the build
helix eval --suite evals/example.suite.json --json > eval-report.json  # machine-readable, for trend tracking

Exit codes

0 success (a run reached ok, or an eval met its --min-pass / was report-only) · 1 the run did not reach ok, or the eval gate failed · 2 bad usage (unknown command, malformed flag, missing API key, unloadable suite).

Design notes

• The tested core is render.ts (pure event -> terminal text, control-byte sanitized) and driver.ts (drives EngineHost over RPC + SSE). main() is a thin argv shell with injectable host and I/O, so it is testable without touching the real process.
• The real runner wires AnthropicProvider + WorkspaceTools (path-contained read/list/search/write/edit) + ExecutionPolicy (allow/ask/deny) + secret redaction + CommandVerifier behind the frozen contract. See runner.ts.