Summary

pi-coding-agent-test runs tests against real Pi instead of mocking it. Pi loads your extensions, runs real tools and hooks, creates sessions, changes files, and renders terminal output.

The test controls the initial prompt and scripted assistant responses. It does not fake tool execution or extension behavior. Each run gets a fresh session and workspace, records artifacts, and can be replayed without starting Pi.

Executable behavior contracts live in ./test/; demo scripts live in ./scripts/.

Scenario settings

PiIntegrationTest supports explicit skills, systemPrompt, and appendSystemPrompt settings. Skills are loaded from the listed files or directories even though ambient skill discovery is disabled for isolated runs. Use getSystemPrompt(run) to inspect the final effective prompt after extension before_agent_start handlers have run.

Raw debugging interface

Raw mode is a debugging view for inspecting runs, not just a test assertion format. It shows complete tool arguments and call IDs, followed by the text result and the full result envelope, including internal details, error state and progressive updates. Long payloads are wrapped, not shortened to a preview.

Scripted TUI runs use raw mode by default. Set rawMode: true explicitly when needed, or rawMode: false to inspect Pi's normal tool panels. Raw shows Pi's parsed tool arguments; it is not a byte-for-byte capture of an HTTP request.

The Pi 0.99.1 runtime checks cover Ubuntu and native Windows in CI, including installed-package launch and PTY cleanup. On Pi 0.99.1, a process-local Node loading hook installs the renderer on the class used by the bundled CLI. It does not edit the Pi installation and does not run in native mode. The adapter depends on Pi's internal renderer signatures and rejects changed or ambiguous matches instead of silently switching to native rendering.

Headless scenarios

Set transport: "rpc" in PiIntegrationTest options to run Pi through JSONL pipes instead of a PTY. The default is "tui". RPC always starts a fresh process, including when the shared test runner is active. Use RPC for tool and evaluation scenarios; keep TUI for rendering and interactive UI tests.

RPC keeps the harness trace, effective system prompt, tool results, messages, and session in run.jsonl. terminalOutput contains raw RPC stdout; tuiRenderedOutput and frameDelaysMs are empty. tuiSize remains a placeholder for the shared result shape, not an actual terminal. The artifact directory also contains rpc-stdout.jsonl and rpc-stderr.log, including on failure. Raw rendering and live TUI pacing do not apply. Explicit scripted delays still apply. Dialog requests fail with an error instead of hanging or inventing a user answer.

See the executable RPC example.

Installation

Install the test package:

npm install --save-dev pi-coding-agent-test

The package provides the TypeScript API and the pi-test CLI. Use it with Vitest, Jest, Mocha, node:test, Playwright Test, or another Node.js test runner.

Install Chromium for browser fixtures

The package pins the Playwright runtime, but installing the package does not download Chromium. Provision the matching browser before using a browser fixture:

npx playwright install chromium

On a Linux machine or container that also needs Chromium's system libraries, run:

npx playwright install --with-deps chromium

From this package's source checkout, the same commands are available as npm run browser:install and npm run browser:install:with-deps.

If you set PLAYWRIGHT_BROWSERS_PATH, use the same value when installing Chromium and when running tests. Behind a proxy, set Playwright's supported HTTPS_PROXY environment variable for the install command. Playwright also supports NODE_EXTRA_CA_CERTS for a custom certificate authority. Do not put credentials in scripts or committed configuration.

In CI and Docker, provision Chromium during job or image setup. Cache the configured PLAYWRIGHT_BROWSERS_PATH when useful, and restore the cache before tests run. Do not rely on an npm postinstall hook to download the browser.

Prerequisites

• Node.js 22.19 or newer;
• Pi installed with the pi executable available on PATH.

Dedicated browser fixture

Import createBrowserFixture or withBrowserFixture from the package root. The callback helper always closes the fixture, including when the callback throws:

import { spawn } from "node:child_process";

import { withBrowserFixture } from "pi-coding-agent-test";

await withBrowserFixture(async (fixture) =>
{
    const application = spawn("my-test-application", [], {
        env: fixture.childEnvironment({
            ...process.env,
            BROWSER: fixture.openerCommand,
        }),
        stdio: "inherit",
    });

    try
    {
        const url = await fixture.waitForUrl();
        await fixture.page.goto(url);
        await fixture.page.getByRole("button", { name: "Confirm" }).click();
    }
    finally
    {
        application.kill("SIGTERM");
    }
});

Configure the application under test to call fixture.openerCommand with one HTTP or HTTPS URL. childEnvironment() adds the fixture's isolated opener state without changing process.env. Keep the application or decision server running while the page is in use, and stop it normally yourself; the fixture does not own it.

Each fixture owns its Chromium process, isolated browser context, and page. It never opens the system or default browser. Headless mode is the default. Set { headless: false } explicitly for a visible browser; headed Linux runs need a display such as Xvfb.

close() is bounded, safe to call more than once, and reports cleanup errors. withBrowserFixture() keeps a callback error visible if cleanup also fails. If matching Chromium is missing, fixture creation fails with the recovery command npx playwright install chromium.

Use createBrowserFixture() instead when its lifetime cannot fit inside one callback. Always call await fixture.close() in a finally block.

Install optional Pi skills

pi install npm:pi-coding-agent-test

Runnable integration tests

From a checkout of this package, run the demonstration integration tests with real Pi:

npm run test:examples

For the interactive terminal demo, watch a real Pi TUI live until you stop it with Ctrl+C:

npm run demo:live

To replay the checked-in recording offline:

npm run demo:replay

| Test | Covers | | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | evaluation-smoke | Isolated workspaces, built-in tools, filesystem effects, and artifacts | | extensions | Custom extensions, tool selection, and runtime registration | | raw-tooling-renderer | Stable raw tool-call and tool-result rendering | | streaming | Streamed text, partial tool-call JSON, and real tool execution | | native-tui | Pi's native renderer and visible terminal assertions | | replay | All built-in tools, large TypeScript edits, and offline replay |

Development and releases

Development and releases use this public repository. Open changes against develop; release candidates go to main. There is no private publisher or local-registry step.

runtime.yml checks Ubuntu and native Windows. release.yml validates source tests, browser behavior and a clean tarball install. Publishing a GitHub Release runs the npm OIDC publisher on that exact validated archive. A manual workflow run validates without publishing. npm must trust alexshpunt/pi-coding-agent-test, workflow release.yml, with direct publishing allowed.

Browser package verification

Run node test/fixtures/verify-browser-package-consumers.mjs preflight to check prerequisites. Run the same command with pack to build a tarball, install it in a clean project, and exercise the installed browser fixture. Both commands are bounded and never publish.

License

MIT