Probe Kit

Business-agnostic probing library. Playwright is one built-in subject kind (the browser handle); Probe Kit is not limited to the browser.

AI writes real TypeScript step code; Probe Kit executes it, keeps a long-running session alive across steps, collects evidence, and archives the full probing process. The object being probed is called the subject and is exposed to every step as ctx.subject (a Handle).

Probe Kit does not model business state, project topology, worktrees, containers, branches, deployment environments, or concurrency policy. Those are external environment concerns expressed through project configuration, ports, env vars, data dirs, and filesystem paths.

The authoritative specification is doc/PROBE_KIT_SPEC.md. This README is a usage guide; when they disagree, the SPEC wins.

Core Concepts

| Concept | What it is | |---|---| | Probe | A probing-purpose container. Answers: "What are we investigating/validating?" Lives at .probe/<title>/. | | Session | One controller-backed long-running session under a Probe. Answers: "How do I reach the running subject?" Each probe start creates a new sessionId. A Probe may have many sessions over time. | | Step | One execution of AI-written TypeScript code against a Handle. Source lives in steps/; evidence lives in step-runs/<slug>/. | | Handle | The runtime handle to the probed subject (ctx.subject): browser / electron / process / none. Browser-only APIs (page, screenshot, ...) exist only on browser handles. | | Probe Kit | Stateless executor. It never knows about current, runId, worktrees, or business entities. |

Installation

pnpm add -D @probe/kit @playwright/test
# one-time browser install
pnpm exec playwright install chromium

Add a probe.config.ts at your project root (see below).

Quick Start

# 1. Create a Probe container (purpose template generated)
probe create --title workflow-trigger

# 2. Start one long session under it
probe start --probe workflow-trigger
# -> sessionId / sessionFile printed

# 3. Run steps against that session (AI writes steps/*.step.ts)
probe step --session-file .probe/workflow-trigger/sessions/<sessionId>.json \
  .probe/workflow-trigger/steps/001-smoke.step.ts

# 4. Inspect, then stop (session file is retained as history)
probe status --session-file .probe/workflow-trigger/sessions/<sessionId>.json
probe stop  --session-file .probe/workflow-trigger/sessions/<sessionId>.json

The normal probing loop is:

create probe -> start session -> run steps -> stop
            -> change code -> start NEW session -> run more steps

Directory Structure

.probe/<title>/
  probe.json                 probe metadata (title, project, status)
  purpose.md                 what we're investigating (template)
  sessions/
    <sessionId>.json         one session's runtime record (port/pid/baseURL)
  logs/
    <sessionId>/
      backend.log            backend process stdout/stderr
      frontend.log           frontend process stdout/stderr
      dev.log                electron dev server log
  browser/
    <sessionId>/
      storage-state.json     (optional) captured browser state
  steps/                     authored step SOURCE code
    001-smoke.step.ts
    002-open-dialog.step.ts
  step-runs/                 step EXECUTION evidence (probe-level, accumulates across sessions)
    001-smoke/
      step.ts                source snapshot
      result.json            pass/fail/duration/error
      screenshot.png         on success
      failure.png            on failure
      console.json
      network.json
      layout.json / aria.txt (when step calls captureLayout/captureAria)
  bugs/
    BUG-001.md               standard bug report
  summary.md                 probe-level aggregated report

CLI Reference

All commands take explicit arguments. There is no current probe.

| Command | Required args | Notes | |---|---|---| | probe create | --title <title> | Creates container + templates. Does not start anything. | | probe start | --probe <title> | New sessionId each run. --session-id/--port optional. | | probe step | --session-file <file> <stepFile> | --probe is not accepted (a probe may have many sessions). --slug optional. | | probe status | --session-file <file> | Controller status + Probe step-runs overview. | | probe stop | --session-file <file> | Stops controller; keeps session file (sets status: stopped). | | probe list | — | Lists probes under probesDir. |

Env: PROBE_ROOT (project root, default cwd).

Step Contract

Steps are real TypeScript, not a DSL.

import { expect } from "@playwright/test"
import type { ProbeStep } from "@probe/kit"

const step: ProbeStep = {
  id: "workflow/open-trigger-dialog",

  async run(ctx) {
    const { page } = ctx

    await page.goto(ctx.baseURL)
    await expect(page.getByRole("heading", { name: "devloop" })).toBeVisible()

    await page.getByRole("button", { name: "触发执行" }).first().click()
    await expect(page.getByRole("dialog")).toBeVisible()

    await ctx.screenshot("dialog.png")
    await ctx.captureLayout(["[role=dialog]", "input", "button"], "layout.json")
  },
}

export default step

StepContext provides: page / context / browser? / electronApp?, baseURL, probeDir, stepRunDir, and the evidence toolbox — screenshot / captureLayout / captureAria / writeJson / writeText / api / markBug.

step.id is a human-readable label and is decoupled from the evidence slug (derived from the filename, or overridden with --slug).

Project Configuration (probe.config.ts)

External projects express isolation entirely through config. Probe Kit never infers the isolation strategy.

Web project

import { defineProbeConfig, webApp } from "@probe/kit"

export default defineProbeConfig({
  name: "devloop",

  ports: {
    WEB_PORT: "auto",     // dynamic allocation
    SERVER_PORT: "auto",
  },

  vars: {
    DATA_DIR: "<probeDir>/data/<sessionId>",
  },

  target: webApp({
    baseURL: "http://127.0.0.1:<WEB_PORT>",
    processes: [
      {
        name: "backend",
        command: "pnpm --filter @devloop/server dev --port <SERVER_PORT>",
        readyUrl: "http://127.0.0.1:<SERVER_PORT>/api/ops/workflows",
        env: { DEVLOOP_DATA_DIR: "<DATA_DIR>" },
      },
      {
        name: "frontend",
        command: "pnpm --filter @devloop/web dev --port <WEB_PORT>",
        readyUrl: "http://127.0.0.1:<WEB_PORT>",
      },
    ],
  }),
})

Electron project

import { defineProbeConfig, electronApp } from "@probe/kit"

export default defineProbeConfig({
  name: "flashbox",

  ports: {
    RENDERER_PORT: "auto",
  },

  target: electronApp({
    appDir: "apps/host",
    devCommand: "pnpm --filter flashbox-host dev --port <RENDERER_PORT>",
    rendererReadyUrl: "http://127.0.0.1:<RENDERER_PORT>",
    userDataDir: "<probeDir>/userData/<sessionId>",
    env: {
      FLASHBOX_PROFILE: "development",
      FLASHBOX_SKIP_ONBOARDING: "1",
      FLASHBOX_USERDATA: "<probeDir>/userData/<sessionId>",
      APP_INSTANCE: "probe-<probeTitle>-<sessionId>",
    },
  }),
})

Template variables

Every config string supports <NAME> interpolation:

<probeTitle>   <probeDir>   <sessionId>   <rootDir>
<any port name, e.g. WEB_PORT>
<any var name, e.g. DATA_DIR>

Ports

ports: {
  WEB_PORT: "auto",      // dynamically allocated, written to session file
  SERVER_PORT: "3583",   // fixed (for reproducibility)
}

How isolation is built (external concerns)

Probe Kit only consumes rootDir + config + probeTitle + sessionId. Everything else is the external project's job:

worktree isolation      -> each worktree runs its own probe create/start/step/stop
container isolation     -> each container has its own PROBE_ROOT + probesDir
data isolation          -> DATA_DIR / userDataDir via <probeDir>/.../<sessionId>
port isolation          -> ports: { X: "auto" }
config isolation        -> different probe.config.ts per environment

Evidence Rules

Each step execution writes to step-runs/<slug>/. Required files:

step.ts        source snapshot of the executed step
result.json    { stepId, status, durationMs, error?, timestamp }
console.json   page console messages during this step
network.json   network requests during this step

On success: screenshot.png (recommended). On failure: failure.png + result.json.

Layout bugs must call captureLayout() — screenshot-only layout reports are insufficient. captureLayout records boundingBox + full computed style (display/visibility/position/zIndex/width/height/padding/margin/gap/flex/grid/overflow/fontSize/color/backgroundColor/border/...).

Bug Reports

ctx.markBug({ id, area, severity, reproduction, expected, actual, evidence? }) writes bugs/<id>.md with the standard template (Reproduction / Expected / Actual / Evidence / Root Cause / Fix / Regression Result). After fixing, rerun the same probe and update the bug's regressionResult to verified.

Invariants (non-negotiable)

• No current probe. All commands take explicit args.
• No runId. Containers are Probes (.probe/<title>/).
• probe step requires --session-file (not --probe).
• probe stop keeps the session file as history.
• All Probe state lives inside .probe/<title>/.
• External projects express isolation through config, never through Probe Kit domain logic.
• No business names (flashbox, devloop, workflow, miniapp, ...) inside the kit.

Architecture

cli.ts                  command routing + process management (no Playwright objects)
controller.ts           the ONLY process holding Playwright objects; HTTP RPC (status/step/stop)
controller-client.ts    CLI-side RPC to the controller (reads explicit sessionFile)
config.ts               load probe.config.ts; port allocation; <NAME> template interpolation
artifact.ts             .probe/<title>/ structure; purpose template; summary
session-store.ts        read/write an explicit session file (no .probe/, no current)
step-runner.ts          jiti-import .step.ts; inject StepContext; archive source; collect evidence
evidence.ts             console/network/layout/aria collectors
bug-report.ts           standard bug markdown
launchers/web-app.ts    spawn backend/frontend; wait readyUrl; launch Chromium
launchers/electron-app.ts  spawn devCommand; wait rendererReadyUrl; launch Electron

The controller is a detached long-running process; the CLI just routes commands to it over local HTTP RPC. Step execution reuses the same browser page across steps within a session.