pi-research-loop

pi-research-loop is a Pi Package plus standalone prl CLI for reproducible research code changes and experiments.

It provides:

• one Git branch/worktree per Agent Task;
• an immutable checkpoint commit per Run;
• a detached local runner that does not depend on Pi TUI lifetime;
• declarative process.exit, log.regex, timer.timeout, and file.created events;
• short event-driven Pi notifications instead of LLM log polling;
• Pi-session ownership so concurrent windows cannot consume or act on each other's Runs;
• optional MLflow REST tracking and W&B environment injection.

Requirements

• Node.js >= 22
• Git with git worktree
• Pi >= 0.82.1 for the Extension tools

Each server runs independently. PRL does not require a central service. MLflow and W&B are optional services for cross-machine tracking.

Installation after npm release

The CLI and Pi resources are installed separately:

# Installs the `prl` executable
npm install --global [email protected]

# Installs the Pi Extension, Skill, and Prompt
pi install npm:[email protected]

Verify:

prl --help
pi list

For upgrades, use the same versioned commands with a newer version, or:

npm install --global pi-research-loop@latest
pi update npm:pi-research-loop

Git installation

Install directly from the public repository and pin the release tag:

export PRL_REPO=https://github.com/kuan-er/pi-research-loop.git
export PRL_VERSION=v0.6.1

npm install --global "git+${PRL_REPO}#${PRL_VERSION}"
pi install "${PRL_REPO}@${PRL_VERSION}"

Local development installation

npm ci
npm run build
npm install --global .
pi install .

The complete multi-machine installation guide is in docs/INSTALL.zh-CN.md. The release procedure is in docs/RELEASING.md.

Quick start

Run PRL commands from the main worktree of a research Git repository:

cd /path/to/my-research-repo
prl init
prl doctor

prl task start --hypothesis H001 --name implementation-route
# Give the returned worktree path to the Pi Agent; reuse it across phases.

prl task checkpoint T-... --message "phase: geometry"
prl run launch --task T-... -- python train.py
prl run inspect R-...
prl run logs R-... --tail 100
prl run retry R-...
prl experiments rebuild
prl task finish T-...

prl init creates:

AGENTS.md
research/
├── PROJECT.md
├── STATE.yaml
├── DECISIONS.md
├── EXPERIMENTS.md
└── hypotheses/
.pi-research/
└── config.yaml
runs/

A Task owns one task_id, branch, and worktree for an implementation route. Repeated prl_task_start calls for an active hypothesis reuse that Task by default. A Task can have multiple phase commits and Runs. Use --new-worktree-reason stable_baseline|concurrent_agents|disposable_experiment only when a second worktree is intentional. PRL never pushes, merges, or removes a worktree automatically; review and merge the finished branch to main once at the end.

Events

Create events.yaml:

events:
  - id: cuda_oom
    type: log.regex
    pattern: "CUDA out of memory"
    source: both
    max_matches: 1
    actions: [terminate, wake_agent]

  - id: checkpoint_ready
    type: file.created
    path: outputs/best.ckpt
    actions: [record]

  - id: timeout
    type: timer.timeout
    after_seconds: 28800
    actions: [terminate, wake_agent]

Launch with an argv array; the command does not go through a shell:

prl run launch --task T-... --events events.yaml -- python train.py --seed 42

process.exit is always added automatically. Event records are persisted in run.yaml. Notifications are durable records under .pi-research/notifications/ with a matched -> queued -> delivered -> acknowledged history, retry metadata, and a critical flag. Pi can exit while the experiment continues. Ordinary wake quota is charged only after delivery; critical failure and process.exit notifications bypass it.

Pi session isolation

Runs launched through the Pi Extension—or through prl from Pi's bash tool—record the current PI_SESSION_ID. Pending events, the monitor widget, prl_context.active_runs, prl_run_inspect, and prl_run_control are then scoped to that owning session.

Other Pi windows leave those pending events untouched. If the owner window is closed, its events remain on disk and are delivered/coalesced after that same Pi session is resumed. Starting /new creates a different owner; use /resume to continue an existing Run conversation and its iterative summaries.

Legacy Runs created before session ownership, or Runs launched from a normal terminal, are unbound and never auto-injected into an arbitrary Pi window. Adopt one explicitly from the intended window:

prl_run_claim {"run_id":"R-..."}

A Run already owned by another session cannot be claimed, inspected, terminated, or retried from the current window. If the original session is lost, use prl_run_transfer (or prl run adopt R-... --confirm) after explicit user confirmation. The owner change and reason are audited, and pending notifications move with the Run. Direct standalone CLI use outside Pi remains available for administrative inspection.

Dependent Runs and GPU handoff

For a GPU handoff, enqueue the successor before the parent finishes. The background worker then waits for the parent to succeed, validates the optional checkpoint, acquires a cooperative GPU lease, and launches without relying on an LLM wakeup:

prl run launch --task T-... --gpu 6 -- python train_long.py
prl run enqueue --task T-... --depends-on R-... \\
  --checkpoint-path /tmp/checkpoint.pt --gpu 6 --gpu-wait-seconds 86400 \\
  -- python train_next.py

The dependent Run remains queued while the parent or GPU is unavailable. A checkpoint is considered usable only after size/mtime stability and streaming SHA-256 verification; it is then recorded as an immutable durable_complete artifact. Dependencies can accept succeeded and/or checkpointed_stop, or an explicit failure class such as cuda_oom for a guarded fallback; they can pin an artifact ID/checksum/complete marker instead of a mutable latest.pt. Other failures do not take an OOM branch.

nvidia-smi is checked before and after lease acquisition, and PRL-managed Runs coordinate through .pi-research/runtime/gpu-leases/. Running jobs warn (without killing training) when a foreign or ghost GPU PID appears. This remains a cooperative lease; unmanaged external processes can race it. Use Slurm or another scheduler when hard reservation is required. If the parent is retried, it is marked superseded and stale dependent/wakeup actions do not launch new work.

Safe termination

The default termination grace is 900 seconds, and SIGKILL is disabled unless the Run policy enables it or the caller explicitly uses force: true. Signals target the full process group (for example torchrun plus workers), and every request records actor, session, reason, source, timestamps, and signal timeline.

termination:
  checkpoint_signal: SIGUSR1
  checkpoint_event_id: checkpoint_complete
  graceful_timeout_seconds: 900
  terminate_signal: SIGTERM
  force_kill_timeout_seconds: 120
  allow_sigkill: false

With checkpoint_signal, PRL sends the checkpoint signal first and waits for checkpoint_event_id (or the graceful deadline) before TERM. A checkpoint-confirmed stop becomes checkpointed_stop, distinct from an uncheckpointed terminated Run. CLI control accepts --reason and an explicit --force.

Forking experiment variants

A fork inherits the parent commit, argv protocol, seed, events, resource/checkpoint policy, and tracking defaults, then records only the requested override diff:

prl run fork R-parent --resume latest-complete \
  --set training.targets_per_source=19 \
  --set training.pairs_per_update=152 \
  --gpu 1,4 --dry-run
# Remove --dry-run after reviewing the minimal diff.

The Extension equivalent is prl_run_fork. Overrides are classified as runtime/resumable, protocol-changing, or structural; structural changes reject exact resume and require weights_only or fresh. Register checkpoints first with prl_run_artifact_register or prl run artifact register.

Storage, health, and performance

Since 0.6.1, the Extension does not poll every historical run.yaml. The first upgraded startup builds small active-Run and notification queue indexes once; subsequent startup, status, and notification work reads only queued/running Runs or deliverable notifications. fs.watch drives immediate updates, with separate 10-second low-cost fallbacks for filesystems that lose watch events. Git-root discovery is cached per process.

Run launch performs a free-space check. Set storage.estimated_checkpoint_bytes to reserve the configured multiplier (default 2x); an undersized filesystem is rejected before training. PRL exports a unique PRL_OUTPUT_DIR (under the Run record by default, or <storage.path>/<run_id>) so experiment families can avoid output collisions. Progress watchdogs emit a warning only—never automatic termination. Run YAML separates process, trainer, checkpoint, durable replica, tracking, and notification lifecycle status. Launch profiles expose Git checkpoint, snapshot, tracking, worker spawn, and storage figures.

Automatic Git checkpointing rejects dangerous files, oversized files, and untracked directories above the configured count. Keep outputs, datasets, model weights, caches, and durable artifacts outside the Task and Run worktrees.

Run inspection

prl run list
prl run inspect R-...
prl run inspect R-... --mode status
prl run inspect R-... --mode events
prl run inspect R-... --mode run_yaml
prl run inspect R-... --mode log_search --query "loss|accuracy"
prl run inspect R-... --mode git_diff
prl run inspect R-... --mode tracking
prl run inspect R-... --mode notifications
prl run inspect R-... --mode artifacts
prl run inspect R-... --mode performance
prl run logs R-... --tail 100

Each Run contains only:

runs/<run_id>/
├── run.yaml
├── console.log
└── result.md

console.log merges stdout and stderr with timestamps and source labels. Writes are asynchronous and batched; logs rotate at the configured size. result.md has an objective section generated by PRL and a preserved Agent analysis section.

Each Run executes from a detached immutable Git worktree created at its recorded commit, not from the mutable Task worktree. PRL verifies HEAD plus checksums of command-referenced files immediately before spawning. Relative outputs therefore live in the Run snapshot; production output, checkpoint, model, and cache paths should be absolute paths outside all worktrees.

Pi tools

The Extension registers:

• prl_context
• prl_task_start
• prl_run_launch
• prl_run_enqueue
• prl_run_claim
• prl_run_transfer
• prl_run_artifact_register
• prl_run_fork
• prl_run_inspect
• prl_run_control
• prl_task_checkpoint
• prl_task_finish

A recommended Pi instruction is:

Call prl_context first; it contains only active Runs owned by this Pi session.
Start a Task with prl_task_start and modify code only in the returned worktree.
Launch experiments with prl_run_launch (or prl_run_enqueue for a dependent Run) and declare useful events.
Do not inspect, terminate, or retry Runs from another Pi session. Do not poll processes or logs.
Wait for [PRL EVENT], use prl_run_inspect in bounded mode, update the Run analysis/research state when useful, and decide whether to inspect, modify, retry, or stop.

The Skill is available at skills/prl-research-loop/SKILL.md.

MLflow and W&B

Enable MLflow in .pi-research/config.yaml:

tracking:
  mlflow:
    enabled: true
    tracking_uri_env: MLFLOW_TRACKING_URI
    experiment_name: my-project
    required: false

PRL records the PRL Run ID, Task ID, hypothesis, Git commit, branch, command, hostname, exit code, duration, and best-effort artifacts. If MLflow is unavailable and required: false, the local experiment continues.

PRL does not reimplement the W&B SDK. It injects:

PRL_RUN_ID
PRL_TASK_ID
PRL_HYPOTHESIS_ID
PRL_GIT_COMMIT
PRL_OUTPUT_DIR
WANDB_RUN_ID
WANDB_NAME
WANDB_GROUP
WANDB_TAGS

The PRL-created W&B identity and the trainer's actual W&B identity are stored separately. A trainer can report the authoritative identity with:

PRL_TRACKING_JSON {"wandb_run_id":"actual-id","wandb_url":"https://wandb.ai/entity/project/runs/actual-id"}

PRL also recognizes normal wandb.ai/.../runs/<id> console URLs. See examples/wandb_minimal.py.

Safety

Automatic checkpoint commits refuse secrets, .env files, model weights, common credential files, experiment outputs, and files larger than the configured limit. Do not rely on this as a complete secret scanner; review diffs before committing or pushing.

Development and release

npm ci
npm run build
npm test
npm run release:check
npm pack

See docs/RELEASING.md before publishing. The GitHub workflows run tests on pull requests and can publish a GitHub Release to npm using the NPM_TOKEN secret.