@jumptag/refine-skill

Refine an Agent Skill via the skill-forge judge → hitl loop, in a sandboxed Docker container.

Quick start

export ANTHROPIC_API_KEY=sk-...
npx @jumptag/refine-skill ./path/to/skill

Default: 3 iterations max, claude-sonnet-4-5, telemetry written to <path>/.refine/log.json.

Requirements

• Node 20+ (for npx).
• Docker (Engine 20.10+, daemon running).
• An API key for one of: Anthropic, OpenAI, Google, xAI, Mistral, Groq, OpenRouter — matched to your --model choice. See MODELS.md for the full list of supported models, env vars, and where to get keys.

Usage

npx @jumptag/refine-skill <path> [options]

| Option | Default | Effect | |---|---|---| | --iterations N | 3 | Max passes before stopping at the cap | | --model M | claude-sonnet-4-5 | Any model pi.dev supports | | --image TAG | ghcr.io/barryroodt/refine-skill:<pkg-version> | Override the image | | --pull POLICY | missing | always / never / missing | | --no-log | off | Skip writing .refine/log.json | | --dry-run | off | Print the docker invocation and exit | | --verbose | off | Stream pi output uncut | | --pi-timeout SECS | 600 | Per-pi-call timeout; auto-scales to 1800 for opus/fable-class models |

The model catalog lives in the image, not the CLI. The pi model catalog is baked into the Docker image (tagged to the package version), so @latest on the npm CLI does not guarantee the newest models. A slug the image predates fails fast with exit 5 and a list of the closest supported slugs. If a specific new model matters, pass a matching/newer --image tag (or --pull always), or run --dry-run first to see the resolved image tag.

Filtering streamed output. Piping the CLI through a redactor or logger makes the shell report the last command's status, masking the CLI's real exit code and defeating the exit-code contract below. Preserve it with pipefail + PIPESTATUS:

set -o pipefail
npx @jumptag/refine-skill@latest ./skill --model claude-opus-4-8 2>&1 \
  | sed -E 's/sk-ant-[A-Za-z0-9_-]+/sk-ant-***REDACTED***/g'
status=${PIPESTATUS[0]}   # capture immediately, before any other command resets $?
exit "$status"

How stopping works

The loop exits at the first matching rule:

1. Pass 1 → never stops.
2. Judge produces zero items → all_obsolete.
3. All items match "already satisfied / no-op / superseded" → all_obsolete.
4. Score fraction (score / max) gains < 0.02 vs. the previous pass → delta_below_threshold. The rubric max can move between passes, so this compares normalized fractions, not raw points — a positive raw delta can still be a normalized regression.
5. All items match trade-off / diminishing / LOW priority → tradeoff_floor.
6. Pass > --iterations → max_iterations (exit 1, still successful).

Telemetry

.refine/log.json contains per-pass score/grade/delta, per-item commit messages + diffs, stop reason, model, image tag, timestamps. Disable with --no-log.

.refine/log.json also carries a summary block — input (pass 1, the original skill), best, and final scores as normalized fractions, plus regressed_below_input. Because the loop can converge on a state scoring below the input, check summary.regressed_below_input before adopting the result, and revert to summary.best (via that pass's per-item commits) if it is true.

Running without npx

docker run --rm -i \
  -v "$PWD/path/to/skill:/work" \
  -e ANTHROPIC_API_KEY \
  ghcr.io/barryroodt/refine-skill:latest \
  /work --iterations 3 --model claude-sonnet-4-5

Exit codes

| Code | Meaning | |---|---| | 0 | Natural convergence (any of rules 2-5) | | 1 | Max iterations reached (still successful) | | 2 | Bad path / missing SKILL.md | | 3 | Missing / mismatched API key | | 4 | Docker not available | | 5 | Model not found in image catalog | | 10 | Pi crash | | 11 | Judge output malformed | | 12 | Hitl partial apply | | 13 | Disk full / OOM | | 14 | Another refine running on the same path | | 130 | SIGINT | | 143 | SIGTERM |

Spec

specs/2026-05-20-deftly-refine-cli-design.md

Credits

refine-skill is a thin orchestration harness around two existing pieces of work:

• Skill Forge by @WrathZA — skill-forge-judge + skill-forge-hitl provide all the actual refinement logic (scoring rubric, per-item HITL loop). Apache 2.0; pinned tag 2026.04.30; baked into the image at build time and copied verbatim. See NOTICE.
• pi.dev coding agent by @mariozechner — provider-agnostic LLM harness that runs the two skills inside the container.

This project (@jumptag/refine-skill, MIT) just wires them together: Node CLI + bash outer loop + deterministic stop rules + telemetry.