@intentsolutions/jrig-cli

The J-Rig seven-layer binary evaluation CLI for Claude Skills (SKILL.md artifacts) — published as a self-contained npm package so any repo can install the j-rig command and gate skill changes in CI.

Scores every skill change across package integrity, trigger quality, functional quality, regression protection, baseline value, model variance, and rollout safety — all binary yes/no criteria with an evaluator that is always separate from the skill under test.

The published binary is named j-rig (the J-Rig brand and CLI identity are unchanged). Only the npm package scope is @intentsolutions/*.

Install

# Global — gives you the `j-rig` command everywhere
npm install -g @intentsolutions/jrig-cli

# Or per-repo (recommended for CI pinning)
npm install -D @intentsolutions/jrig-cli
pnpm add -D @intentsolutions/jrig-cli

This package is self-contained: the internal eval engine is bundled into the published artifact. It pulls only real npm runtime dependencies (notably the native better-sqlite3 for evidence persistence and the published @intentsolutions/refiner for the j-rig refine loop) — there are no unpublished workspace packages to resolve.

Usage

j-rig --version                      # report the installed CLI version
j-rig --help                         # list all commands

j-rig check <skill-dir>              # deterministic package-integrity checks
j-rig validate <eval-spec.yaml>      # validate an eval spec / contract YAML
j-rig eval <skill-dir> --spec ...    # binary evaluation (5 of 7 layers by default; regression + baseline are opt-in)
j-rig eval-batch <skills-root>        # scaffold missing baselines and evaluate a skills root
j-rig suite <suite.yaml>              # balanced, resumable Task × Config target-N suite
j-rig run --task ... --config ...     # generic shell-free task/config raw Run
j-rig grade --run-id ... --grader ... # named immutable Grade over a completed Run
j-rig sample-plan --manifest ...      # balanced target-N top-up plan
j-rig batch --manifest ...            # deprecated: use j-rig suite
j-rig report --unified ...            # selected-Grader JSON/Markdown/HTML report
j-rig report --unified ... --html --serve # serve a report on loopback
j-rig report                         # show results from the SQLite evidence DB
j-rig optimize                       # cluster failures, propose one change
j-rig drift                          # check whether a skill needs reevaluation
j-rig emit-evidence                  # wrap a gate-result into a signed Statement
j-rig migrate <dir>                  # codemod v0.1.0-draft → gate-result/v1
j-rig refine                         # eval-guided SKILL.md improvement loop

j-rig eval <skill-dir> expects an eval-spec.yaml (or --spec <path>) and writes evidence to a local SQLite DB (--db <path>, default j-rig.db).

Execute MCP tools during skill evaluation

Pass --mcp-config ./eval-mcp.json to let the execution model call explicitly configured stdio MCP servers. Without this option, execution stays a single text completion. This option requires a real provider and functional evaluation; it does not discover commands from a skill or ambient MCP configuration.

{
  "servers": {
    "outreach": {
      "command": "/absolute/path/to/server",
      "args": ["--stdio"],
      "cwd": "./fixture-workspace",
      "env": ["OUTREACH_FIXTURE_PATH"],
      "tools": ["search_properties", "get_property"]
    }
  },
  "limits": {
    "maxTurns": 8,
    "maxCalls": 32,
    "maxResultBytes": 65536,
    "maxTotalBytes": 1048576,
    "timeoutMs": 60000
  }
}

Replace the example command and tool names with those your server supports. cwd is relative to the configuration file. env lists names to copy from the CLI environment, in addition to the MCP SDK's small default environment (such as PATH and HOME); it never contains literal secret values. Missing listed variables or tools refuse execution. The model sees names such as outreach__search_properties. Only the listed tools are exposed. Commands run without a shell, with the user's filesystem/network permissions: this is not a sandbox. Use disposable fixtures for tools with effects.

Each test case gets fresh server processes, including each --baseline-check case with the same capabilities and no skill body. Fresh processes do not reset databases or other external state; the operator must isolate that state. Trigger and judge calls remain separate and receive no tools. All model turns are metered; provider billing and the extra baseline cost still apply.

The displayed limits are the defaults. Limits bound model turns, attempted calls, individual tool results, accumulated model/tool text and arguments, and execution time. Tool definitions and transport messages have separate size bounds. Calls in a turn are validated before any of them runs. Unknown tools, malformed protocol data, exceeded limits, and transport failures produce failed execution evidence, not a model-quality verdict. An MCP isError tool result is returned to the model for correction. Server processes are closed after each case; cleanup cannot undo effects or guarantee termination of detached descendants created by a server.

Private files next to the DB (<db>.execution-*/skill.json or baseline.json) retain final text, status, attempted-call counts and tool event receipts. The DB records each file's hash; --emit-bundle includes matching hashes/counts in metadata.tool_execution. By default receipts omit prompts, raw tool arguments/results, environment values and server stderr. Final model text can still contain tool data, so treat the local files as private. These are observations, not proof that a tool's claims or real-world effects were correct; task-specific checks must verify those outcomes. The configuration hash does not pin server code, environment values or external state.

Opt in to tool observations for judges

Set "judgeObservations": true at the top level of the explicit MCP JSON configuration to retain bounded actual call arguments/results and forward them to the selected judge provider. The default remains counts only. This is consent to share tool data with that provider, including any personal or business data tools return; it is not a sandbox or complete secret/PII scrubber. Known credential values from credential-named environment variables, credential fields, and common credential prefixes are redacted in the observation copy. Normal tool conversations and final model output are not scrubbed by this option.

Each private execution receipt then includes a tool-observations.json artifact with schema jrig-tool-observations/v1, the runtime-owned session UUID, redaction version, and ordered calls (id, tool, arguments, status, and completed result). No skill body, system messages, assistant reasoning, or tool definitions are copied into it. Both judge adapters receive the exact artifact as untrusted data and no MCP tools. A completed MCP isError response remains an observed error response, not success. A response claiming approval is evidence of that response; host approval and actual store effects still need independent task-specific checks.

Skill and baseline contexts stay separate. Portable tool_execution.receipts[].judge_contexts contains only case/session identity, SHA-256 and byte count; matching private receipts bind those exact contexts to phase, run and configuration hash. The grader snapshot records the opt-in schema and configuration hash. Consumers can hash each private artifact's UTF-8 content and compare it with the portable reference before auditing which observations the judge received. Arguments/results never enter the portable bundle through these references.

Observation artifacts have a separate serialized maxTotalBytes bound (including JSON encoding overhead), while existing argument/result/turn/call/time limits still apply. A bound may refuse execution after an earlier effect; it cannot undo that effect. Interrupted or failed executions retain bounded partial observations and are not judged.

Evaluation evidence

When --emit-bundle <path> is used, each j-rig:local:<skill>.<model> row also carries the versioned j-rig/skill-promotion/v1 identity, grader, threshold, and regression metadata. A row with no --regression-baseline is honestly advisory for promotion even when the legacy report says ship. See 000-docs/042-AT-SPEC-skill-promotion-evidence-2026-08-02.md.

j-rig eval-batch <skills-root> discovers SKILL.md leaves in sorted order, reuses existing specs, generates missing baseline specs under a batch artifact directory, and invokes the existing evaluator once per skill. Each child emits its own gate-result/v1 Evidence Bundle into the shared SQLite evidence store; the j-rig/eval-batch/v1 manifest retains spec provenance, bundle paths, lineage, model/provider summaries, and failures. It also writes manifest.json, report.json, report.md, and self-contained report.html under the batch artifact directory. The report preserves per-skill lineage and failed/provider-outage rows; it does not synthesize an overall quality score or rollout decision. Source skills are unchanged by default; add --write-specs only when you explicitly want missing baseline specs written beside them. Generated baselines are trigger/safety scaffolds, not deep functional evidence. See 000-docs/035-AT-SPEC-eval-batch-skills-root-2026-08-01.md and 000-docs/039-AT-SPEC-eval-batch-report-projection-2026-08-02.md.

The generic j-rig run command accepts YAML task/config definitions, passes one JSON request to a shell-free harness, and persists an idempotent raw Run before any grading. It retains stdout/stderr for completed, runner-error, and timeout outcomes. See 000-docs/031-AT-SPEC-generic-runner-config-raw-run-2026-08-01.md for the request and environment protocol.

j-rig grade evaluates a completed raw Run with a validated, versioned deterministic or model-judge grader definition and stores an immutable snapshot. Pass --regrade to intentionally add a new version of an existing named Grader; runner errors and timeouts are not gradeable. Model-judge graders accept --provider minimax (or another configured provider), sample the shared judge engine when samples > 1, and persist every vote plus disagreement metadata. The grader's model remains pinned even when a provider preset has a default model. See 000-docs/032-AT-SPEC-named-graders-snapshots-regrade-2026-08-01.md.

j-rig sample-plan reads a YAML manifest of explicit Task × Config × Model cells and reports the next balanced sample indices needed to reach --target-n. j-rig suite plans, executes, grades and reports these jobs from one manifest and resumes from the immutable raw-run ledger. j-rig batch is deprecated in favour of suite; it still works, prints a notice, and is removed in the release after the one that first ships the notice. These surfaces retain runner failures instead of converting them into model grades. See 000-docs/033-AT-SPEC-balanced-sampling-uncertainty-2026-08-01.md.

j-rig report --unified requires --grader-id, --grader-version, and the full --grader-snapshot-sha256. It emits j-rig/unified-report/v1 JSON with --json, a self-contained HTML document with --html, or terminal-friendly Markdown otherwise; --output writes the exact projection to a file. --html and --json are mutually exclusive. This is unsigned local output, not a dashboard ingest or rollout decision. See 000-docs/034-AT-SPEC-unified-report-json-markdown-2026-08-01.md and 000-docs/038-AT-SPEC-unified-report-html-static-2026-08-02.md.

Add --serve to the unified HTML command to serve the generated report on loopback until Ctrl-C; --port 0 chooses an available port and --host accepts only 127.0.0.1 or ::1. j-rig suite <suite.yaml> --serve serves the suite's generated report.html through the same loopback-only server. The server exposes /, /index.html, and /healthz only. It never binds a public address or changes the unsigned-local publication boundary. See 000-docs/041-AT-SPEC-eval-report-live-serve-2026-08-02.md.

For the cell-scoped projection, use --sampling-manifest with the same three Grader identity fields. The unified report is the versioned local projection; neither report surface is a signed dashboard ingest or rollout decision.

j-rig suite <suite.yaml> is the one-command generic lifecycle. The manifest lists Task YAML files, Config YAML files, one named Grader, and target_n. Task/config paths resolve relative to the suite manifest, the Cartesian matrix is planned deterministically, and every raw Run is idempotent by its complete Task × Config × Model × sample identity. The command writes .j-rig/suites/<suite-id>/manifest.json, report.json, report.md, and report.html; rerunning the same command resumes planned/running jobs and adds only the fresh sample indices needed after harness failures. Existing run, grade, sample-plan, report, eval, and eval-batch commands remain valid compatibility seams. See 000-docs/036-AT-SPEC-eval-suite-lifecycle-2026-08-01.md for the manifest schema, validation diagnostics, and migration path.

Each fresh MCP execution generates a correlation ID. The runtime passes it to configured children as JRIG_EXECUTION_SESSION_ID, overriding any ambient value, and retains it in a tool-session.json output artifact inside the private execution receipt. Skill and naked-baseline sessions get different IDs. A scenario host can record that value in its own private trace and join on exact identity; do not infer association from call counts or file creation times. The ID remains available after a partial tool failure once session initialization succeeds. It is correlation metadata, not authentication, attestation of a child process, or evidence that the skill's behavior passed. It adds no model-visible tool or judge capability. Runtimes without an identity keep their existing artifact behavior.

Providers

The evaluator's judge layer talks to an LLM provider. The provider is auto-detected from environment variables (preferring an OpenAI-compatible endpoint) or forced with --provider:

| Provider | --provider | Env var | Model id | | ------------- | -------------------- | -------------------- | ----------------------------- | | DeepSeek | deepseek | DEEPSEEK_API_KEY | deepseek-v4-flash | | Kimi/Moonshot | kimi / moonshot | MOONSHOT_API_KEY | provider default | | OpenRouter | openrouter | OPENROUTER_API_KEY | provider default | | OpenAI | openai | OPENAI_API_KEY | gpt-4o-mini | | MiniMax | minimax | MINIMAX_API_KEY | MiniMax-M3 | | Groq | groq | GROQ_API_KEY | openai/gpt-oss-120b | | NVIDIA NIM | nvidia | NVIDIA_API_KEY | openai/gpt-oss-20b | | Anthropic | anthropic | ANTHROPIC_API_KEY | Claude models |

DeepSeek is reached by setting DEEPSEEK_API_KEY in the environment and selecting it explicitly:

export DEEPSEEK_API_KEY=sk-...
j-rig eval ./my-skill --spec ./eval-spec.yaml --provider deepseek

The DeepSeek adapter is the shared OpenAI-Chat-Completions adapter pointed at the DeepSeek endpoint and the deepseek-v4-flash model — no DeepSeek-specific SDK is required.

MiniMax M3 uses the same adapter with https://api.minimax.io/v1 and MINIMAX_API_KEY. Pin the model for reproducibility and for specs whose declared model belongs to another provider:

export MINIMAX_API_KEY=sk-...
j-rig eval ./my-skill --spec ./eval-spec.yaml \
  --provider minimax --models MiniMax-M3 --json

MiniMax M3 may inline reasoning in a leading <think>…</think> block. The adapter removes only that leading block before trigger/judge parsing; the credential remains in memory and is never written to evidence.

Provider failure boundary

A provider failure is evidence about the evaluator, not a skill grade. If any functional or judge call fails, j-rig eval records no verdict for that model and still emits a gate-result/v1 row with gate_decision: "error", the error class first in gate_reasons, and credential-free metadata.error_detail. The SQLite run is marked failed, the --json result carries gate_decision: "error" plus evaluation_error, and the process exits 2 after writing every artifact. HTTP 402 and messages such as Insufficient Balance are classified as non-retryable quota failures. A completed response with empty text is still retained as a tool-dependent boundary observation, and --samples of 2 or more absorbs transient judge failures through the agreement vote. See 000-docs/037-AT-SPEC-real-provider-failure-boundary-2026-08-02.md.

A built-in stub provider exists for pipeline plumbing only. It is gated behind J_RIG_ALLOW_STUB=1 and its results are not ground truth.

License

Apache-2.0 © Jeremy Longshore / Intent Solutions

Trigger evidence

Every eval model result now carries trigger with schema jrig-trigger-evidence/v1, status, metrics, per-case expected/selected/outcome and a SHA-256 reference. The local SQLite trigger-evidence artifact binds that reference to a fresh mode-0600 receipt in a mode-0700 directory; the receipt also retains prompts and routing reasoning. Portable bundle metadata carries the same summary without prompts or reasoning. Consumers can recompute metrics from the recorded cases and verify the private receipt against its digest.

Status is skipped for --no-trigger, not_applicable when no cases declare trigger expectations, complete when all applicable routing calls returned, and incomplete on any routing-provider error. Complete describes measurement coverage: an incorrect selection remains an explicit false-positive, false-negative or sibling-confusion outcome. It does not imply correct routing. Only complete trigger measurement appears in dimensionsEvaluated; the other statuses appear in dimensionsSkipped and remain distinguishable in metadata.

A trigger-provider error follows the existing infrastructure-failure rule: gate_decision: error, no promotion evidence, failed stored run and exit 2 after artifacts are flushed. Trigger-only execution also retains its artifact and machine-readable error. Trigger and judge calls continue to receive no MCP tools.