@pickled-dev/cli

Pickled tests whether real agents can answer and build with your product, across declared context paths, using deterministic evidence.

The CLI for Pickled. Use it locally or in CI to check whether agents can answer and build from the product context you publish. No LLM grades another LLM.

Full docs: docs.pickled.dev.

Install

bun add -g @pickled-dev/cli
# or run without installing:
bunx @pickled-dev/cli <command>

Commands

• pickled init [path] writes a starter pickled.yml.
• pickled test [path] scores a question's examples.pass / examples.fail offline. No model calls.
• pickled check [path] runs the questions: asks the agents and scores their answers against the fact contract.
• pickled build [path] has edit-capable agents work in a fresh workspace and scores the verifier.
• pickled ci [path] runs the configured questions and builds in one pass, writes a receipt per kind, and gates the run. For CI.
• pickled audit [path] scans agent-facing files for broken refs and oversized sections. No model calls.
• pickled report <file> re-renders a saved receipt as terminal, markdown, or JSON. No model calls.

Minimum config

A registered source is the context Pickled is allowed to use: a local file or a URL. Anything not registered does not count.

schemaVersion: 2

product:
  name: my-product
  description: short one-liner about what your product does

sources:
  readme: { path: ./README.md }

agents:
  quick:
    provider: claude-code
    model: claude-haiku-4-5

contexts:
  from_readme: { mode: inject, source: readme }

facts:
  install_command:
    statement: my-product installs with bunx my-product.
    match:
      allOf: ["bunx my-product"]

questions:
  - id: install
    question: How do I install my-product?
    agents: [quick]
    contexts: [from_readme]
    expects: [install_command]

thresholds:
  questions: 80

That runs one question with your README injected. Add more contexts to compare model memory, injected sources, web discovery, and MCP discovery. A build instead has the agent edit a workspace and runs your verifier; run it with pickled build.

Cost controls

For paid agents, a run can expand to many (agent × context) cells. These flags keep that in check:

pickled check . --plan                           # dry run: no model calls
pickled check . --max-cells 10                   # fail before spending
pickled check . --sample 2 --seed nightly-2026   # deterministic sample per task
pickled build . --plan                           # preview build cells and executions
pickled build . --verify-only                    # prove the harness; no agent runs

pickled build --verify-only runs each build's preflight and reference control (the untouched fixture must fail failToPass and pass passToPass; a declared referenceSolution must apply and clear the verifier) and prints one line per build: proven, unproven (no reference declared), or broken. No agent runs, so it is the cheap way to check a fixture before spending tokens, and it exits non-zero if any harness is broken. --output file.json writes the per-build proof results as JSON (or --json to print them to stdout). It accepts only --task, --json, --output, and --verbose; the agent, sampling, and threshold flags are rejected rather than silently ignored.

The receipt records expandedCells, selectedCells, and seed so a reviewer can see what ran and rerun the same sample. Builds also report selectedExecutions (cells x trials), which is what --max-cells gates.

Narrow a run by the names in pickled.yml:

pickled check . --task install
pickled check . --agent quick
pickled check . --context from_readme

In CI

pickled ci runs the configured questions and builds in one pass, writes a CI-safe JSON receipt per kind, appends a markdown summary to the job summary, and exits non-zero if any thresholded run fails:

- run: |
    pickled ci . \
      --questions-max-cells 20 \
      --builds-max-cells 4 \
      --report-dir pickled-reports
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: pickled-reports
    path: pickled-reports/*.json

It appends markdown to $GITHUB_STEP_SUMMARY automatically when that variable is set (override with --summary-file). It runs both kinds to completion even if one fails its threshold, so the job always uploads every receipt. With no --questions/--builds flag it runs both kinds the config declares.

The receipts are CI-safe by default: verdicts, evidence ids, provenance, and build attempts, without full answers, source text, transcripts, diffs, or command output. To re-render a saved receipt later (for example a different format from an uploaded artifact) without rerunning paid agents, use pickled report <file> --format markdown|terminal|json; pass --verbose there for a forensic receipt.

Current support

| Concept | Works today | | --- | --- | | Sources | local files, URLs, codebase globs | | Context modes | memory, inject, web, mcp | | Agents | Claude Code, Codex CLI, Anthropic API, OpenAI API | | Output | terminal and JSON for runs; markdown from saved receipts and audits |

Read more

• Getting started
• pickled.yml reference
• GitHub Actions

License

MIT