rad-experiment

CLI for publishing, reproducing, and browsing AI-generated optimization experiments on the Radicle network. Each experiment is a signed Collaborative Object (COB) of type cc.experiment.

Install

npm install -g rad-experiment

Requires Radicle and Node.js >= 18. Both rad-experiment and rad-cob-experiment (the Radicle COB helper) are installed onto $PATH.

From source

git clone <repo> && cd rad-experiment-ts
npm install && npm run build && npm link

Usage

All commands run inside a Radicle-tracked git repository.

Publish an experiment

Record benchmark results for a candidate commit vs. its base:

rad-experiment publish \
  --base abc123 --head def456 \
  --metric p95_latency --unit ms --direction lower_is_better \
  --runner amd64 --os linux --cpu xeon-8375c \
  --baseline-median 142500 --baseline-n 10 --baseline-std 8200 \
  --candidate-median 98700 --candidate-n 10 --candidate-std 5100 \
  --delta 3074 \
  -d "Connection pooling for /api/search handler"

Experiment published: 5dfc3d665c12dbb6247454c44c4d2dc6cc422a5e
  metric:    p95_latency +30.74%
  baseline:  142.500 ms
  candidate: 98.700 ms

Numeric values are integers scaled x1000 to avoid floating-point ambiguity (e.g. 142500 = 142.5 ms). Delta is percentage x100 (e.g. 3074 = +30.74%).

Secondary metrics

Track additional metrics alongside the primary one:

rad-experiment publish \
  ... \
  --secondary "rss_peak:MB:256000:241000:-586" \
  --secondary "p99_latency:ms:310000:285000:-806"

Format: name:unit:baseline_x1000:candidate_x1000:delta_x100[:regressed]

List experiments

rad-experiment list

5dfc3d6 p95_latency +30.74%
bd3f5b8 p95_latency +12.40% [1 verified]

Filter by verification status:

rad-experiment list --reproduced    # only experiments with confirmed reproductions
rad-experiment list --unverified    # only experiments without reproductions

Show experiment details

rad-experiment show 5dfc3d665c12dbb6247454c44c4d2dc6cc422a5e

Experiment 5dfc3d665c12dbb6247454c44c4d2dc6cc422a5e

  Connection pooling for /api/search handler

  base:         abc123...
  head:         def456...

  metric:       p95_latency (ms)
  direction:    lower_is_better

  baseline:     142.500 ms (n=10)
  candidate:    98.700 ms (n=10)
  delta:        +30.74%

  runner:       amd64 (linux, xeon-8375c)
  build:        ok
  tests:        ok
  agent:        claude-code/claude-opus-4-6
  author:       did:key:z6Mk...

Use --json for machine-readable output:

rad-experiment show --json 5dfc3d66...

Reproduce (verify) an experiment

Anyone can independently verify an experiment's results:

rad-experiment reproduce 5dfc3d665c12dbb6247454c44c4d2dc6cc422a5e \
  --verdict confirmed --runner arm64 \
  --baseline-median 148200 --baseline-n 10 \
  --candidate-median 101500 --candidate-n 10 \
  --delta 3148 \
  --notes "Reproduced on staging cluster"

Reproduction added to 5dfc3d6
  verdict: confirmed

Verdicts: confirmed, failed, inconclusive.

Global options

rad-experiment --repo /path/to/repo <command>   # specify repo path (default: cwd)

How it works

The CLI shells out to rad cob create/update/list/show for all storage operations. Radicle handles signing, replication, and conflict resolution.

When Radicle evaluates a cc.experiment COB, it invokes the rad-cob-experiment helper binary via the external COB protocol (JSON Lines on stdin/stdout). This helper applies operations to experiment state — it's installed alongside the CLI and found on $PATH by name.

Architecture

See ARCHITECTURE.md for the data flow diagram, source file map, and field naming conventions.

Testing

npm test                # type-check + all tests (87 tests)
npm run test:unit       # pure function + golden-file serialization tests
npm run test:protocol   # rad-cob-experiment stdin/stdout protocol tests
npm run test:integration  # full E2E with a real radicle repo (requires rad)

Golden files in src/__tests__/golden/ contain reference JSON generated by the Rust implementation to verify byte-identical COB serialization.

License

MIT OR Apache-2.0