Antibody

Turn your agent's failures into tested immunity.

Every AI agent learns from its successes. Nobody learns from the failures — the 40-minute session that went nowhere, the same file edited five times blind, the mistake you corrected last week that happened again today. Antibody mines those failures out of raw session transcripts, root-causes them, distills a preventive guardrail — and proves each one with a replayable A/B regression test before it enters your rule set.

failed transcript ──▶ detect ──▶ root-cause ──▶ guardrail ──▶ A/B replay proof ──▶ signed SKILL.md

Why this doesn't exist yet

| | learns from unlabeled failures | proof per rule | signed / shareable | |---|---|---|---| | Mem0 / Letta / Zep (memory) | — | — | — | | skilldistill / autoharness (success traces) | — | — | — | | claude-reflect / error-collector (corrections) | needs human label | — | — | | NeMo / Invariant / agentjail (guardrails) | hand-written rules | — | — | | Trace2Skill (research) | benchmark-only | task-level | — | | Antibody | ✓ | ✓ A/B replay | ✓ Ed25519 |

Quick start

git clone https://github.com/hanishkeloth/antibody && cd antibody
export OPENAI_API_KEY=sk-...        # or ANTHROPIC_API_KEY

# mine the bundled example (an agent stuck in an edit loop)
npm run demo

# mine your own Claude Code history
node bin/antibody.mjs mine ~/.claude/projects/<your-project>/ --max 5

# re-prove every guardrail (run after every model upgrade)
node bin/antibody.mjs test antibodies

Output: antibodies/*.md — Agent-Skills-compatible guardrail files:

---
name: missing-precondition-read-the-target-file-before
category: missing-precondition
proven: true
discriminative: true
---
# Guardrail
Before editing a file, read it first and verify the exact string you intend
to replace exists — never guess file contents from the task description.
...
## Replay test
**Scenario:** ...   **Pass criteria:** ...

Drop them into .claude/skills/, compile them into your guardrail engine, or share the signed bundle with your team.

How it works

1. Detect — scans transcripts for failure signals with zero human labels: repeated tool errors, edit loops (4+ edits to one file), user corrections ("that's not what I asked", "same error again").
2. Learn — a hybrid embedding + structure classifier trained by weak supervision over your real sessions catches failures no rule matches (--learned).
3. Distill — an LLM analyst turns each failure episode into a root cause class + an imperative pre-action guardrail + a minimal replay scenario with explicit pass criteria. Lesson and test are born together from the same evidence.
4. Prove — the scenario runs twice: bare model vs. model + guardrail; a judge scores both. proven = guarded run passes. discriminative = the guardrail is causally responsible (bare run fails). Non-discriminative rules on a new model = safe to prune. Newly failing rules = capability regression caught.
5. Sign — SHA-256 manifest + Ed25519 signature per bundle, so shared immunity has verifiable provenance.

The bundled starter pack

antibodies/ ships 4 guardrails mined from real Claude Code sessions — not toy examples. Every one is proven (the guarded run passes its replay test) and two are discriminative (the unguarded run fails the same test, so the guardrail is causally responsible). A sample:

Before any call to the Agent tool, verify current concurrent subagent usage is below the limit; if at/near the limit, do not spawn another — queue the task, wait for completion, or ask the user to raise the limit. missing-precondition · proven · discriminative Mined from a session that burned turns spawning subagents into a full queue.

Before stating any real-world affiliation, location or partnership as fact, quote the exact sentence from the retrieved source that supports it; if no quote exists, label it unknown and seek another source. stale-assumption · proven

Mining a real history produces the same lesson phrased three different ways, so bundles are deduplicated by guardrail similarity — strongest member of each cluster wins (discriminative > proven > unproven, then shorter, because every guardrail costs context on every turn):

node bin/antibody.mjs dedupe antibodies --apply

The bundle is not curated by hand. 11 antibodies were mined, 4 collapsed as restatements, and antibody test pruned 3 more that failed to re-prove on a later run — LLM judges are not perfectly stable, and a guardrail that cannot survive its own regression test does not deserve to sit in your context on every turn. That pruning is the point: the suite is the filter.

The learned detector

Regex finds what somebody thought to write a rule for. Antibody treats those rules as a weak labeller and trains a detector to generalise past them — on 411 real Claude Code sessions (2312 windows).

| | value | |---|---| | precision / recall / F1 | 0.983 / 0.835 / 0.903 | | ROC-AUC / PR-AUC | 0.976 / 0.989 | | hard-negative specificity | 0.846 (does not fire on ordinary retries) | | ablation: embeddings only → hybrid | F1 0.766 → 0.903 | | novel discoveries | 70% of high-confidence windows the heuristics missed are real failures (blind LLM judge, n=20) |

Splits are grouped by transcript (windows inside one session are correlated — a random split leaks). The threshold is chosen on validation, never on test. Features are a 1536-d embedding plus 14 structural features (per-tool error concentration, longest failing run, edits concentrated on one file, ended-recovered). That hybrid is what separates real thrash from a retry that worked — the slice where a purely semantic model collapses to 0.27 specificity.

The exported model is a single linear layer, so runtime inference is one embedding call plus a dot product — no Python, no ML dependency:

node bin/antibody.mjs mine ~/.claude/projects/ --learned --max 5

Retrain on your own sessions:

python3 -m venv .venv && .venv/bin/pip install numpy scikit-learn
npm run ml:dataset && npm run ml:embed && npm run ml:train
npm run ml:discovery      # measure what the model finds that rules don't

Paper

paper/antibody.md — Antibody: Converting Agent Failures into Verified, Portable Guardrails (design, related-work positioning vs. Trace2Skill / claude-reflect / guardrail engines, limitations). PDF: paper/antibody.pdf.

Privacy

Everything runs locally. Transcripts leave your machine only as distillation prompts to the model provider you configure.

License

MIT