Skillkeeper

Keep every Agent Skill observable, testable, and reversible.

Skillkeeper is a local-first control plane for the skills your AI agents depend on. It scans open SKILL.md packages, measures explicit replay requirements, stages bounded repairs, validates them against held-out cases, and keeps promotion and rollback auditable.

Healing creates a candidate. It never silently edits the live skill.

Install

Skillkeeper is a Python CLI. The npm package bundles the Skillkeeper Python source and a small cross-platform launcher, so npm users do not need pip or uv. Both installation paths require Python 3.10 or newer on the machine.

npm

Try it without a permanent installation:

npx skillkeeper --help

Or install the command globally:

npm install --global skillkeeper
skillkeeper --help

uv

Run the tagged release without cloning the repository:

uvx --from git+https://github.com/jyotipravatiitm/[email protected] \
  skillkeeper --help

From source

git clone https://github.com/jyotipravatiitm/skillkeeper.git
cd skillkeeper
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
skillkeeper --help

The runtime has no third-party Python dependencies.

First scan

Run Skillkeeper in a project containing .agents/skills, .claude/skills, or another Agent Skills root:

npx skillkeeper scan .agents/skills .claude/skills
npx skillkeeper inventory

State is stored locally in .skillkeeper/. Pass --state-dir PATH before the command to use a different location.

Why it exists

Agent runtimes execute skills. Registries help you find them. Optimizers can rewrite them. Skillkeeper answers the operational questions around the whole inventory:

• Which skills are invalid, untested, duplicated, or unhealthy?
• What repeatable evidence shows that a skill is failing?
• Does a proposed change improve training cases and still pass unseen cases?
• What exact version was evaluated and promoted?
• Can I inspect the diff and restore the previous version immediately?

Skillkeeper follows the open Agent Skills folder format. It does not introduce a replacement for SKILL.md.

Research origin

Skillkeeper is inspired by Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills by Huang et al. (2026).

Read the original paper (PDF) · Paper page · Authors' code · Technical lineage and citation

The paper proposes a training-time reinforcement-learning loop in which a proposer, solver, and skill controller co-evolve an inventory of skills. Skillkeeper adapts that core feedback-loop idea for everyday agent operations: capture repeatable failures, evaluate a staged skill change, promote it only with evidence, and retain a rollback path.

Skillkeeper is an independent, local SkillOps project—not an implementation or reproduction of the paper's model-training system. The technical lineage explains exactly what was carried over, what was adapted, and what remains future work.

See the improvement lifecycle

The included presentation demo starts with a generic, text-heavy deck skill and adds an evidence-backed quality contract. The deck artifacts make the before/after change easy to inspect.

| Before | After | |---|---| | | |

Open the before PDF, after PDF, or editable before and after PowerPoint files. The demo report contains the evaluation receipts and the honest boundary of the result.

Repository demo

After cloning the repository, use the included presentation fixture so your real skills remain untouched:

export SKILLKEEPER_STATE="$PWD/.skillkeeper-demo"

skillkeeper --state-dir "$SKILLKEEPER_STATE" scan \
  examples/presentation-skill

skillkeeper --state-dir "$SKILLKEEPER_STATE" health \
  examples/presentation-skill/before \
  examples/presentation-skill/replays

Create and validate a repair candidate:

skillkeeper --state-dir "$SKILLKEEPER_STATE" heal \
  examples/presentation-skill/before \
  --train \
    examples/presentation-skill/replays/01-investor-pitch.train.json \
    examples/presentation-skill/replays/02-product-review.train.json \
  --holdout \
    examples/presentation-skill/replays/03-board-update.holdout.json

The result includes a candidate ID. Inspect it without touching the live fixture:

skillkeeper --state-dir "$SKILLKEEPER_STATE" diff CANDIDATE_ID
skillkeeper --state-dir "$SKILLKEEPER_STATE" events --limit 20

Promotion is always explicit:

skillkeeper --state-dir "$SKILLKEEPER_STATE" promote CANDIDATE_ID

The promotion result contains the backup path required by rollback.

Architecture

flowchart LR
    subgraph Inputs["Inputs"]
        Roots["Agent Skill roots"]
        Cases["Training + held-out replays"]
        Traces["Runtime traces<br/>planned adapters"]
    end

    subgraph Observe["Observe"]
        Scan["Scanner + validator"]
        DB[("SQLite inventory")]
        Eval["Replay evaluator"]
        Health["Explainable health"]
    end

    subgraph Improve["Improve safely"]
        Repair["Bounded repair"]
        Stage["Hash-checked candidate"]
        Gate{"Promotion gate"}
    end

    subgraph Operate["Operate"]
        Diff["Readable diff"]
        Live["Live SKILL.md"]
        Backup["Version backup"]
        Audit[("Audit events")]
    end

    Roots --> Scan --> DB
    Cases --> Eval --> Health
    Traces -.-> Eval
    DB --> Health
    Health --> Repair --> Stage --> Gate
    Cases --> Gate
    Gate -->|"passes"| Diff --> Backup --> Live
    Gate -->|"fails"| Reject["Reject with evidence"]
    Backup --> Rollback["Rollback"] --> Live
    Scan --> Audit
    Stage --> Audit
    Live --> Audit

Commands

| Command | Purpose | |---|---| | scan | Discover skills, validate packages, and index dependencies | | inventory | Show the indexed inventory | | evaluate | Run explicit replay requirements against one skill | | health | Explain package validity, replay coverage, and pass rate | | heal | Create and gate a staged candidate | | diff | Show the exact candidate change | | promote | Promote a passing, untampered candidate | | rollback | Restore a named backup | | events | Inspect the local audit trail |

Run skillkeeper COMMAND --help for command-specific arguments.

Safety model

• State is local in SQLite and ordinary files.
• Candidate generation writes to .skillkeeper/staging, never directly to a live skill.
• Training improvement and a held-out threshold are both required.
• Promotion verifies that neither the live baseline nor staged candidate changed after evaluation.
• Every promotion creates a complete rollback copy first.
• Scan, candidate, promotion, and rollback events are recorded.

Keep the state directory outside any scanned skill root.

What v0 proves—and what it does not

The current evaluator verifies explicit textual skill contracts deterministically. That is sufficient to prove the staging, evidence, promotion, tamper-checking, audit, and rollback lifecycle.

It does not claim that keyword checks measure arbitrary agent quality or presentation taste. Behavioral trace adapters, artifact verifiers, and a terminal quality view are the next product layer. See the Roadmap for implementation order.

Documentation

• Research origin and technical lineage
• Demo and receipts
• Roadmap
• Contributing
• Security

Contributing

The most valuable contributions are real, sanitized failure fixtures, deterministic verifiers, skill-root detectors, and runtime adapters. Start with CONTRIBUTING.md and open an issue before building a large adapter.

License

Apache-2.0. See LICENSE.