uinaf/skillcheck

Lint and eval harness for agent skills. One CLI with two halves: a keyless structural lint any repo can run in CI, and a promptfoo-driven eval loop that grades what a skill actually makes an agent do.

Built for uinaf skill repos. Nothing in it is uinaf-specific. It ships no opinion about what a skill should say, only about where skills sit and how a scenario is scored.

Install

pnpm add -D @uinaf/skillcheck

Node 24 or newer. The package ships compiled ESM, runs no install scripts, and has no regular dependencies, so a runner using --ignore-scripts is fine and the lint-only install stays at a handful of packages. The promptfoo eval engine and provider SDKs are optional peers, installed only on the operator machine that runs evals (adoption). Consumers still pinned to the pre-npm git tags are covered there too.

Use

skillcheck lint                              # structural lint, no credentials
skillcheck run skills/wat/evals/basic        # one scenario, graded end to end
skillcheck sweep --trials 3                  # every scenario, three trials each
skillcheck summarize                         # per-scenario and per-skill scorecard

lint needs nothing. run and sweep need model auth, which is why sweeps stay operator-run and consumer repos never hold credentials. Every command, flag, and auth variable is in usage.

Layout contract

Frozen, not configurable. Every command reads one root: --root <dir>, or the current directory.

<root>/skills/<skill>/SKILL.md                       linted
<root>/skills/<skill>/evals/<scenario>/task.md       the problem + input files
<root>/skills/<skill>/evals/<scenario>/criteria.json the weighted checklist
<root>/.skillcheck/                                  results, scorecards

cli/*/skills/<skill>/ is scanned too, for repos that keep a skill next to the CLI it documents.

Docs

| Doc | When | | --------------------------------------------------------------- | ------------------------------------- | | Usage | Every subcommand, flag, and auth path | | Scenarios | Writing an eval scenario | | Authoring | Writing and auditing the skill itself | | Adoption | Wiring the lint into another repo | | Releasing | The npm pipeline | | Contributing | Local setup and the verify gate | | Security | Reporting a vulnerability |

License

MIT · undefined is not a function LLC