skilltune

A CLI tool that optimizes Claude Code skill description fields to improve trigger accuracy.

Overview

Claude Code skills are invoked based on their description field. skilltune automatically improves descriptions through the following loop:

Generate queries → Evaluate (measure trigger rate) → Propose new description → Repeat

Installation

npm install -g skilltune

Requirements

• Claude Code (CLAUDE_CODE_OAUTH_TOKEN environment variable must be set)
• Node.js 18+

How evaluation works

Each evaluation run creates an isolated environment under .skilltune/eval-XXXXX/ in the current directory, copies only the target SKILL.md into it, then runs Claude with cwd pointing there. This ensures the project's CLAUDE.md is never loaded during evaluation, keeping trigger rate measurements clean.

.skilltune/          # auto-created, gitignored
  eval-XXXXX/        # temporary, deleted after each evaluation
    .claude/
      skills/
        <name>/
          SKILL.md

The skill itself can live anywhere — .claude/skills/, a custom path, or an absolute path.

Usage

Optimize a skill directory (recommended)

Pass a skill directory path directly to run the full pipeline — query generation through optimization — in one shot. An error is raised if SKILL.md does not exist in the specified directory.

skilltune .claude/skills/git-commit
skilltune /absolute/path/to/my-skill

Generate queries

Automatically generate evaluation queries from a skill using Claude.

skilltune generate-queries --skill .claude/skills/git-commit --output queries.json
# or use a short name (resolves to .claude/skills/<name>)
skilltune generate-queries --skill git-commit --output queries.json

| Option | Default | Description | |--------|---------|-------------| | --skill | (required) | Path to the skill directory, or a skill name (resolves to .claude/skills/<name>) | | --count | 20 | Number of queries to generate (half should_trigger:true, half false) | | --output | queries.json | Output file path |

Evaluate only

Measure trigger rates against an existing query file.

skilltune eval --skill .claude/skills/git-commit --queries queries.json

Outputs:

• Positive rate: fraction of should_trigger: true queries that actually triggered
• Misuse rate: fraction of should_trigger: false queries that incorrectly triggered
• Failed indices: query indices where the result did not match expectations

| Option | Default | Description | |--------|---------|-------------| | --skill | (required) | Path to the skill directory, or a skill name | | --queries | queries.json | Path to the query file | | --runs | 3 | Number of runs per query |

Optimization loop

Iterates evaluate → propose description → evaluate, writing the best description back to the skill file.

# Auto-generate queries and optimize
skilltune optimize --skill .claude/skills/git-commit

# Use an existing query file
skilltune optimize --skill .claude/skills/git-commit --queries queries.json

| Option | Default | Description | |--------|---------|-------------| | --skill | (required) | Path to the skill directory, or a skill name | | --queries | (optional) | Query file path; auto-generated if omitted | | --runs | 3 | Number of runs per query | | --max-iterations | 5 | Maximum number of optimization iterations | | --train-ratio | 0.6 | Train/validation split ratio | | --count | 20 | Number of queries to generate when --queries is omitted | | --patience | 3 | Early stopping: halt if validation does not improve for this many iterations |

Query file format

[
  { "query": "A prompt that should trigger the skill", "should_trigger": true },
  { "query": "A prompt that should not trigger the skill", "should_trigger": false }
]

Architecture

Structured following Feature-Sliced Design (FSD).

src/
  app/                        # Entry point (gunshi CLI)
  features/
    evaluate/                 # Trigger rate evaluation
    generate-queries/         # Automated query generation via Claude Agent SDK
    optimize/                 # Description optimization loop
  entities/
    query/                    # Query type + train/validation split
    result/                   # QueryResult type + aggregation
    skill/                    # Skill file read/write, parsing, and path resolution
  shared/
    claude/                   # Claude Agent SDK wrapper
    lib/                      # General utilities