Kradle CLI

Kradle's CLI for managing Minecraft challenges, evaluations, agents, and more!

• Installation
• Autocomplete
• Configuration
• Challenge
• Evaluations
• Publishing a New Version
• Development
• Architecture

Installation

1. Install Kradle's CLI globally

npm i -g @kradle/cli

2. Initialize a new directory to store challenges and evaluations

kradle init

3. Congrats 🎉 You can now create a new challenge or a new evaluation:

kradle challenge create <challenge-name>
kradle evaluation create <evaluation-name>

In addition, you can enable autocomplete.

Autocomplete

Kradle CLI supports shell autocomplete for faster command entry. After installation, enable autocomplete for your shell:

kradle autocomplete
# Follow the instructions printed

The command will display instructions for your specific shell.

After setup, you will be able to use Tab to autocomplete:

kradle challenge <TAB>        # Shows: build, create, list, run, upload, watch, etc.

Challenge Commands

Create Challenge

Create a new challenge locally and in the cloud:

kradle challenge create <challenge-name>

This creates a challenges/<challenge-name>/ folder with:

• challenge.ts: The entrypoint defining challenge behavior
• config.ts: TypeScript file with challenge metadata (auto-generated from cloud API)

Build Challenge

Build challenge datapack and upload both config and datapack:

kradle challenge build <challenge-name>

This command:

1. Creates the challenge in the cloud (if it doesn't already exists)
2. Uploads config.ts to cloud (if it exists)
3. Builds the datapack by executing challenge.ts
4. Uploads the datapack to GCS

Delete Challenge

Delete a challenge locally, from the cloud, or both:

# Will ask confirmation for local & cloud deletion
kradle challenge delete <challenge-name>

# Doesn't ask for confirmation
kradle challenge delete <challenge-name> --yes

List Challenges

List all challenges (local and cloud):

kradle challenge list

Watch Challenge

Watch a challenge for changes and auto-rebuild/upload:

kradle challenge watch <challenge-name>

Uses file watching with debouncing (300ms) and hash comparison to minimize unnecessary rebuilds.

Run Challenge

Run a challenge in production or studio environment:

kradle challenge run <challenge-name>
kradle challenge run <challenge-name> --studio  # Run in local studio environment

Multi-Upload

Interactively select and upload multiple challenges:

kradle challenge multi-upload

Provides an interactive UI to select multiple challenges and uploads them in parallel.

Evaluation Commands

Evaluations allow you to run batches of challenge runs with different agents and configurations, then analyze the results. This is useful for benchmarking agents, testing challenge difficulty, or gathering statistics across many runs.

Concepts

Evaluation: A named collection of run configurations defined in a config.ts file. Each evaluation lives in evaluations/<name>/.

Iteration: A snapshot of an evaluation execution. When you run an evaluation, it creates an iteration containing:

• A copy of the config.ts at that point in time
• A manifest.json with the generated list of runs
• A progress.json tracking the status of each run

Iterations are stored in evaluations/<name>/iterations/001/, 002/, etc. This allows you to:

• Resume an interrupted evaluation from where it left off
• Re-run the same evaluation with --new to create a fresh iteration
• Compare results across different iterations

Create Evaluation

Create a new evaluation with a template config file:

kradle evaluation create <name>

This creates evaluations/<name>/config.ts with a template that you can customize. The config exports a main() function that returns a manifest with:

• runs: Array of run configurations (challenge + participants)
• tags: Optional tags applied to all runs for filtering in analytics

Run Evaluation

Execute or resume an evaluation:

kradle evaluation run <name>                  # Resume current iteration or create first one
kradle evaluation run <name> --new            # Start a new iteration
kradle evaluation run <name> --max-concurrent 10  # Control parallelism (default: 5)

The run command:

1. Creates a new iteration (or resumes the current one)
2. Generates a manifest by executing config.ts
3. Displays an interactive TUI showing run progress
4. Saves progress periodically (allows resuming if interrupted)
5. Opens Metabase dashboard with results when complete

List Evaluations

List all local evaluations:

kradle evaluation list

Publishing a New Version

The CLI uses GitHub Actions for automated releases. To publish a new version:

1. Go to Actions in the GitHub repository
2. Select "Create Release PR" workflow from the sidebar
3. Click "Run workflow" and choose the release type:
   • patch - Bug fixes (0.0.5 → 0.0.6)
   • minor - New features (0.0.5 → 0.1.0)
   • major - Breaking changes (0.0.5 → 1.0.0)
4. Review and merge the automatically created PR
5. Done! The package is automatically published to npm when the PR is merged

Development

Setup

This CLI requires linking to be used locally:

npm install
npm run build
npm link

kradle vs kradle-dev

The repository provides two CLI commands:

• kradle: Production CLI that runs compiled JavaScript from dist/

  • Requires npm run build after every code change
  • This is what end users will use

• kradle-dev: Development CLI that runs TypeScript directly

  • No build step required
  • Changes are reflected immediately
  • Always use this during development

Example usage:

kradle-dev challenge list
kradle-dev challenge build <challenge-name>
kradle-dev challenge run <challenge-name>

Build & Lint

npm run build              # Compile TypeScript to dist/
npm run lint               # Check for linting issues
npm run lint:fix           # Auto-fix linting issues
npm run format             # Format code with Biome

Challenge Structure

Each challenge is a folder in challenges/<slug>/ containing:

• challenge.ts: Entrypoint that defines challenge behavior using the Sandstone API
• config.ts: TypeScript file exporting challenge metadata (name, visibility, roles, objectives, etc.)

Workflow:

1. kradle challenge create <slug> creates the folder with challenge.ts
2. The create command automatically builds, uploads, and downloads the config from the cloud API
3. The downloaded JSON is converted into a typed TypeScript config.ts file
4. kradle challenge build <slug> automatically uploads config.ts (if it exists) before building the datapack
5. You can modify config.ts locally and run build to sync changes to the cloud

Configuration Note

The CLI relies on a .env file in the parent directory (kradle-sandstone root), not in the kradle-cli directory itself. The .env file should be at the same level as both kradle-cli/ and challenges/ folders.

Architecture

The CLI is built with:

• oclif: CLI framework
• enquirer: Interactive prompts
• listr2: Task list UI
• picocolors: Terminal colors
• zod: Schema validation
• chokidar: File watching
• biome: Linting and formatting

Project Structure

kradle-cli/
├── src/
│   ├── commands/             # CLI commands
│   │   └── challenge/        # Challenge management commands
│   │       ├── build.ts      # Build & upload datapack + config
│   │       ├── create.ts     # Create new challenge
│   │       ├── list.ts       # List local & cloud challenges
│   │       ├── multi-upload.ts  # Interactive multi-select upload
│   │       ├── run.ts        # Run challenge (prod or studio)
│   │       ├── watch.ts      # Watch for changes & auto-rebuild
│   └── lib/                  # Core libraries
│       ├── api-client.ts     # Typed API client with Zod validation
│       ├── arguments.ts      # Shared CLI arguments with autocomplete
│       ├── challenge.ts      # Challenge class (build, upload, hash, config)
│       ├── config.ts         # Environment config with Zod schemas
│       ├── schemas.ts        # Zod schemas for type-safe API interactions
│       └── utils.ts          # Utility functions
├── static/                   # Template files (challenge.ts)
├── biome.json               # Biome linter & formatter config
├── package.json
└── tsconfig.json            # TypeScript ESM configuration