Kaizen

Kaizen is an agentic eval platform for AI systems. It helps a coding agent create a system definition, curate a Langfuse-backed dataset, write an eval script, record a starting evaluation, and iterate on variants while Kaizen records scored runs under kaizen/.kaizen/runs/.

Install In A Target Repo

pnpm add @percepta/kaizen
pnpm exec kaizen skills install
pnpm exec kaizen init
pnpm exec kaizen create system <system-id>
pnpm exec kaizen create view <system-id> --type trace
pnpm exec kaizen create view <system-id> --type dataset-item
pnpm exec kaizen run \
  --system <system-id> \
  --variant starting-point \
  --diagnostic \
  --title "Establish the baseline" \
  --description "Validates the current system against the configured dataset and metric." \
  --hypothesis "The current system establishes a reproducible starting score."
pnpm exec kaizen studio

kaizen skills install detects Claude Code and Codex, registers the Percepta plugin marketplace bundled with the npm package, and installs the Kaizen plugin for both available clients. The plugin provides:

• Claude Code: /kaizen:guide and /kaizen:autoresearch
• Codex: $kaizen:guide and $kaizen:autoresearch

The command is idempotent and uses each client's native plugin lifecycle for updates, enablement, and removal. After a Kaizen package upgrade, the CLI attempts one short, non-interactive refresh through the client's plugin manager. If that refresh fails, it warns without blocking later commands; rerun kaizen skills install to retry interactively.

Kaizen is installed inside the customer repo. The customer-owned footprint is intentionally small:

• kaizen/config.ts
• kaizen/systems/<system-id>/system.md
• kaizen/systems/<system-id>/eval.py|ts
• optional kaizen/systems/<system-id>/trace.tsx
• optional kaizen/systems/<system-id>/dataset-item.tsx
• optional kaizen/systems/<system-id>/rubric.md
• kaizen/.kaizen/runs/

Package-owned agent guidance lives in the Kaizen plugin. Customer-specific durable notes belong in kaizen/systems/<system-id>/system.md; Kaizen does not create repo-level agent markdown such as KAIZEN.md, AGENTS.md, or CLAUDE.md.

Typed System loading (runtime foundation)

The CLI's loadKaizenSystem(workspaceRoot, systemId) loads the default export of kaizen/systems/<systemId>/system.ts from the supplied candidate checkout. It accepts plain KaizenSystem objects and shared SDK builder exports, including class-based runners and graders. The export's id must match the directory ID. TypeScript imports use the candidate's root tsconfig.json when present.

System definitions and customer-authored TypeScript wrappers must use ES modules. Set "type": "module" in their nearest package.json. This can be scoped to kaizen/package.json when the rest of the checkout uses CommonJS. A System entrypoint without an explicit ESM package scope is rejected before execution. Ordinary package dependencies may use CommonJS, including transitive dependencies; they load through normal module interoperability.

All four Zod schemas are required. Input, output, and reference schemas remain distinct; an absent reference is permitted only by an explicit schema that accepts undefined. Metrics must be an object schema with named numeric fields or classification fields declaring nonnegative integer TP/FP/FN counts. Optional historical metrics, number refinements, and numeric-only unions and intersections are supported; metric types must be statically discoverable, so opaque transforms are rejected. Type discovery ignores metadata annotations and IDs; the original schemas retain them. validateSystemMetricConfig checks comparison overrides without changing the System default, using the shared contract for aggregation parameters.

Loading imports trusted customer code and therefore runs module initialization, but never calls the runner or grader. The caller must select a trusted local checkout: imported code runs with the CLI process's filesystem and environment access. This loader is not a sandbox or a handler for remote paths. System IDs must be a single directory name containing only letters, digits, underscores, and hyphens, starting with a letter or digit; they cannot supply a module path.

Error messages identify the System and field path without including customer exception messages or invalid values. Original exceptions remain available as error.cause for local debugging. Causes and their stacks may contain customer data; API submissions must use the safe error.toJSON() projection, which omits them. system.md and Catalog are not execution configuration. This is the loader foundation for the API-backed runtime; execution and the kaizen run cutover are separate steps. The current run command still uses the legacy eval.

The shared SDK exports RuntimeKaizenSystem for consumers that discover schemas at runtime; authoring still uses KaizenSystem with inferred schema types. The CLI imports this SDK type only. The SDK is currently private and remains a development dependency; publish it and move it to runtime dependencies before adding SDK runtime imports to the CLI.

Dataset loading and Experiment preparation (runtime foundation)

parseDatasetSelection requires an explicit --dataset <datasetId> and accepts --max-items N only as an integer from 1 to 2,147,483,647. loadDatasetSelection consumes the typed Dataset client and a loaded System with an explicit Catalog/System locator. It checks ownership, passes the opaque Dataset hash on every page, and restarts from page one on HASH_MISMATCH (at most three loading attempts). An empty Dataset, malformed pagination, duplicate items, and invalid selected inputs or references are rejected before any runner or grader is called.

The result retains parsed values, exact selected IDs, the original Dataset size, and the unchanged Dataset ID/hash. Missing and null references follow the System's schema; schema defaults and asynchronous transforms are applied. Validation errors identify the System, item, and field path while masking payload keys and omitting values, custom messages, and exception causes.

prepareExperiment uses the shared API target and authentication settings to load/validate the selection and create an API Experiment with that population. The caller supplies the loaded committed System and verified candidate provenance. If creation reports HASH_MISMATCH, preparation reloads and revalidates before trying creation again (at most three creation attempts). Other creation failures are not retried because the write may have committed; inspect API history before starting another attempt. The returned comparison configuration is invocation state for later result reads and is never stored on the Experiment.

New Experiments persist manifest.population: { datasetSize, selectedItemIds }. The API verifies that selection against the hash and item limit under the Dataset lock, and accepts completed Runs only for those IDs even after Dataset edits or removals. Completion describes the selected work; ranking remains a separate API decision. The current rank endpoint excludes all Experiments with an item limit. The population migration is additive; older Experiments omit this field because their original selection cannot reliably be reconstructed from current data.

These modules ship in the CLI's dist for the staged runtime implementation. KZN-171 supplies execution, KZN-173 completed-Run reporting/finalization, and KZN-176 wires the typed kaizen run command. Until that cutover, the legacy command continues using its existing Dataset configuration; these new helpers do not claim execution or completion.

Manage Datasets Through the Kaizen API

kaizen datasets --help lists the Dataset creation, import, inspection, labeling, editing, move, and removal commands. They work from any directory against either a local or hosted Kaizen service. See the Dataset workflow for JSON shapes, normalization examples, and pagination.

export KAIZEN_API_URL=http://localhost:3000
# Set KAIZEN_API_KEY securely to match the service's API_KEY.
pnpm exec kaizen datasets list --json
pnpm exec kaizen datasets create --file dataset.json --json
pnpm exec kaizen datasets items import --dataset <id> --file items.jsonl --format jsonl --json
pnpm exec kaizen datasets items label <item-id> --file label.json --json
pnpm exec kaizen datasets items move <item-id> --dataset <training-id> --to-dataset <validation-id> --json

--api-url overrides KAIZEN_API_URL. URLs are application origins, without /api/dataset-service; remote origins require HTTPS, while loopback HTTP works locally. There is no implicit default or fallback installation. KAIZEN_API_KEY is required, sent as x-api-key, and must match the selected service's API_KEY. The client refuses redirects. API commands optionally read .env and .env.local from KAIZEN_WORKSPACE, or the current directory when it is unset. .env.local overrides .env; exported environment variables override both files, and --api-url takes precedence over all of them. No workspace configuration is required, and customer environment scripts are not executed.

Reads and writes support --json, with errors on stderr and a nonzero exit status. Dataset list returns the API Page envelope; item list also includes the Dataset, its System owner, and its opaque hash. Each invocation fetches one page (default 50, maximum 500). --catalog and --system filter Dataset lists and must be supplied together. Dataset locators are IDs unless --slug is supplied.

Dataset labels use reference for System-defined ground truth or grading criteria. Imports do not load customer schemas or execute customer code. An agent or script normalizes source data before submission. Existing run, log, and legacy studio retain their current execution/storage behavior.

Inspect Experiments and Runs From the Kaizen API

These commands use the same API target and credentials as Dataset listing. They work from a fresh customer checkout or any directory without local Experiment history, System code, or Catalog access.

pnpm exec kaizen experiments list --catalog <catalog-id> --system <system-id> --json
pnpm exec kaizen experiments list --catalog <catalog-id> --system <system-id> --dataset-hash <hash> --page 2 --page-size 20 --json
pnpm exec kaizen experiments get <experiment-id> --json
pnpm exec kaizen runs list --experiment <experiment-id> --page 1 --page-size 20 --json

Experiment history preserves hypotheses, candidate Git commits, parent IDs, Dataset identity/hash, execution status, and recorded progress. nDone/nTotal describes completed versus selected items; maxItems records a diagnostic limit. New Experiments also include manifest.population with the original full Dataset size and exact selected IDs; older records omit it. A parent ID remains present when that parent is outside the current page or Dataset hash. Per-item Runs expose experimentId, itemId, exact stored output, metrics, and an optional opaque traceId.

List commands return one complete Page envelope with items, page, pageSize, total, and totalPages. --page defaults to 1; --page-size defaults to 50 and is bounded to 500. get returns one record. JSON goes to stdout, and errors go to stderr with a nonzero exit status. Without --json, lists show escaped table cells and details use indented JSON.

Reads return stored data by default. get calculates a result in the API against saved metrics when you supply all three selection flags; rank requires them. list always returns stored records.

pnpm exec kaizen experiments get <experiment-id> --metric accuracy --metric-type number --aggregation mean --json
pnpm exec kaizen experiments get <experiment-id> --metric classifications --metric-type classification --aggregation f1 --json
pnpm exec kaizen experiments get <experiment-id> --metric latencyMs --metric-type number --aggregation percentile --percentile 95 --json

Numeric aggregations are mean, median, and percentile (requiring --percentile between 0 and 100). Classification aggregations are precision, recall, f0.5, f1, and f2. Invalid or incomplete selections fail before any request. A selected read adds primaryMetricConfig and a result containing either { status: "available", value } or { status: "unavailable", reason }. API calculation failures retain their structured details and the saved record; unfinished evaluations report experimentNotComplete. An unavailable result is a successful inspection, distinct from a measured zero and execution status.

experiments rank calls the API's ranked query with one System, Dataset hash, and comparison selection. It preserves the API's ordering and pagination; ordinary list retains limited, failed, unfinished, and unscorable evaluations. If a ranked response reports unfinished or limited records, incomplete progress, or missing results, inspection fails without printing or filtering the page.

pnpm exec kaizen experiments rank --catalog <catalog-id> --system <system-id> --dataset-hash <hash> --metric accuracy --metric-type number --aggregation mean --json

These commands use the Experiment API's experiments and runs endpoints. Experiment events and failure journals have a separate CLI delivery ticket (KZN-230).

Inspect Trace Evidence

Trace commands use the same explicit local or hosted API target and KAIZEN_API_KEY. They read through Kaizen's authenticated TraceStore; Langfuse credentials stay on the service. No local history, System code, or Experiment is required to look up a production Trace.

pnpm exec kaizen traces get '<opaque-trace-id>' --json
pnpm exec kaizen traces list --limit 20 --json
pnpm exec kaizen traces list --limit 20 --cursor '<nextCursor>' --json
pnpm exec kaizen experiments traces <experiment-id> --page 1 --page-size 20 --json

traces get returns the common Trace record, including available input, output, metadata, and spans/tool calls. IDs are opaque strings; quote shell special characters. traces list returns { items, nextCursor? } with summary records. Pass nextCursor unchanged to continue, keeping the same target and limit; its absence marks the last page. The limit defaults to 50 and is bounded to 100. Both commands print indented JSON by default and compact JSON with --json.

experiments traces returns a Page envelope of { experimentId, itemId, traceId } references derived from persisted Runs. Pagination counts only Runs with Trace references and retains separate rows when items share a Trace. It does not contact the provider or check evidence availability. Fetch a reference using traces get; a missing Trace does not remove its reference or saved Run output and metrics. Empty reference pages succeed; a missing Experiment fails.

Missing evidence, provider failures, access denials, and malformed responses exit nonzero, print a safe diagnostic to stderr, and leave stdout empty. Raw provider messages, payloads, and credentials are never included in diagnostics, including with KAIZEN_DEBUG. Successful JSON output contains the requested evidence and can be consumed by customer scripts.

For an Online Evals failure import, use the online record's original traceId with traces get --json, combine its input/output with the online record's current reference, and transform the case into the destination System's schema before using datasets items import. Preserve the original source.traceId and stable source identity. Imports and repeat-import deduplication remain explicit customer-script operations; Trace inspection does not create items.

Lifecycle

1. Run kaizen init once in the target repo.
2. Run kaizen create system <system-id> and fill in kaizen/systems/<system-id>/system.md.
3. Use Studio Data to create or select a Langfuse dataset, add useful source traces, and label dataset items.
4. Replace kaizen/systems/<system-id>/eval.py|ts with a real eval that reads the dataset named by dataset_version.
5. Run a diagnostic starting point, then a full starting run.
6. Set the full run id as best_run in system.md.
7. Run variants with kaizen run, inspect kaizen log, and use Studio to compare runs and failures.

The eval script emits NDJSON events to --out-fd; the runner owns process supervision, kaizen/.kaizen/runs/, and crash recording. kaizen run records the current Git commit in every new run and refuses to start when staged, unstaged, or untracked changes would make that commit an incomplete description of the evaluated source. Commit the candidate before invoking kaizen run. Baselines committed on the repository's main branch are valid. Kaizen records results without automatically selecting a winner. The explicit best_run in system.md identifies the current reference, and diagnostics can never be the best run. For Langfuse-backed evals, the eval should also link each dataset item to the fresh trace generated by that run and write the primary metric as a trace score.

While an eval is active, Kaizen monitors the worktree and fails the run if its commit or file status changes. Because items scored around a detected edit may be ambiguous, that run becomes restart-only. Restore its original commit and use kaizen resume <run-id> --restart to discard the journal and try again. Ordinary resumes and restarts both require a clean worktree at the run's original commit. Legacy runs without commit provenance remain visible in Studio, but cannot be resumed safely.

The runner persists runs through the asynchronous RunStore interface. The CLI uses FilesystemRunStore by default, preserving the existing kaizen/.kaizen layout. A future service-backed store can send the same run, event, progress, restart, and finalization operations to the hosted Kaizen API without changing the evaluation lifecycle. The service may use its own RunRepository backed by Postgres; that database boundary is deliberately not exposed to the CLI. System definitions, local process locks, and eval log files remain outside run storage.

Every new experiment should include a concise --title, a longer --description stating exactly what the run validates, and a --hypothesis describing the expected outcome and why. These flags are optional so existing kaizen run automation remains compatible; Studio falls back to the variant name when no title is present.

Custom Views

Custom views are plain React components co-located with the system:

kaizen create view <system-id> --type trace
kaizen create view <system-id> --type dataset-item

trace.tsx receives the full Langfuse trace payload plus actions for writing scores. dataset-item.tsx receives the dataset item, the linked source trace when available, and actions for updating the dataset item or linking run items. Browser-side credentials are not required; Studio proxies the write actions through local API routes.

Use the kaizen:guide plugin skill for the exact prop and action interfaces.

Run The Kaizen Service Locally

This package ships compose.yml, which runs the hosted Kaizen service as a pulled image (ghcr.io/percepta-core/kaizen-service) alongside its own Postgres. A customer monorepo includes it from its local docker-compose.yml so developers can run Kaizen against their own portal stack and catalog without a mosaic checkout:

include:
  - path: node_modules/@percepta/kaizen/compose.yml

The fragment assumes the including stack runs the portal as a service named portal, published on http://localhost:3001 (the Mosaic monorepo template does). Two pieces of wiring live on the including side:

• Trusted origin. Add http://localhost:3002 (or your KAIZEN_PORT) to the portal's BETTER_AUTH_TRUSTED_ORIGINS (portal.local.env in the template) or the post-login callback is rejected.
• Application registration. Customer admins reach Kaizen through the core/application:kaizen_service#customer link. Add - appNamespace: "kaizen_service" under applications in access/reconcile.yaml and run pnpm access:reconcile, or grant users directly from the portal's access-management UI.

Point the service at your catalog with KAIZEN_CATALOG_URLS (comma-separated oRPC endpoints, fetched from inside the container — a catalog on the host is http://host.docker.internal:<port>/api/catalog, which is the default on :3100). Datasets need LANGFUSE_BASE_URL, LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY exported in the shell; they pass through when set.

| Variable | Default | | ---------------------------- | ---------------------------------------------- | | KAIZEN_IMAGE_TAG | latest (tags mirror [email protected]) | | KAIZEN_PORT | 3002 | | KAIZEN_POSTGRES_PORT | 5436 | | KAIZEN_PORTAL_BASE_URL | http://localhost:3001 | | KAIZEN_PORTAL_INTERNAL_URL | http://portal:3000 | | KAIZEN_CATALOG_URLS | http://host.docker.internal:3100/api/catalog | | KAIZEN_API_KEY | kaizen-local-api-key |

The image lives in a private GitHub Container Registry, so docker login ghcr.io once with a token that has read:packages (the same login the portal image needs).

Developing This Repo

pnpm install
pnpm --filter @percepta/kaizen-contract build
pnpm --filter @percepta/kaizen dev:studio

This starts Studio at http://localhost:6789 against examples/demo-workspace, a local fixture for package development. The CLI lives in src/; the bundled Next.js Studio lives in dashboard/.

The CLI owns its API request helper. @percepta/kaizen-contract is a private development dependency whose used schemas, including RankedExperiment, are included under dist in the published CLI. Consumers do not need a separate contract package release or installation. @orpc/client remains a runtime dependency so the transport is installed with the published CLI, rather than emitted into a nested node_modules directory that package publishing would exclude.

Useful scripts:

| Script | What it does | | ------------------------------------------- | ---------------------------------- | | pnpm --filter @percepta/kaizen dev:studio | Start Studio with the demo fixture | | pnpm --filter @percepta/kaizen dev:next | Start only the Next.js dev server | | pnpm --filter @percepta/kaizen typecheck | Typecheck the package | | pnpm --filter @percepta/kaizen test | Run package tests |

Environment

API commands use KAIZEN_API_URL and KAIZEN_API_KEY as described above. See .env.example for their configuration. These variables are separate from the existing local Studio configuration below.

Create .env.local in the workspace repo root:

LANGFUSE_HOST=https://...
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LINEAR_API_KEY=lin_api_...

Langfuse credentials power the Data surface and custom view actions. LINEAR_API_KEY lets the Data surface load the current Linear user and project members for dataset-item assignment.

To populate the assignee list, configure a stable Linear project URL or ID in system.md:

linear_project: https://linear.app/<workspace>/project/<project-slug>

Publishing

Publishing @percepta/kaizen to npm is automated with Changesets. For changes that affect the published package, add a changeset:

pnpm changeset