@rohal12/spindle-playtest

Plays Spindle stories many times, asking a System One decision model which action a player would take at every turn. It then prints a summary: the endings reached, dead ends, errors, and the choices the model was least sure about. Low confidence usually means the wording doesn't separate the options well.

Two models are supported:

• Jev: TypeSafe's hosted model.
• Kev: an open-source model you host yourself (jaredpalmer/kev).

See docs/design.md for the design; it's also in the package.

Install

It needs Node 22 or later.

npm install --save-dev @rohal12/spindle-playtest

Run it with npx spindle-playtest. It builds on @rohal12/spindle/play, which compiles .twee stories with the installed Spindle format.

Usage

spindle-playtest [story] [--config file] [--runs n] [--max-turns n] [--seed n]

• story is a path in any form Spindle accepts: a .twee file, a glob, a project directory or a compiled .html file. When it's left out, the config's story is used. A story given on the command line is relative to the working directory; one in the config file is relative to the config file's directory.
• The config file defaults to spindle-playtest.json in the working directory, if it exists.
• A flag overrides the matching config field.
• --help (or -h) prints the usage line.

The summary goes to stdout. A progress counter (run 7/20) goes to stderr, and only when stderr is a terminal.

The exit code is 0 when the command finished, even when runs hit dead ends, errors or the turn cap. It is 1 for a config problem, a story that can't be resolved, a decision server that can't be reached, or an API error that stops the command (such as a rejected API key).

With Jev

Set your API key and run:

export SPINDLE_PLAYTEST_API_KEY=...
npx spindle-playtest src/story

provider defaults to "jev", so no config file is needed.

With Kev, running locally

Start a Kev server:

git clone https://github.com/jaredpalmer/kev.git
cd kev
uv sync --extra serve
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b --port 8009

Then point the playtest at it:

{ "story": "src/story", "provider": "kev" }

Kev listens on http://127.0.0.1:8009 by default, which is the kev provider's baseUrl. If the server sets KEV_API_KEY, set SPINDLE_PLAYTEST_API_KEY to the same value; otherwise leave it unset and no Authorization header is sent.

Config

{
  "story": "src/story",
  "provider": "kev",
  "baseUrl": "http://127.0.0.1:8009",
  "model": "kev-latest",
  "runs": 20,
  "maxTurns": 100,
  "parallel": 4,
  "historyLimit": 5,
  "instructions": "You are a first-time player. Pick what this player would most likely do next.",
  "endings": ["Epilogue"],
  "inputs": { "$name": ["Alice", ""], "$age": [7, 30] },
  "seed": 1
}

Every field is optional except story, which can also come from the command line.

| Field | Default | Meaning | | ------------------ | ------------------------------ | ------------------------------------------------------------- | | story | none (required) | The story to play | | provider | "jev" | "jev" or "kev"; sets the defaults below | | baseUrl, model | from provider | Where the decision API is, and which model to ask | | runs | 20 | Number of playthroughs | | maxTurns | 100 | Steps per run before it's stopped as "turn cap" | | parallel | 4 | Runs at once, each in its own play session | | historyLimit | from provider | Transcript entries put in state | | instructions | the sentence in the example | The Choice question's instructions | | endings | [] | Passages that count as endings, besides those tagged ending | | inputs | {} | Candidate values for free-text inputs, keyed by variable | | seed | random, printed in the summary | Seed for sampling |

Provider defaults:

| Provider | baseUrl | model | historyLimit | | -------- | ------------------------- | ------------ | -------------- | | jev | https://api.typesafe.ai | jev-latest | 10 | | kev | http://127.0.0.1:8009 | kev-latest | 5 |

• API key: read from the SPINDLE_PLAYTEST_API_KEY environment variable, never from the config. An apiKey field in the config file is an error.
• Validation: an unknown field, a wrong type, a baseUrl that isn't an http(s) URL, runs, maxTurns or parallel below 1, or historyLimit below 0 stops the CLI with exit code 1 and a message naming the field.
• Inputs: a key with or without the $ ("name" or "$name") means the same variable; giving both is an error. Candidates are strings, numbers or booleans. Each time a run meets a text input, each candidate that differs from the current value is an option, named like $name = "Alice" or $age = 30.
• Seed: the same seed gives the same sampling. Run i uses seed + i.

Reading the summary

spindle-playtest: 20 runs, kev (kev-latest), seed 1, 1,412 decisions, 512k input tokens

Endings        Epilogue 12 · Drowned 5
Dead ends      Cellar 3
Errors         Market: $price is undefined (2 runs)
Unused inputs  $nickname (textbox in Tavern)

Least confident choices (average confidence, at least 2 visits)
  0.31  Crossroads   "Go north" 0.38 · "Take the high road" 0.36 · "Rest" 0.26
  0.44  Tavern       "Ask about the map" 0.51 · "Buy a drink" 0.49

• Endings are passages tagged ending or listed in endings. A run ends as soon as it reaches one, even if the passage offers actions such as "Play again". Dead ends have no actions and aren't endings. Turn cap counts runs stopped by maxTurns.
• Errors are grouped by passage and message. An error shown on screen doesn't end the run, but is counted here.
• Unused inputs are inputs the story showed that had no candidate values in inputs.
• A line with nothing to report is left out, except Endings, which says none.

The least-confident list

Choices are grouped by passage and option set, and a group needs at least two visits. Each row shows the average confidence, then the average probability the model gave each option. It lists the 10 lowest.

• Probabilities that are close to equal (0.38 · 0.36 · 0.26 above) mean the model couldn't tell the options apart. The wording of the links, or the text before them, may not say what each leads to.
• One option far ahead of the rest, with low confidence overall, means the model is guessing even though it leans one way.
• A choice only shows up after two visits, so raise runs to see rare ones.

Low confidence is a hint, not a verdict: a choice that is meant to be a coin flip will always look uncertain.

Limits

• No free text. Jev and Kev only choose. A text input is played from the candidates in inputs; without candidates, it's reported as an unused input.
• 255 options. A request carries at most 255 options. A passage with more ends the run as an error (too many choices in <passage>).
• Kev's validated length. Kev's 0.8B, 4B and 9B models are only validated up to 8,192 tokens of state. Keep historyLimit low and passages short. Kev answers 422 for a state longer than 65,536 tokens. A 413, a 400 max_tokens_exceeded, or a 422 whose body mentions the token limit stops the command with state too long for <model>; lower historyLimit.
• Retries. 429, 529, 5xx, requests with no answer within 60 seconds, and other network errors (such as a dropped connection) are retried up to 5 times, after 0.5, 1, 2, 4 and 8 seconds. When retries run out, that run ends as an error.
• Unreachable server. A refused connection or an unknown host stops the command at once with cannot reach <baseUrl> (<code>); is the server running?.

Library use

import {
  playtest,
  renderSummary,
  resolveConfig,
} from '@rohal12/spindle-playtest';

const config = resolveConfig({ story: 'src/story' }, {}, process.env, () => 1);
console.log(renderSummary(await playtest(config)));

Development

npm run build && npm run typecheck && npm test && npm run format:check
npm run duplication

npm run duplication runs scripts/duplication.ts directly with node, which needs Node 22.18 or later (type stripping). Its CPD step needs Java and PMD (set PMD_BIN to PMD's pmd script, or put it on PATH); without them it's skipped locally, and CI runs it.

The CLI tests build the package and run dist/cli.js against a fake System One server. To also run the live test against a real Kev server, set KEV_URL:

KEV_URL=http://127.0.0.1:8009 npm test