ravensight-playtest

AI personas play your game, then file a report you can read, mark up and argue with. The personas run on your machine, against your build; Ravensight stores the reports and renders them.

You need a Ravensight account and Node 22 or newer. You do not need an Anthropic API key: model calls go through Ravensight under the same token you log in with, so there is one credential on your machine and you can revoke it from the dashboard.

Install

Run it without installing anything:

npx ravensight-playtest login

Or install it globally:

npm install -g ravensight-playtest
ravensight-playtest login

Nothing is downloaded until you need it. Playwright is an optional dependency, so login, check, init and brief work without a browser installed; run --driver playwright_web is what asks for one. keytar is optional too, which is why a machine with no keychain still installs cleanly.

First run

ravensight-playtest login                  # device code, approved in the browser
cd path/to/your/game
ravensight-playtest init --game <gameId>   # links this repo
ravensight-playtest check                  # one row per setup step, reserves nothing
ravensight-playtest brief                  # four questions, two minutes
ravensight-playtest run                    # prints the price, asks, then plays

login stores the token in your OS keychain (macOS Keychain, Windows Credential Manager, libsecret on Linux). With no keychain available it falls back to a file with 0600 permissions under your home directory, and says so. In CI, set RAVENSIGHT_TOKEN instead and skip login entirely.

Upgrading to 0.3.2

• A plain run plays one persona, not two. The default persona set is now curious-kid alone, so a first run costs one persona run. Name more with --personas curious-kid,grumpy-veteran (or personas in .ravensight/config.json); ravensight-playtest personas lists the slugs. A script that relied on two personas by default has to name them.
• run checks the build before it prices anything. A web build's URL has to answer 2xx or 3xx, or the preflight URL in .ravensight/config.json has to answer what you set (see run). A failure exits 1 with no job registered. --skip-preflight skips it, as it already skipped the native probe.
• brief and check quote the run that run would start: its modules and its personas, the default one when you name none, each unit with its own price. They used to quote a game profile with no personas.
• Two new brief fields stay on this machine: expected_noise and test_account (see the brief --from file). init adds .ravensight/brief.local.json to .gitignore.
• Out of range report values are normalised before upload instead of failing the run (see how the report step reads what happened, under run).
• A CSS selector is no longer a target. A target is a ref, coordinates, a line copied from the screen, or text on the screen; anything else is refused at once instead of searched for.

Commands

Thirteen commands. Flags below are the ones the code accepts, not the ones the prose elsewhere describes; --help on any command prints the same list.

Every command takes these four:

| Flag | Effect | |---|---| | --api-url <url> | Point at another Ravensight deployment. Defaults to RAVENSIGHT_API_URL. | | --repo <dir> | Work in a different directory. | | --json | Machine readable output, and nothing else on stdout. Stable for CI. | | -y, --yes | Answer every confirmation with yes. Required for CI, since a prompt with no terminal is treated as no. |

login

Device code flow. Prints a code, waits for you to approve it in the dashboard, stores the token.

| Flag | Effect | |---|---| | --no-browser | Do not try to open a browser. RAVENSIGHT_NO_BROWSER=1 does the same thing. |

The token goes in your OS keychain (macOS Keychain, Windows Credential Manager, libsecret on Linux). With no keychain available it falls back to a file with 0600 permissions under RAVENSIGHT_HOME, and says which store it used. In CI, set RAVENSIGHT_TOKEN and skip login entirely.

logout

Revokes the token on the server, then forgets it locally. No flags of its own. A token from RAVENSIGHT_TOKEN is left alone: it is probably a CI secret shared with other jobs.

check

The doctor. Each setup question is its own row, so a problem in one is never reported as another: node, api reachable, cli version (the installed version against the server's latest and min_supported), credential store, login (a token is present, the server accepts it, and which games it is scoped to), project binding (the game and environment this directory is linked to, and whether that game is in the token's scope), brief (present, complete, and which of goals, success criteria, controls and how to start are empty), build (the URL, Godot project or native build a run would use), automation server (native drivers only, see below), then the content pack verified by sha256, the balance against the price of the run run would start (its modules and personas, the default persona when none is named, each unit with its price), disk, recommended concurrency, and the two optional tools. A --personas slug the pack does not have fails a personas row that lists the valid slugs, and nothing is priced.

For a web build linked by URL (--build-url is a run flag; check reads the URL from .ravensight/config.json), the browser row also loads the first screen headless and a first screen row measures it the way a persona would read it. When the whole text is over the 4,000 characters a persona reads per step, or the screen has a list of 50 or more look-alike items, the row is a warning that suggests run --long with the reason ("first screen is 18,400 characters, about 4.6x the 4,000 a persona reads per step") and both prices, read from the server's free estimate. A light screen passes with its size. A first screen that does not load within 20 seconds is a warning, never a failure. The page is marked as a playtest before the game's scripts run (the same window.__RAVENSIGHT_PLAYTEST__ a run sets, with the token check), so a build shipping the Ravensight web SDK has its session refused rather than recorded as a real player.

Every row is pass, warn or fail, and every row that is not pass prints its fix. Exit code 3 when any row fails; a warning never fails, so CI does not break on a missing optional tool or a newer release. check reserves no units and starts no job, and says so at the end of its output: everything it asks the server is a read, including the price estimate.

For a native driver (ios_driver, android_driver, desktop_driver, unity_driver, unreal_driver) the build row reads the driver config and confirms the app it names exists on disk (an app named by bundle id or package can only be confirmed by the server, so that is a warning), and the automation server row is the free capability probe: GET /status on the config's loopback endpoint, POST /session with the run's own capabilities, GET /session/:id/source to confirm the app opened and answers, then DELETE /session/:id. The detail names the step that failed. A session the probe could not close is a warning. This talks only to your own local server. docs/NATIVE_RUNNER.md walks through an iOS simulator setup with the expected rows at each step.

check --json prints one object:

{
  "ok": true,
  "reserves_units": false,
  "note": "check reserves no units and starts no job: ...",
  "cli": { "installed": "0.3.2", "latest": "0.3.2", "min_supported": "0.1.0" },
  "login": { "present": true, "valid": true, "games": [{ "gameId": "g1", "name": "Pocket Knight" }] },
  "project": { "game": "g1", "environment": null, "in_scope": true },
  "brief": { "present": true, "complete": true, "missing": [] },
  "build": { "driver": "ios_driver", "ref": "/absolute/path/PocketKnight.app" },
  "automation": { "endpoint": "http://127.0.0.1:4723", "reachable": true, "session": true, "app": true },
  "checks": [{ "name": "login", "status": "pass", "detail": "...", "fix": null }]
}

automation is null for a driver that is not native. A value of null inside it means that step was not reached because an earlier one failed. environment comes from the server when it reports one, otherwise from an environment key in .ravensight/config.json, otherwise null.

For a web game there is also a browser row: check imports Playwright, launches a headless Chromium and closes it, which is exactly how a web run starts. A missing browser fails the check when the linked build is a URL or you pass --web. When no build is linked yet and the configured driver is the default playwright_web, it is only a warning, so a Godot repo that never changed the default is not failed over a browser it will not use. A repo linked to a Godot project or a driver config gets no browser row. The fix is:

npm i -D playwright && npx playwright install chromium

Past --personas in POST /jobs, requesting more than 100 personas or the same slug twice is refused with 400 too_many_personas, and a slug this game does not have is 400 unknown_persona; run checks the count against the same ceiling before it asks the server anything. Past 8 personas the job is still allowed, but a warning row explains that the later runs may not get to play inside the job's wall clock; any run that does not play is refunded automatically. check shows that same warning as a row when it applies.

| Flag | Effect | |---|---| | --game <gameId> | Check against this game instead of the linked one. | | --modules <list> | Price this module list, comma separated. Default the modules run registers, persona_playtest,aggregate_report. | | --personas <list> | Price this persona list. Without it, a module list with persona_playtest prices the repo config's personas or the default one, as run would. | | --godot-path <path> | Where Godot is, if it is not on the PATH. | | --web | Check the browser even when this repo is not linked to a web build. | | --driver <name> | Check the setup for this driver instead of the configured one. | | --driver-config <file> | The native driver config to probe, instead of the configured build. |

init

Links this repo to a game and writes .ravensight/config.json. Refuses a game your token cannot see, so the first job does not fail with a 404. Adds .ravensight/jobs/ and .ravensight/brief.local.json to an existing .gitignore.

Without --game, one visible game is linked as is. With several, init lists them numbered and asks Which game? [1-N] when it is running in a terminal; in CI or a pipe it prints the list and asks you to pass --game instead. A token that can see no games is told so, with the fix (approve a game for the token at login).

| Flag | Effect | |---|---| | --game <gameId> | The game to link. Optional with one visible game, or in a terminal. | | --force | Relink a repo that is already linked. |

personas

Lists the persona slugs --personas accepts, from the content pack this game gets: each one's name, action budget (its patience) and play style, with the default marked. A free read. Takes --game <gameId> and --json.

brief

Writes the expectations brief: your goals, what success looks like, how the game is controlled, where a persona starts. Then prints what the run run would start costs ("A run of 1 persona (curious-kid) would cost $1.00: a persona run $1.00 each"; the game profile is a separate job), whether your balance covers it, and, past 8 personas, the same wall-clock warning run shows. A brief is required before run; the server answers 409 brief_required without one.

| Flag | Effect | |---|---| | --show | Print the current brief and stop. | | --open | Open the brief form in the dashboard instead of asking here. | | --draft | Save as a draft even when it looks complete. | | --from <file> | Read the fields from a JSON file instead of asking. | | --game <gameId> | The game, when this repo is not linked. | | --modules <list>, --personas <list> | Price this module and persona list in the estimate it prints. Every unit is named with its own price, such as "profile job $4.00, a persona run $1.00 each". |

The brief --from file

A JSON object of brief fields, or the same object wrapped as { "fields": { ... } }, which is how a saved brief carries them. The first four are what makes a brief complete; the rest are optional.

| Field | Shape | Notes | |---|---|---| | goals | [{ "goal": "...", "priority": 1 }] | At least one. goal is one of find_bugs, onboarding, balance, level_flow, narrative, accessibility, performance. priority is 1 (highest) to 5. | | success_criteria | string | What success looks like, one criterion per line. The aggregate report scores each line. | | build_notes.controls | string | How to play: keys, buttons, the verbs a persona needs. | | build_notes.how_to_start | string | Where a persona begins: the URL, the menu path, the first thing to press. | | design_intent_notes | array of strings | Deliberate choices a persona must not report as bugs. | | known_issues | array of strings | Issues you already know about, so they are not reported again. | | expected_noise | array of strings | Console errors and failed requests that are expected, such as "401 /auth/me before login". Never a finding. Kept on this machine. | | test_account | { "email", "password", "how_to_sign_in" } | A throwaway account the persona signs in with, so a run starts past sign-up. The password must be at least 8 characters. Kept on this machine. |

expected_noise and test_account never go to the server: its brief schema refuses keys it does not list, and a password does not belong in a record the studio shares. brief --from splits them into .ravensight/brief.local.json, readable by you only, which init adds to .gitignore; brief --show prints them with the password masked, and run and resume read them from there.

An expected_noise pattern matches an automatic console note when every status code (401) and every path (/auth/me) in it appears in the note; the other words describe it for the reader. A pattern with neither is matched as text, and /.../ is a regular expression. One rule is built in: a 401 on a who am I check, a request whose last path segment is /me, /whoami, /session or /current_user (such as /auth/me), is what a signed out page does and is not a finding. A 401 anywhere else (/me/inventory, /user/42/save, /token, /login) is left to the report step. A matching note is marked "expected noise" in the report prompt, and the runner drops a finding whose evidence is only expected noise, listing it in report.md. A note the persona wrote itself is never noise.

The test account password crosses the model proxy during play only: it is in the persona's prompt on every play turn, because that is how a persona can type it. Use an account made for testing, with nothing behind it you would mind losing. The report step never sees it: its prompt has no test account, and the password is replaced by [test password], wherever it stands as a whole token, in the notes, the action log, the transcript, and in report.md and report.json before anything is uploaded. A password field's value is never shown in an observation.

{
  "goals": [{ "goal": "onboarding", "priority": 1 }, { "goal": "find_bugs", "priority": 2 }],
  "success_criteria": "A new player reaches level 2\nNobody gets stuck in the menu",
  "build_notes": { "controls": "Arrow keys to move, space to jump", "how_to_start": "Press Start on the title screen" },
  "design_intent_notes": ["The fog on level 1 is intentional"],
  "expected_noise": ["401 /auth/me before login"],
  "test_account": { "email": "[email protected]", "password": "a-throwaway-password", "how_to_sign_in": "Log in from the top right" }
}

profile

Builds the game profile and the capability report: what your game is, what can drive it, what a persona will be able to do. Takes --game <gameId> and all the run options below.

run

Registers a job, plays it, writes the reports, uploads them, finishes the job. Prints the price and asks before spending. Before any charge, even with --yes, it prints one bold line with the persona count and the total ("This run: 1 persona (curious-kid) on playwright_web, $1.00 in total: a persona run $1.00 each"), the balance, and what a persona run buys: the action budget (the persona's patience in the pack, lowered by --max-actions or the server's ceiling) and that sign-up alone can take 8 of those actions. A test_account in the brief lets the persona sign in instead. An unknown --personas slug is refused with the valid slugs before anything is priced.

--long makes every persona run a long run: twice the actions, model budget and wall time of a standard run, for a game whose screens are heavy or whose goal is far in. It costs twice a standard run. The price is the server's estimate for the long run, printed and confirmed like any other, never a figure built into the CLI. Locally a long run uses the run's own action limit when the server's register answer carries one, otherwise twice the standard ceilings (the persona's patience and the per run ceiling); --max-actions still only lowers it. check suggests --long when the first screen is heavy.

Before the estimate, run checks the build is up, so a down server is never paid for. Set preflight in .ravensight/config.json to name what to check:

{ "preflight": { "url": "/api/health", "expect_status": 200 } }

url may be relative to the build URL; expect_status is a number or a list of numbers, and without it any 2xx or 3xx passes (redirects are not followed). With no preflight, a web build's own URL has to answer 2xx or 3xx; other drivers are not checked unless a preflight.url is set. A failure prints the status it saw (or the network error), exits 1, and registers nothing.

With a native driver, run first runs the build and automation server rows of check. When either fails it exits 1 before the estimate, the confirmation and the register, so no job exists and nothing is reserved, and the message names the failing row and its fix. --skip-preflight registers without the probe, and then the job is registered and charged even if the automation server turns out to be down. The probe prints probing <endpoint>, this can take up to 3 minutes on a fresh simulator before it starts, and each row as it completes.

There is no product limit on how many personas one job can name; an unplayed run is refunded automatically when the job settles, so a large job never costs more than what actually got played. --personas past 100, or the same slug twice, is refused before anything is charged, checked locally against the content pack so the round trip to the server is never spent finding that out. Past 8 personas the job is still allowed; run prints a warning above the price prompt saying the later runs may not fit inside the job's wall clock, and splitting the job in two is what gets everything played.

| Flag | Effect | |---|---| | --game <gameId> | The game, when this repo is not linked. | | --personas <list> | Comma separated persona slugs. Defaults to one, curious-kid; ravensight-playtest personas lists them. | | --driver <name> | playwright_web, godot_driver, cli_stdio, or a native driver: desktop_driver, unity_driver, unreal_driver, android_driver, ios_driver. | | --build-url <url> | The web build to play. | | --godot-project <dir> | The Godot project directory. | | --driver-config <file> | The native driver config, or the JSON-RPC bridge config for cli_stdio. | | --skip-preflight | Register without the free checks first: the build health check (any driver) and the automation probe (native drivers). The job is registered and charged even if the build or the automation server turns out to be down. | | --cli <command> | The command that starts a text build. | | --max-actions <n> | Lower the actions one persona may take. It can only lower the persona's own patience (and the server's per run ceiling), never raise it. | | --long | A long persona run: twice the actions, model budget and wall time of a standard run, at twice the price. The price is read from the estimate and confirmed before any charge. | | --concurrency <n> | Personas at once. Default 2; check recommends a number for this machine. | | --upload-video | Upload session.webm as well as the reports. | | --upload-transcript | Upload transcript.jsonl as well as the reports. | | --dry-upload | List every file and byte that would be sent, and send nothing. | | --key-modifier <meta\|control> | The modifier the browser driver uses for editing shortcuts such as select all. Defaults to this machine's own (Command on macOS, Control elsewhere); set it when the browser's platform is not this machine's. RAVENSIGHT_KEY_MODIFIER does the same. | | --headed, --headless | Show or hide the browser. | | --verbose | Print each turn. |

Three flags are accepted and reported as not applied by this release, rather than silently ignored: --quality (the model tier is the server's, enforced at the proxy), --offline-upload (uploads already queue) and --build-dir (serve the build and pass --build-url). There is no spending flag: each run's model budget is set by the server from what the job was charged, and --budget-usd is an unknown option (the refusal says where the budget comes from).

Each run hands its playtest identity to the game, so the game's own Ravensight events from a playtest are marked synthetic rather than counted as a player. A Godot game reads it from its environment. A web game's JavaScript SDK reads it from the page: the browser driver sets window.__RAVENSIGHT_PLAYTEST__ and localStorage['ravensight.playtest'] before the game's scripts run. It is never sent as a request header, which would reach your game's own API too.

With no ingest key configured, a Godot run says "no ingest key configured, so no synthetic telemetry was sent". A web run says "synthetic telemetry comes from the game's own SDK on the page, sent with the run's playtest token" instead, because the page's SDK already sends the game's events marked synthetic.

When a run reaches its model budget before the report step, nothing more is asked of the model. The persona's notes are written to a partial report.json (partial: true, one unreviewed finding per note with the nearest screenshot as evidence) and uploaded, so the job's aggregate still counts them.

When the budget runs out during the report step itself, the same partial report is uploaded and the run is completed as budget_exceeded. The job does not exit 2 for it, and the CLI prints "Run ran out of budget; the partial report was uploaded and the refund decision is the server's". When the server's answer says a refund is expected, the line ends "a refund of $X.XX is expected when the job settles" instead, and "the run was refunded" once the job has settled and the run was refunded. If the server refused the partial report, it says so and where the notes are instead. An older API that refuses that completion prints its own message, the run is completed as failed, and the job exits 1.

How the report step reads what happened:

• Values the report schema does not allow are normalised before upload, so a run is not lost to them when the model has no budget left for a repair: a severity that is a category word ("severity": "ux") moves to category when the category is missing or not allowed and the severity becomes minor; any other unknown severity becomes minor; an unknown category takes a close match the schema has (ui is ux, crash is bug), else other when the schema has it, else ux (the server's schema has no catch-all today); a value that differs only in case is respelled; any other enum field takes the schema's default. Each change is a runner note at the end of the finding's description and is listed in report.md under "Normalised by the runner". The server's schema stays strict.

• After the report is written, each blocker or major (at most 3) is checked against its own evidence: the screenshot it cites taken nearest after the observation it cites is sent with its claim, and the question "does this image contradict the claim: yes, no, unclear". A yes downgrades it one step, with a note naming the screenshot, in both report files. The cost is 3 calls at most, each one screenshot in and 32 tokens out, drawn from the run's own model budget (the one the 80 percent warning keeps room in for the report), never charged separately; a refusal ends the checks and keeps the report as written.

• Console notes that match the brief's expected_noise, or the built in rule (a 401 on a who am I check while signed out), are expected noise and never a finding; see the brief --from file.

• Console errors are recorded as automatic notes. A 4xx on a request (Failed to load resource, HTTP 404, status: 401, 401 Unauthorized and similar) is a network note: context, not a bug. On the web a Failed to load resource note names the request's method, URL (with its query string and fragment removed) and status, and the report lists those requests under network notes. A finding whose only evidence is automatic network notes is capped at minor by the CLI, and its description says so.

• The report step is given every action that went through (the newest 60): what was clicked or typed into, by name or ref, and whether the screen changed. A claim that something was never done or did nothing has to agree with that list. A finding that says a named control "did nothing", "does nothing", "never kicked off", "never started", "no response" or "nothing happened", when the most recent click on that control changed the screen without bringing up an error or a dialog, at or after the screen the finding cites, is downgraded one severity in report.json and report.md with a note saying at which step. It is never dropped.

• Every note carries after, the URL and title of the screen that followed it. A blocker or major finding has to cite the observation where the failure is visible and must not be contradicted by that next screen or by a later persona note. It also has to cite the screenshot taken at or after that observation (the prompt says which observation each screenshot followed), and that screenshot must not show what the finding says is missing.

• Every evidence entry is {"type": ..., "ref": ...}. A screenshot path the model put under path or file is moved to ref before the report is checked; the file still has to exist in the run directory. If the server still refuses report.json at complete, the CLI prints every validation message it sent, the run is not counted as succeeded or in the aggregate, and the job exits 1.

• The model is told never to write calendar dates. report.md carries the CLI's own Generated at timestamp under its title, and the aggregate report's header uses its generated_at.

• A template slot the model copied instead of filling (**Date:** <run date>, <driver>, <N actions>) is filled from the CLI's own facts, or "not recorded" when it has none, in report.md, the prose of report.json and the aggregate. Other angle brackets are left as written: keys such as <Enter>, elements such as <form>, types such as Array<string>, links, code and quoted game text.

• The server keeps a finding only when one of its evidence references points at a file the run uploaded. After delivery the CLI marks the findings it dropped (by the server's dropped_finding_ids, or by applying the same rule to the files it uploaded when its count matches): dropped: true in report.json, out of the summary in report.md and into a section headed "Dropped by the server (no usable evidence)", and out of the aggregate and the synthetic telemetry, so the local report counts what the dashboard counts. When it cannot tell which, report.md says how many and that the dashboard is the record. The rewritten files are journalled as local edits, and upload <job> never re-sends a completed run's report files.

What personas do well

A persona plays like a first time player at human speed: menus, onboarding, the first few minutes, unclear text, screens that do not respond, and whether it would come back. On the web, when a persona replaces a field's value (the type tool with replace), the driver clears it with fill, the same way on macOS, Windows and Linux; ordinary typing is keystrokes. An editing shortcut such as select all uses the platform's own modifier (Command on a Mac, Control elsewhere, or --key-modifier).

On the web each button, link and field the persona sees carries a short ref such as [e12], and its centre @(x,y) when it is on screen. The persona clicks and types by ref, which names exactly one element even when two share a name; two such elements are both shown, each with where it is ("in header strip", "in main"). A name that matches twice is answered with the refs to choose from, and a target written as locator code (getByRole(...), page.locator(...), css=, xpath=) is refused at once. A whole line copied from the screen (button "Run the week" [e12] @(1000,303), or any part of one) is clicked by its ref, else its @(x,y) (refused like a stale ref when the page has loaded a new document since); a ref or point inside a quoted name is part of the name, and a bare e4 is text when something on screen is named exactly that. A target that is none of a ref, coordinates or text on the screen is refused at once rather than searched for. A name that matches one element on screen is clicked by that element's ref, and a text search skips hidden elements, so a mobile only copy hidden by CSS is never the one clicked. Failures like these are the driver's, not the game's: they are recorded as driver_error, and a blocker or major resting only on them is dropped before upload, as is any finding resting only on the persona's "waiting for X to finish" notes.

An action counts as changing the screen when the URL changed, the page loaded a new document, the visible tree (field values included) changed, or for typing, the field's own value changed, with one short second look for a navigation still starting. That is the "screen changed" the report step's action log and the downgrade rule read.

Every field, checkbox, radio, slider and select the persona sees carries what it holds now, read from the page rather than from the accessibility text: value="..." (cut at 80 characters), checked or not checked, or selected option="...". A password field shows value=(hidden), and a field the page could not answer for shows no value at all, so a typed password never reaches the model.

On a long web page the persona's text view of the screen is capped at 4,000 characters. A run of 20 or more look-alike items is always shown as its first two and "and N more like it", under the cap too; a shorter run under the cap is shown as it is. Over the cap, buttons, links and fields are kept first, a long run of look-alike items (a list of 249 cards) is shown as its first two and "and 247 more like it", and what changed since the last look comes next, so a form after a long list still reaches the persona.

Each persona starts at the game's URL as a fresh player with no account, so a sign-up form with validation can use up a whole unit before the game itself is reached. Every run gets its own identity to sign up with, an email persona-<persona>-<short run id>@example.test and a display name ending in the short run id, so a second run never hits "that email is taken" from the first. example.test is a reserved name and never delivers mail. Point --build-url at a guest or demo path, or say where to start in the brief's how_to_start note (for example "choose Play as guest"), when the sign-up is not what you want tested, or give the brief a test_account the persona signs in with instead.

fix

Takes one finding, attempts the change on a branch of its own, replays the finding to see whether the change worked, and prints the pull request command for you to run.

ravensight-playtest fix pt_job_1/F002 --build-url http://localhost:8080/
ravensight-playtest fix fnd_7g3k --no-verify

The target is either <jobId>/<label>, where the label is the one the report gives the finding, or a bare finding id.

| Flag | Effect | |---|---| | --game <gameId> | The game, when this repo is not linked. | | --branch <name> | The branch to create. Defaults to ravensight/fix-<label>-<6 hex>. | | --no-verify | Do not replay the finding afterwards. The changeset then carries no verdict. | | --allow-dirty | Fix on top of uncommitted changes instead of refusing. Your changes end up in the diff too. | | --max-actions <n> | Cap the turns the fix may spend. Capped at 60 whatever you ask for. | | --build-url <url>, --godot-project <dir>, --driver <name> | Where the verify run plays, resolved exactly as run resolves them. | | --json | Print the changeset instead of the summary. |

What happens, in order:

1. The finding and its run report are read, with up to two of its screenshots.
2. Git is checked: a directory that is not a repository, or a machine with no git, stops here with one sentence, and so does a working tree with uncommitted changes unless you passed --allow-dirty.
3. The price is printed and confirmed. Nothing is charged and no branch is created until you say yes.
4. The server is asked whether the finding is a bug at all. A finding it reads as design intent with confidence 0.7 or more stops the attempt before anything is edited.
5. The change is planned, then made with a write tool that can only write text files inside your repository, at most 200 kB and at most 20 files.
6. The verify run replays the finding on the build you pointed at.
7. changeset.json and report.md are written, uploaded and attached to the fix run, and the pull request command is printed.

It never pushes and never commits. There is no git commit, no git push and no forge client anywhere in the command. You are left on the branch with the change in your working tree, and the last thing printed is the command to open the pull request once you have read the diff.

A fix attempt is one unit. It is refunded when the finding turns out to be design intent, and refunded when the attempt changes nothing, in both cases without a model turn being spent on the edit itself. The verify run is included in the price.

A verify can only mean something against a build this machine serves. The change exists in your working tree and nowhere else, so a --build-url pointing at a hosted build would answer about the last upload instead. Any non-local URL is refused unless you pass --no-verify, because a confident "fixed" from the wrong build is worse than no answer. Rebuild and serve the branch locally before the verify, or skip it.

The verdict is printed in words: fixed below 0.35, still reproduced above 0.65, unknown in between, and "not verified" when there was no verify run.

A fix cannot be resumed. Ctrl-C, or cancelling the job from the dashboard, abandons the attempt and gives the unit back: whatever was written stays on the branch for you to read or delete, and nothing is committed. That is different from run, which resume picks up where it left off, because a half-applied edit is not a playthrough to carry on with.

resume

With no job id, lists the jobs this repo has a journal for, newest first. With one, keeps finished runs, restarts or replays the rest, and never charges twice: the register is replayed under the journal's own idempotency key, which answers the stored job without pricing anything.

| Flag | Effect | |---|---| | --game <gameId> | The game, when the job journal is missing. | | --status | Report and stop, without restarting anything. | | --restart-interrupted | Replay an interrupted run from the start. By default a run that got far enough is finalised from the transcript it already has, which spends no model turns; this forces the replay, which spends the action budget again. |

upload

Finishes the uploads a crashed or offline run still owes. Safe to run repeatedly: files already sent are skipped by checksum.

| Flag | Effect | |---|---| | --game <gameId> | The game, when the job journal is missing. | | --dry-upload | List what would be sent and send nothing. | | --video | Include the session video. | | --transcript | Include the transcript. |

review

Opens the post job review form. Your marks on the findings are what tunes the personas, so this is the useful half of the loop. Takes --game <gameId> and an optional job id.

open

Opens the dashboard page for a job, or for the game. Takes --game <gameId> and an optional job id.

Playing a Godot game

--driver godot_driver launches your own Godot binary against your own project, with a small driver addon injected so the persona can read the scene tree and send input.

Your project directory is never written to. The project is copied to a temp directory and the addon is injected into the copy, through an override.cfg and the copy's own project.godot. Nothing about the copy is written back. The copy excludes .git and node_modules and is refused above 2 GB, and its temp root is 0700, because a project directory can contain a .env.

The addon only ever activates deliberately: it needs --ravensight-driver on the command line or RAVENSIGHT_DRIVER=1 in the environment, it binds 127.0.0.1 and nothing else, it requires a per run token in a hello call before it will answer any other method, it accepts one client at a time, and it frees itself the moment any of that fails. On a release export it is stripped from the build unless the preset carries the ravensight_driver feature tag.

Headless Godot routes no synthesized mouse or GUI input. A click on a button does nothing under --headless and works windowed, and nothing takes keyboard focus without a click first. So --headless is a smoke test mode, not a playtest mode, for anything with a user interface: on a machine with no display, use Xvfb and a windowed Godot rather than --headless. The driver prints this in the run's console log when it launches headless.

Video from the Godot driver is a frames/ sequence at 4 fps rather than a container, so stitching needs ffmpeg and ffmpeg stays optional.

docs/godot-driver.md has the protocol, the method set and the error codes.

What a report has to contain

report.json is validated against the pack schema before it is sent, and report.md is linted against the four sections the server expects:

• What worked
• Top issues
• What this run didn't cover
• Suggested next step

A missing section is a lint finding rather than a rejection, so a report is never lost to it. The CLI lints locally first and feeds any miss back into one repair turn, which costs a model turn; a report written with the four headings from the start avoids that.

The repository's tests check rendered reports against test/fixtures/server/, a byte copy of the server's report schema from game_tracker commit 864027e, which has to be refreshed by hand whenever the server's schema changes.

Exit codes

| Code | Meaning | |---|---| | 0 | Success | | 1 | Failure | | 2 | Budget exceeded during play. Partial results were written and uploaded. A run that ran out during the report step and was completed does not make the job exit 2. | | 3 | Environment check failed | | 4 | Authentication failed | | 5 | Model API error after retries | | 10 | Canceled |

CI can treat 2 as a success with a warning.

Environment variables

| Variable | Effect | |---|---| | RAVENSIGHT_TOKEN | Use this token instead of the keychain. The CI path. It must be a gt_cli_ token, the kind login stores. A gt_mcp_ token is an MCP personal access token and a gt_live_ token is a game's ingest key; check refuses both and says which one it saw. | | RAVENSIGHT_API_URL | The API base URL. Defaults to https://api.ravensight.io. | | RAVENSIGHT_DASHBOARD_URL | Where the dashboard is, if it is not derivable from the API URL. | | RAVENSIGHT_GAME_ID | The game, when there is no linked repo. | | RAVENSIGHT_KEY_MODIFIER | meta or control: the browser driver's modifier for editing shortcuts, as --key-modifier. The flag wins when both are set. | | RAVENSIGHT_INGEST_KEY | The game's ingest key, so a web run can emit its synthetic session into your event stream. ingest_key in .ravensight/config.json does the same. Without either the run still reports; it just sends no telemetry. | | RAVENSIGHT_GODOT | Path to the Godot binary. Tried first, then $GODOT, then $GODOT_PATH, then the PATH, then the platform's usual install locations. --godot-path beats all of them. | | RAVENSIGHT_FFMPEG | Path to ffmpeg. | | RAVENSIGHT_HOME | Where per user state lives. Defaults to ~/.ravensight-playtest. | | RAVENSIGHT_CACHE_DIR | Where the content pack cache lives. Defaults to the OS cache directory. | | RAVENSIGHT_NO_BROWSER | Set to 1 to never try to open a browser. |

What ends up on disk

In your repo:

.ravensight/
  config.json                 the game link and your defaults
  brief.local.json            expected_noise and test_account, this machine only (0600)
  jobs/<jobId>/
    state.json                the resumable journal
    upload-queue.jsonl        what was uploaded, and what is still owed
    runs/<runId>/
      report.md  report.json  usage.json
      screenshots/NN-slug.png
      transcript.jsonl        only when you opt in
      session.webm            only when you opt in

A fix job's run directory holds changeset.json, report.md (the pull request body) and usage.json, plus the screenshots it was shown as evidence, and a local transcript.jsonl of the apply loop (which tool, on what, how much came back) that is never uploaded and is how an attempt that wrote nothing gets explained afterwards.

Add .ravensight/jobs/ and .ravensight/brief.local.json to your .gitignore; init does it for you when you already have one.

fix also leaves a branch on your working copy, checked out, with the change in your working tree and nothing committed. It is the only thing in this CLI that writes to your source at all, and it never pushes: no commit, no push, no forge client exists in it. Delete the branch and you are back where you started.

What leaves your machine

The dashboard has the full answer, artifact by artifact, at

https://app.ravensight.io/games/<gameId>/playtest/what-leaves-your-machine

which is the page to trust: it is rendered from the server's own allowlist, so it cannot drift from what is actually accepted. ravensight-playtest open links to it. The short version:

Uploaded by default: report.md, report.json, usage.json, changeset.json, the aggregate and capability reports, and screenshots/. Uploaded only when you ask: transcript.jsonl (--upload-transcript) and session.webm (--upload-video). Source code never, at any path. The upload allowlist has no entry for it, the CLI refuses it locally, and the server refuses anything not on its own copy of the list. src/upload/allowlist.js is that copy, and a test pins it equal to the server's file when a checkout is on the same machine.

Findings cite a file and a line range. The source text behind that line is not uploaded.

.ravensight/brief.local.json is never uploaded. Its expected_noise goes into the persona and report prompts. Its test_account, password included, goes into the persona's prompt during play only, through the model proxy; the report step never gets it, and the password is scrubbed to [test password] from the transcript, the notes and both report files before upload.

run's build health check is one GET to your own build (or the preflight URL you set), made from your machine.

A fix uploads changeset.json and report.md. The changeset carries the finding id, the branch name, the commit it branched from, the file paths it changed with their added and removed line counts, the verify verdict, and the pull request title and body. Not the diff, and not the file contents. What the model read while working is what goes over the wire as part of its own reasoning, exactly as profile already does, and the same guards apply: a file that trips the secret scan is withheld rather than read, and both files a fix writes are scanned again before they are uploaded.

Run upload --dry-upload or run --dry-upload to see the exact file list and byte count before anything leaves the machine.

Model calls go to Ravensight's proxy rather than to Anthropic, under the token you logged in with. No Anthropic key exists on your machine, and the CLI builds its client with apiKey explicitly null, so an ANTHROPIC_API_KEY already in your environment is never picked up and never sent anywhere.

Before anything is written to a run directory, the transcript and usage.json are scanned for secrets. A hit is replaced by a line naming the shape that matched, never the value.

The 90 day artifacts

The session video and the transcript expire after 90 days; everything else lives for the life of the game. That is enforced as an S3 object tag rather than as a filename rule, because a lifecycle rule can filter on a tag or a key prefix and the filename is at the end of the key.

The tag is a contract between three parties, and the CLI is one of them: the presign response hands back the exact x-amz-tagging value that was signed, and the CLI sends it verbatim on the PUT. Changing it, or dropping it, is a SignatureDoesNotMatch rather than an untagged object, which is the failure you want: an untagged 200 MB video would sit in the bucket forever and nothing would say so. The CLI never computes the value itself.

Offline

After one online fetch the content pack is cached and verified by sha256, so a run works with the network down except for the model calls themselves. Uploads and progress reports queue up and flush when the server is reachable again; upload <jobId> drains them by hand.

Embedding

src/api/README.md documents the one module other code may import, which is also this package's entry point:

import { createClient, packs, uploadRunDir } from 'ravensight-playtest';

Reference

| File | What is in it | |---|---| | docs/api-contract.md | Every server route this CLI calls, with the headers it sends and the status codes it handles. The list to check a server change against. | | docs/godot-driver.md | The Godot driver protocol, its method set, its error codes and its safety rules. | | docs/ADAPTERS.md | Every driver, its target and how far it has been validated. | | docs/NATIVE_RUNNER.md | A minimal iOS simulator runner through a local W3C automation server, step by step, with the check rows expected at each step. | | src/api/README.md | The module other code may import, and what the runner half expects from it. |

License

MIT. Copyright Reality Software Entertainment.

Ravensight Playtester action selection

The optional --controller jev mode uses the existing persona run. A screenshot observer proposes actions, Ravensight selects one on the server, and the local driver executes only that choice. Ambiguous controls stop the run for review. Reports, screenshots and findings use the normal upload and review flow.

ravensight-playtest run --driver playwright_web --controller jev --build-url https://your-game.example/ --max-actions 12 --headed

Sign in, initialize the project and save its expectations brief first. The server must advertise this controller in its content pack; otherwise the command stops before registering a paid job. No provider key belongs in the CLI. Selection costs share the existing persona-run budget and do not create another wallet charge. Resume preserves the controller recorded in the job journal.

For Jump Bunny, use https://reality-software-entertainment.github.io/poc-jump-bunny/ and the public player label Ravensight Playtester. This is a local browser run, not a hosted browser service. Human playtesting is still needed for subjective feedback and timing-sensitive gameplay.

Build adapters

Browser and Godot runners share the paid playtest lifecycle with desktop, Unity/Unreal export, Android/iOS and custom JSON-RPC adapters. Native adapters require a locally configured automation server and remain experimental until validated against your build. See adapter setup and validation boundaries and the native runner starter; check probes the automation server for free before a paid native run.