@excellent-so/usertrial

A usability lab of synthetic users. Recruit personas from your ICP, send each one through your product to do the job they came for, and get a verdict — did it work, could they do it themselves, is it robust, would they recommend it — plus every bug, dead-end, and missing feature they hit, filed as fixable work.

npx @excellent-so/usertrial run      # run the panel, score the four value questions
npx @excellent-so/usertrial open     # read the study in the viewer

Zero runtime dependencies. Fully local — no cloud, no account. Every trial runs in a fresh, disposable instance, so the persona genuinely starts where a real new user starts: onboarding.

Why

You can't ship a product you haven't watched a real user try. But real user tests are slow, expensive, and unrepeatable — you can't run one on every commit. usertrial makes user testing synthetic, fresh, and repeatable:

1. Personas from your ICP — each with a real job-to-be-done and a disposition (patient? AI-native? skeptical of marketing?).
2. Fresh every time — each trial provisions a brand-new instance (a disposable data dir), so nobody tests against your pre-seeded dev database.
3. Two lanes — the same job is attempted by an agent (over the MCP tool belt) and by a human (in a real browser). The gap between them is itself a finding.
4. A verdict, not a log — every trial answers four questions a founder actually cares about, and produces a survey the persona "fills out" in first person.
5. Findings become fixable work — bugs, UX dead-ends, visual issues, and missing features are filed into Excellent as tasks on the ship/verify gate. Fix them, re-run, watch them turn green. Measure → fix → re-measure.

No existing tool does this. Playwright/Cypress assert the flows you wrote. Cloud "synthetic user" services run against a deployed URL with no product-market- fit verdict and no path back to your backlog. usertrial runs locally against your dev build, scores PMF-shaped questions, and routes what breaks into your tracker.

The unit of work: a trial

A trial is persona × job, run in a fresh instance, on one or both lanes:

| Lane | What it is | Answers | |---|---|---| | agent | a fresh excellent-mcp tool belt on a disposable data dir | Could an agent walk in and do this? Did the world actually change? | | human | the real app booted on a spare port, driven over a zero-dep CDP client | Could a person do it themselves? — it completes the job through the UI and reads the DOM to confirm |

The agent lane is the spine — it needs no running server, is offline and deterministic. The human lane actually does the job in the browser: it walks the real 15-step onboarding to completion (screenshotting each step), then runs a per-job UI plan (e.g. open Customers → Add customer → Create) whose DOM oracle verifies it happened — so a human-only run is scored on what the browser really did. When both lanes run, an agent-vs-UI mismatch is filed as a cross-lane gap finding (e.g. the UI can post a job the fresh MCP server can't).

The verdict

Every trial is scored on the questions that matter:

1. Solved? Did it solve the problem they came for — and was it even a problem you solve (out-of-scope is reported as such, never as a failure).
2. Self-serve? Could they do it themselves, in a time that makes sense (measured from steps, wall-clock, dead-ends).
3. Robust & valuable? Did it hold up and deliver the value they needed (we inspect the resulting artifact, not just "no error").
4. Recommend? Would they recommend it — a 0–10 NPS + the Sean-Ellis "how disappointed if it vanished."
5. Agent-operable? Could an agent do these actions itself (the agent lane, directly).

A grounded, skeptical buyer even flags overclaims ("local AI", "nothing leaves your machine", "on autopilot") as red flags — a real signal that costs NPS.

Install & run

# First touch — zero install
npx @excellent-so/usertrial run          # run the whole panel (agent lane)
npx @excellent-so/usertrial open         # serve the viewer and open it

# Scope it down
npx @excellent-so/usertrial run --persona founder           # one persona's jobs
npx @excellent-so/usertrial run --job founder-post-job-shortlist
npx @excellent-so/usertrial run --lane both --base-url http://127.0.0.1:3100

# File what broke into Excellent (gated tasks, idempotent)
npx @excellent-so/usertrial run --record

Requires Node ≥ 20, an excellent-mcp bin for the agent lane (npm i -g @excellent-so/cli, or run from inside the monorepo — it auto-detects packages/excellent-mcp/dist), and — for the human lane — a running app instance and a system Chrome (found automatically, never bundled).

Commands

| Command | What it does | |---|---| | run | run a panel of trials and score them | | surfaces | sweep every app surface (modes + settings + sections) for reachability + console errors | | report | print the last study readout | | open / serve | serve the viewer (open also launches your browser) | | record | file the last run's findings into Excellent as gated tasks | | personas | list the ICP personas | | jobs | list the jobs-to-be-done |

Key run flags: --persona <id>, --job <id>, --lane agent|human|both, --brain auto|claude|anthropic|mock, --limit <n>, --record, --base-url <url>, --auto-boot, --project <slug>, --json.

The two brains

The persona's navigation runs one of two ways:

• scripted (default) — each job carries a deterministic plan (agent lane: real tool calls; human lane: navigate/fill/click steps). Reliable, offline, no key — the always-green regression path.
• auto — an LLM is the user: it voices the persona's first-person survey and can drive exploratory navigation. Pluggable transport (claude CLI → Anthropic API key → mock); it falls back to scripted when no model is reachable, so the whole harness runs green with no key and no network.

Findings → fixable work

Everything that goes wrong is captured as a typed finding — bug, ux, visual, or missing-feature — with the persona/job that hit it and evidence (a screenshot, a console line, a tool error). usertrial record (or the viewer's File to Excellent button) opens them as tasks that travel the ship/verify gate, idempotent by a [usertrial:<key>] marker so re-runs never duplicate. When the issue is fixed and verified by a different agent, the task closes — and the next run re-tests it.

Configuration

Drop a usertrial.config.mjs in your project root (all keys are optional; env vars and CLI flags override):

export default {
  // Where findings are filed (else the monorepo dev project).
  project: "micro",
  // A running instance for the human lane (else --auto-boot a fresh one).
  baseUrl: "http://127.0.0.1:3100",
  // The auto-brain transport: "auto" | "claude" | "anthropic" | "mock" | "off".
  brainTransport: "auto",
  // Explicit excellent-mcp bin, if not on PATH / not in a monorepo.
  // excellentMcp: "/path/to/excellent-mcp",

  // Bring your own personas (a module path, resolved relative to this config).
  // personas: "./usertrial.personas.mjs",
};

Bring your own personas

The built-in catalog is a generic B2B/prosumer cast. Point the personas config key at a module that exports your ICP, and usertrial merges it into the catalog: a persona with a new id is appended, a persona reusing a built-in id overrides it, and any BURNED_CLAIMS you export are unioned with the built-ins. Built-in ids always remain, so the built-in jobs keep resolving. A malformed module fails fast with a one-line error naming the file and field.

// usertrial.personas.mjs — resolved relative to usertrial.config.mjs.
// disposition fields are 0..1: tech · aiNative · patience · skeptic.
export const PERSONAS = [
  {
    id: "revops",                          // new id → appended
    label: "Nadia — VP of RevOps",         // "Name — role" (the " — " is required)
    segment: "team",
    role: "VP of Revenue Operations at a scale-up",
    voice: "Cares about pipeline hygiene and clean handoffs; punishes vague AI claims.",
    disposition: { tech: 0.6, aiNative: 0.5, patience: 0.6, skeptic: 0.7 },
  },
  // Reuse a built-in id (e.g. "founder") to OVERRIDE that persona instead.
];
export const BURNED_CLAIMS = ["bank-grade", "military-grade"]; // unioned with the built-ins

Then in usertrial.config.mjs: export default { personas: "./usertrial.personas.mjs" }. Working inside Excellent? scripts/generate-usertrial-personas.mjs writes this module straight from your market_icp_profiles segments.

Invariants (the rails)

• Never touches the live vault except to file findings. Every trial runs in a disposable EXCELLENT_DATA_DIR; the recorder is the only writer to your real workspace, and it's fail-closed against implicit live-DB writes.
• Fresh means fresh. A persona starts at onboarding, never in a seeded world.
• Scripted stays green offline. No key, no network needed.
• We only take credit/blame for problems we solve. Out-of-scope jobs are reported as out-of-scope, not failures.
• Findings are reversible + attributable. Idempotent by marker; a fix closes only when a different agent verifies it through the gate.
• Zero runtime deps. System Chrome (found, never bundled); the mcp bin resolved find-first.

License

MIT © Excellent