日本語版はこちら

AI agents that experience your app — and help it grow.

Your test suite checks the things someone already thought to check. The bugs that reach users are usually in the paths nobody wrote a test for: the flow that works but is confusing, the empty state nobody designed, the admin screen that breaks when a field is blank.

shoal drops a swarm of AI agents onto a running web app. Each has a distinct persona and explores as a real user would — navigating, taking actions, noticing friction — then triage merges what they found and files it as issue tickets. No test scripts, no test data, no prior knowledge of the app. Just a URL.

npm install -g @m8i-51/shoal && shoal init && shoal

Who it's for

Teams with a staging or preview environment who want exploratory coverage they do not have time to write by hand — a weekly pass over the whole app that files what it finds, in front of the people who can fix it.

What it will not do

Worth knowing before you install it:

• It costs real money per run. Every agent turn is an LLM call. SHOAL_MAX_USD caps a run, and shoal doctor warns when your model has no published price and the cap therefore cannot fire.
• It finds some of the bugs, not all of them. On shoal's own benchmark — a sample app with seeded bugs and ground-truth labels — the measured detection rate is 71% (7 seeded bugs, claude-sonnet-4). See shoal-bench; you can run it yourself.
• Two runs differ. Agents are LLM-driven, so this is exploration, not a regression suite. Coverage is tracked across runs so successive sessions push into unexplored areas rather than repeating each other.
• Some findings will be wrong. Triage merges duplicates and skips what matches an existing issue, but a human still decides. The bench reports precision alongside detection for exactly this reason.
• It never reads your code. It only sees the app the way a user does. That is the point, and it is also the limit: it can tell you a flow is broken, not which commit broke it.

It complements a deterministic test suite. It does not replace one.

How it works

Target App (any URL)
        │
        ▼  autonomously learns what the app does + its goals
  Product Discovery
        │
        ▼  generates a user persona team for that app
  Org Design
        │
        ▼  creates and maintains the agent roster
  HR Agent
        │
        ├──────────────────────────────────┬──────────────────┐
        ▼                                  ▼                  ▼
  API Agents  ×N                   Browser Agents  ×N   Threshold Agents ×M
  explore via API                  browse the real UI   probe boundaries
        │                                  │                  │
        └──────────────────┬───────────────┴──────────────────┘
                           ▼  deduplicates and files issue tickets
                     Triage Agent

Each agent carries a distinct perspective — accessibility, security, business logic, UI design, new user experience, and more. They operate on a shared understanding of the app's purpose and goals. Coverage is tracked across runs, so each session naturally focuses on areas that haven't been explored yet.

What it finds

At the end of each run:

• Bugs — broken flows, errors, inconsistent data
• UX issues — confusing interactions, dead ends, unclear states
• Feature suggestions — things that would add real value
• Goal gaps — where the app falls short of what it's trying to achieve

Findings are filed as issue tickets (GitHub Issues, Jira, Notion, Backlog, or Asana) or saved as a self-contained HTML report. A web dashboard lets you start runs, watch live progress, review findings by category, see what triage filed, and track adoption and LLM cost across runs.

Product Edge — findings that would sand the product down

A swarm of personas each asking for what their kind of user expects is also a machine for making a product average. Fix every ticket and the sharp thing you built converges on the conventional version of its category.

So shoal lets you declare what the product is deliberately sharp about — and what it gives up to stay that way:

Sharp edges      Every flow is keyboard-first — no mouse path is provided, on purpose
Trade-offs       No onboarding wizard — the product assumes a trained operator

Product discovery drafts this from the app itself; you correct it in the dashboard's Product Edge panel, and your version survives re-discovery.

Triage then files findings exactly as it would otherwise — nothing is dropped or softened — but labels an issue edge-risk when the obvious fix would blunt a declared edge, with the edge at stake and what would be lost written into the ticket:

⚠️ Edge risk — decide before fixing

• Edge at stake: Every flow is keyboard-first — no mouse path is provided, on purpose
• Why the obvious fix would blunt it: Adding the drag-and-drop this persona expects makes the keyboard flow optional, and optional flows stop being maintained.

Bugs are never marked this way: a defect is a defect whatever the positioning, and that rule is enforced in code, not just asked for in the prompt. Without a declared edge the whole mechanism stays off.

Severity

Every filed issue carries a severity, judged from the impact the agent actually observed rather than from how hard a fix looks — the agent cannot see your code:

| | | | --- | --- | | critical | blocks the core task entirely, loses data, or exposes something it should not | | major | completable only via a workaround a real user would likely give up before finding | | minor | noticeable friction or a wrong detail that does not stop the task | | trivial | cosmetic, or an improvement nobody is currently blocked by |

It is written into the issue body and applied as a severity:<level> label, so a tracker can filter on it. Severity is independent of category: a ux finding can be critical and a bug can be trivial. When triage returns a level that is not one of the four and cannot be mapped from a common synonym (blocker, high, P2…), the issue is still filed — with no severity at all rather than a guessed one, so a team sorting by severity never mistakes a fabricated level for a real one.

Quick Start

Install globally:

npm install -g @m8i-51/shoal
npx playwright install chromium

Move to the directory that should hold shoal's config, then run:

cd your-project          # or a subdirectory — see [Where config lives](#where-config-lives)
shoal init               # creates .env with all available options

Open .env and set at minimum:

ANTHROPIC_API_KEY=sk-ant-...
BASE_URL=http://localhost:3000   # URL of the app to explore

Then run from that same directory:

shoal doctor   # check the setup before spending a run on it
shoal serve    # open web dashboard at http://localhost:4000
shoal          # or run agents directly from the terminal
shoal config   # update settings in existing .env (e.g. issue trackers)

shoal doctor makes no LLM call and costs nothing. It checks the Node version, whether .env exists and is readable only by you, that the provider has the credential it needs, that the model is one SHOAL_MAX_USD can actually be enforced against, that a Playwright browser is installed, that BASE_URL is a usable URL, and that any enabled tracker is fully configured. It exits non-zero only when something would genuinely stop a run, so CI can gate on it:

  ✓ Node.js             v22.22.2
  ✓ LLM credentials     ANTHROPIC_API_KEY set
  ! Spend cap           SHOAL_MAX_USD=$5 is set, but no price is known for "x/unlisted" — the cap cannot fire
                        → use a model with published pricing, or treat the run as uncapped
  ✗ Target app          BASE_URL is not set
                        → set BASE_URL to the app you want explored

On startup shoal prints which .env it loaded (or that it found none). If you see 0 variables injected, you are not in the directory that contains .env.

Or clone and develop locally:

git clone https://github.com/m8i-51/shoal
cd shoal
npm install && npx playwright install chromium
cp .env.example .env   # set ANTHROPIC_API_KEY and BASE_URL
npm start

Web dashboard

shoal serve        # global install
# or
npm run serve      # from cloned repo

Opens at http://localhost:4000. From there you can:

• Start a run — configure agent count, target URL, and custom instructions
• Watch agents swim live — the Swarm tab shows an animated real-time view of agents as they explore. When a finding is discovered, the agent's chip flashes with the finding title.
• Review past runs — findings by category, agent count, duration, and estimated cost
• See what triage did — the Triage tab of a run shows the issues it actually filed: which findings were merged into each one, the tracker link, why a finding was skipped, and which tickets were flagged edge-risk along with the declared edge at stake
• Track finding adoption — how many filed issues the team actually fixed versus closed as not planned, broken down by lens and category. This is the signal shoal already feeds back into persona hiring; the panel makes it visible.
• Track LLM cost across runs — cumulative spend, per-run average, the last 30 days, and total tokens, alongside a per-run trend
• Generate an Agent Diary — after a run completes, one LLM call turns the raw log into a story-style narrative of the exploration, readable by anyone on the team
• Hall of Issues — browse all findings across every run with full-text search and category filter. Export as JSON to share, or paste a GitHub raw/gist URL to import findings from other projects.
• Edit app goals — guide the goal-gap detector by defining what the app should achieve
• Declare the product edge — name what the product is deliberately sharp about, so triage flags tickets whose fix would flatten it (see Product Edge)
• Schedule a weekly run — pick a day and time directly in the dashboard for automatic recurring runs (the shoal serve process must stay running; for a serverless alternative see Scheduled runs below)

Dashboard access

The dashboard can start runs against any URL your machine can reach, so it is treated as a control surface, not a viewer:

• It binds to 127.0.0.1 — only this machine can reach it. No token needed.
• To expose it (a container, a dev box you reach from elsewhere), set SHOAL_HOST=0.0.0.0. shoal refuses to start on a non-loopback address unless you also set SHOAL_ALLOW_INSECURE=1, because the listener is plain HTTP — exposure has to be a decision, not a typo in a container spec. A token then becomes required: set SHOAL_TOKEN, or shoal generates one and prints a ready-made ?token=… URL at startup.

shoal dashboard → http://0.0.0.0:4000
[auth] bound to 0.0.0.0, which is reachable from other machines — a token is required.
[auth] generated token: 6f1c…
[auth] open: http://0.0.0.0:4000/?token=6f1c…

The ?token=… URL is a bootstrap: the server exchanges it for an HttpOnly session cookie and the dashboard drops it from the address bar, so the token is never kept anywhere page JavaScript can read.

The listener itself is plain HTTP, so an exposed dashboard needs a TLS-terminating reverse proxy in front of it. An SSH tunnel (ssh -L 4000:localhost:4000 host) is usually a better answer than opening the port at all. See SECURITY.md for the full picture.

Behind a reverse proxy — the recommended shape keeps shoal itself on SHOAL_HOST=127.0.0.1 and lets the proxy (nginx, Caddy, Traefik) listen publicly and terminate TLS, forwarding to shoal over loopback. That needs one more variable: set SHOAL_ALLOWED_HOSTS to the proxy's public hostname (comma-separated for more than one), or every request fails shoal's own Host/Origin checks — a proxy that preserves the Host header sends one shoal doesn't recognize, and one that rewrites it to shoal's own address (nginx's default) still forwards the browser's real Origin unchanged, which then no longer matches. Setting SHOAL_ALLOWED_HOSTS also makes a token mandatory, the same as a non-loopback SHOAL_HOST.

Cross-run intelligence

shoal gets smarter with each run.

Diff exploration — after every browser navigation, shoal hashes the page content (SHA-256 of innerText). On the next run, agents that land on an unchanged page are nudged to move on: "page content unchanged since last run — consider exploring a different area." The hashes accumulate in cache/page-hashes/ and steer future agents toward parts of the app that have actually changed.

Finding hotspots — the persona designer has access to a get_finding_hotspots tool that aggregates findings by URL area across all past runs. It uses this to recruit agents toward under-investigated parts of the app, or to send specialists into zones where problems keep clustering.

Agent memory — each persona remembers its own experience from the last few runs: what it struggled with, what it reported, what it accomplished. On the next run it returns as a returning user — first revisiting what frustrated it (confirming improvements, or re-reporting with "still broken since my last visit"), then moving on to new areas. Findings gain the continuity of a real user relationship.

Returning-user sessions — each browser agent's storage state (cookies, local storage) is saved per persona in cache/sessions/ and restored on the next run. Agents come back to the app as the same user with the same session: still logged in, with the data they created last time. Scenario design includes one returning-user journey per run — resuming a draft, reviewing accumulated data — so lifecycle states (empty → populated, notifications, stale sessions) get tested the way real users hit them.

Multi-actor scenarios — real concurrency bugs live where two users touch the same data at the same time. When the Account Manager discovers two or more roles, the scenario designer creates one two-actor scenario per run — an admin revoking access while a user is mid-flow, two users editing the same record — and two browser agents play it out simultaneously. Each agent is paired to an actor by persona role (not roster order) and logged in as the matching test-account session, watching for stale data, silent overwrites, and permission changes that don't take effect mid-session.

Swarm signals — agents in the same run share a blackboard. A check_swarm_signals tool shows each agent what the others have just reported; when a signal matches the area an agent is in, it tries to reproduce the problem from its own perspective. Findings corroborated by multiple different personas merge into much stronger issues at triage.

Environment personas — a persona's environment is part of who they are. The persona designer can give recruits a real browsing environment — a phone (actual Playwright device emulation with touch and mobile viewport), a non-default locale, dark mode, reduced motion, or a slow 3G connection — matched to the persona's life. Mobile and accessibility findings come from actually experiencing the app in that environment, not from guessing. Browser agents also carry a run_a11y_audit tool (axe-core) that measures WCAG violations on the current page, so accessibility findings cite specific rules and elements as evidence.

Adoption feedback — when triage files an issue, shoal remembers which perspectives (lenses / scenarios) produced it. On later runs it checks how your team closed those issues — fixed counts as adopted, closed as not-planned counts as rejected — and feeds the adoption rates back into persona recruitment and scenario design. Perspectives whose findings the team acts on get recruited more; ones that keep getting rejected fade (but never disappear — a rejected finding may just be low priority).

Experience Score — a 0–100 health score of your app's user experience, tracked across runs. It blends three signals: scenario success rate (did agents accomplish realistic user tasks?), friction (how many steps it took), and regressions (did fixed bugs come back?). The score, its trend, and the delta against the previous run appear on the dashboard and at the top of each HTML report — so you can see at a glance whether the app is actually getting better.

All signals work passively — no configuration needed. They improve automatically as runs accumulate.

Where config lives

shoal reads .env, test-accounts/, shoal.config.ts, and writes logs/ / findings/ from the current working directory — not from the global install location, and not automatically from a monorepo root.

On startup:

[env] working directory: /path/you/ran/from
[env] loaded /path/you/ran/from/.env (12 variables)

If that directory has no .env:

[env] no .env found at /path/you/ran/from/.env (0 variables injected)

With 0 injected variables, shoal falls back to the default LLM provider (Anthropic) with no API key, which then fails. Logs also go to the working directory, so it is easy to miss that a subdirectory config was never read.

Putting config in a subdirectory (typical in a monorepo):

mkdir -p apps/shoal
cd apps/shoal
shoal init                 # writes apps/shoal/.env
# add test-accounts/accounts.json here too
shoal serve                # dashboard; logs and findings land in apps/shoal/

Or stay at the repo root and point shoal at that directory:

shoal serve --dir apps/shoal
shoal --dir apps/shoal
# load a specific file but keep cwd for logs:
shoal serve --env-file apps/shoal/.env

--dir also changes where test-accounts/, logs/, findings/, and shoal.config.ts are resolved. SHOAL_DIR and SHOAL_ENV_FILE are the env-var equivalents.

Configuration

| Variable | Default | Description | |---|---|---| | TARGET | none | Target config name (example | none | your custom name) | | BASE_URL | http://localhost:3000 | Target app URL | | MAX_EXPLORERS | 4 | API explorer agent count (0 to disable) | | MAX_BROWSERS | 2 | Browser agent count | | MAX_THRESHOLDS | 1 | Threshold agent count (0 to disable). Probes input/business/experience boundaries inferred by Product Discovery (browser-primary). Cached specs without thresholdCandidates need REFRESH_SPEC=1 | | ANTHROPIC_API_KEY | — | Required | | ISSUE_TRACKERS | — | Comma-separated list of active trackers: github, jira, notion, backlog, asana | | SHOAL_MODE | safe | Safety mode: read-only | safe | full (see below) | | SHOAL_DESTRUCTIVE_PATTERNS | — | Comma-separated extra regex sources (case-insensitive) appended to safe mode's destructive-click detector, on top of the built-in English and Japanese patterns. Invalid entries are skipped with a warning | | SHOAL_TRACE | 1 | Record Playwright traces of browser agent sessions (0 to disable). Each finding gets a trace chunk at save time (logs/traces/<run>/<findingId>.zip); the agent session trace remains at logs/traces/<run>/<agentId>.zip | | REFRESH_SPEC | — | Set to 1 to re-run product discovery | | SHOAL_RETENTION_DAYS | 30 | Days of logs/screenshots/run_* and logs/traces/run_* to keep; older run directories are deleted at the start of each run. 0 disables pruning | | SHOAL_MAX_USD | — | Hard spend cap for a run (estimated USD). Once reached, no further LLM call starts and the remaining lanes are skipped — findings already collected are still saved and reported | | SHOAL_LANG | — | Language agents write findings, issue titles and bodies in. Accepts a code (ja, pt-BR, zh-TW) or a plain name. Unset means the model decides from the app it is looking at | | SHOAL_LOG_LEVEL | info | How much shoal prints: silent, error, warn, info, debug. error is the quiet CI setting — progress and warnings go, but the run summary, dashboard URL and report path still print; only silent suppresses those | | SHOAL_HOST | 127.0.0.1 | Dashboard bind address. Loopback by default; set it to expose the dashboard (see Dashboard access) | | SHOAL_TOKEN | — | Dashboard token. Required whenever SHOAL_HOST is not loopback or SHOAL_ALLOWED_HOSTS is set; generated and printed at startup if unset | | SHOAL_ALLOWED_HOSTS | — | Comma-separated public hostname(s) of a reverse proxy in front of the dashboard (see Dashboard access) | | SHOAL_BROWSER_ITERATIONS | 12 | Turns a browser agent may take | | SHOAL_THRESHOLD_ITERATIONS | 12 | Turns a threshold agent may take | | SHOAL_EXPLORER_CONCURRENCY | 2 | API explorer agents run in parallel batches of this size | | SHOAL_VIEWPORT | 1024x640 | Browser viewport for agent sessions (a persona's device emulation overrides it) |

Safety modes — agents write data as they explore, so choose how much they're allowed to touch:

• read-only — no writes to the target app at all. Mutation requests (POST/PUT/PATCH/DELETE) from browser agents are blocked at the network layer. This only means shoal won't write to your app — everything agents read (page text, accessibility tree, console/network output, screenshots) is still sent to your configured LLM provider. See What is sent to the LLM provider before pointing it at production data.
• safe (default) — creating and editing test data is fine, but agents are instructed to stop before irreversible actions: deleting records, payments, sending emails or invitations. The destructive-click detector recognizes both English and Japanese phrasing (削除, 購入する, 決済, etc. — set SHOAL_DESTRUCTIVE_PATTERNS to add your own) and blocks matching clicks programmatically.
• full — no restrictions. Use only against disposable environments.

In safe and read-only modes, API tools marked destructive: true in your target config are removed from the agents' toolset. The mode can also be selected per run in the dashboard's start dialog.

Spend cap — set SHOAL_MAX_USD to stop a run once its estimated cost reaches a limit. Turn budgets bound how long each agent explores; this bounds what the whole run can cost.

Content from the target app is untrusted — agents read your app and act on what they read, so any text the app can display reaches the model. shoal fences everything it reads from the app and tells agents that fenced content is data, never instructions — but fencing is a mitigation, not a guarantee. Don't point an authenticated swarm at an app carrying content you don't control. See SECURITY.md.

Issue tracker variables (set only what you need):

| Tracker | Variables | |---|---| | GitHub Issues | GITHUB_TOKEN, GITHUB_REPO (owner/repo) | | Jira | JIRA_BASE_URL, JIRA_EMAIL, JIRA_API_TOKEN, JIRA_PROJECT_KEY | | Notion | NOTION_API_KEY, NOTION_DATABASE_ID ¹ | | Backlog | BACKLOG_SPACE, BACKLOG_API_KEY, BACKLOG_PROJECT_ID | | Asana | ASANA_ACCESS_TOKEN, ASANA_PROJECT_ID |

¹ The Notion database must have Name (title), Labels (multi_select), and Status (select) properties.

Multiple trackers can be active at the same time — findings are posted to all of them. If ISSUE_TRACKERS is not set but GITHUB_TOKEN and GITHUB_REPO are present, GitHub is used automatically (backward compatible).

Backlog projects do not share issue type or priority IDs. On each filing, shoal fetches the project's type and priority lists, matches shoal categories (bug / ux / feature-request / goal-gap) to names such as バグ・要望・タスク / 高・中・低, and asks the LLM only when no name matches. The chosen type and priority are logged ([backlog] selected issueType "バグ" (id=…) for category=bug).

Adding a target

shoal loads shoal.config.ts from the current working directory at startup. Two common setups:

Option A — config in your project directory (recommended)

# Copy the example from the repo (or create from scratch)
curl -O https://raw.githubusercontent.com/m8i-51/shoal/main/shoal.config.example.ts
mv shoal.config.example.ts shoal.config.ts
# Edit shoal.config.ts, then:
shoal

Option B — config inside the cloned repo (simplest for development)

cp shoal.config.example.ts shoal.config.ts
# edit shoal.config.ts, then:
npm start

shoal.config.ts exports a target object. For API explorer agents, include appTools and execute:

// shoal.config.ts
export const target = {
  appTools: [
    { name: "list_items", description: "Get all items.", input_schema: { type: "object", properties: {}, required: [] } },
  ],
  async execute(toolName: string, input: Record<string, unknown>) {
    if (toolName === "list_items") {
      return fetch(`${process.env.BASE_URL}/api/items`).then(r => r.json());
    }
  },
};

Alternatively, copy targets/example.ts, register it in targets/index.ts, and set TARGET=my-app.

appTools and execute are required only for API explorer agents. Login does not depend on them: test-accounts/accounts.json is enough, and a config that only sets credentials (or projectPath) is still applied.

MCP server — close the fix loop

shoal can act as an MCP server, so coding agents like Claude Code can drive the full find → fix → verify loop:

shoal mcp   # stdio transport

Register it in your agent's MCP config (e.g. .mcp.json):

{ "mcpServers": { "shoal": { "command": "shoal", "args": ["mcp"] } } }

Exposed tools:

| Tool | Purpose | |---|---| | start_run | Launch an exploration run (URL, agent counts, safety mode) | | get_run_status | Poll progress: findings so far, regression results, log tail | | list_findings | Read findings across runs, filtered by run / category / text | | verify_fix | Spawn a single verifier agent that retraces one finding's flow and reports fixed / still_broken / inconclusive | | get_experience_score | Cross-run Experience Score trend — did the fix actually improve the experience? |

A coding agent can pick a finding with list_findings, fix the code, redeploy, and call verify_fix to have an agent retrace the exact reported flow — closing the find → fix → verify loop without a human in the middle.

PR Experience Diff

Get per-PR feedback on how a change feels to users, not just whether tests pass:

shoal diff                    # diff vs origin/main
shoal diff --base origin/dev  # diff vs any ref

shoal diff maps the PR's changed files to routes (Next.js pages/app router and views//routes/ conventions), sends a small focused swarm (2 browser agents by default) to those areas of your preview deployment, and posts a summary as a PR comment — findings, plus the Experience Score delta:

Experience Score: 72/100 (▲5 vs previous run) Agents focused on the areas this PR touches: /checkout 🐛 [bug] Checkout button unresponsive — Nadia I tapped the checkout button and nothing happened.

If GITHUB_TOKEN isn't available the summary is saved to logs/diff_<runId>.md instead. For CI, shoal init's example lives at .github/workflows/shoal-diff.example.yml — it runs on every PR against your PREVIEW_URL.

Scheduled runs

To run shoal weekly against a staging environment, add a GitHub Actions workflow to your repo.

Run shoal init — it will offer to generate .github/workflows/shoal-weekly.yml automatically. Or copy the example from this repo:

curl -O https://raw.githubusercontent.com/m8i-51/shoal/main/.github/workflows/shoal-weekly.example.yml
mv shoal-weekly.example.yml .github/workflows/shoal-weekly.yml

Then add ANTHROPIC_API_KEY to your repo's Actions secrets (Settings → Secrets and variables → Actions).

The workflow runs every Monday at 09:00 UTC and can also be triggered manually from the Actions tab. Findings are filed as GitHub Issues using the built-in GITHUB_TOKEN.

shoal-bench

How well does the swarm actually detect problems? bench/ ships two sample apps with seeded bugs and ground-truth labels:

| Variant | App | Seeded bugs | Labels file | |---|---|---:|---| | store (default) | Tiny store with cart/admin/nav | 7 | bench/labels.json | | forms | Support ticket form | 3 | bench/labels-forms.json |

Each label includes lens, path, and category metadata for per-area scoring.

npm run bench                         # store variant
npm run bench:forms                   # forms variant
SHOAL_BENCH_MIN=60 npm run bench      # exit non-zero below 60% detection (CI regression gate)
BENCH_RECORD=1 npm run bench          # append model score to bench/scores.json

The scorer matches findings to labels and prints a detection report:

Detection rate: 5/7 (71%)
  ✓ cart-total-wrong
      └ "Cart total doesn't match item quantities"
  ✗ low-contrast (accessibility @ /) — The Buy button text is nearly the same color as its background

Published detection scores

Scores recorded with BENCH_RECORD=1 (see bench/scores.json):

| Variant | Model | Detection | Precision | Findings | Unmatched | Date | Config | |---|---|---:|---:|---:|---:|---|---| | store | claude-sonnet-4-20250514 | 71% | —¹ | 11 | —¹ | 2026-08-15 | MAX_BROWSERS=3, default prompts |

¹ Scored before the matcher required word boundaries (a substring matcher counted "alt" inside "Although" and "total" inside "Totally" as detections), and before precision/unmatched were recorded — not comparable to rows scored after that fix.

One configuration, one run. Read that number as a single sample, not a benchmark: detection rate varies run to run (agents explore non-deterministically), and nothing here says how the swarm behaves on the forms variant, on other models, or at other agent counts. Treat it as a regression baseline for this configuration until the table has more rows.

Contributing a score — run the bench against your provider and open a PR with the new bench/scores.json entry and matching table row:

BENCH_RECORD=1 npm run bench          # store variant
BENCH_RECORD=1 npm run bench:forms    # forms variant

Each run costs real LLM spend, so cap it: SHOAL_MAX_USD=2 BENCH_RECORD=1 npm run bench. Note the model and agent counts in the config field — a score without its configuration cannot be compared against anything.

Use it as a regression test when changing prompts, models, or exploration logic — and don't fix the seeded bugs (the app's test suite pins them in place).

Account Manager

For apps that require login, shoal includes an Account Manager agent that tests credentials and injects session state into explorer agents so they can reach authenticated routes.

Create test-accounts/accounts.json (gitignored) with your test credentials. That file alone is enough — target.credentials in shoal.config.ts is optional:

[
  { "email": "[email protected]", "password": "testpassword", "role": "user" },
  { "email": "[email protected]", "password": "adminpassword", "role": "admin" }
]

On startup shoal reads this file and runs Account Manager: it logs in with each account on the login URL found during product discovery (not only BASE_URL), saves Playwright session state, and passes those sessions to browser agents. If a seed admin is available (from accounts.json or target.credentials), it also explores user management and tries to create one test account per role.

If session injection fails, browser agents are not left to invent logins. They either receive the accounts.json credentials and the discovered login path, or they explore as a guest with an explicit instruction not to guess usernames or passwords.

Startup logs always report whether accounts.json was found, whether config credentials were set, and why Account Manager started or was skipped.

shoal.config.ts appTools and execute are required only for API explorer agents, not for login. A config that has credentials (or projectPath) but no tools still applies those fields.

LLM providers

shoal defaults to Anthropic Claude. To use a different provider, set these variables in .env:

| Provider | Variables | |---|---| | Anthropic (default) | ANTHROPIC_API_KEY | | Amazon Bedrock | LLM_PROVIDER=bedrock, AWS_REGION (keys optional — default credential chain) | | OpenAI | LLM_PROVIDER=openai, LLM_API_KEY, LLM_MODEL | | OpenRouter | LLM_PROVIDER=openrouter, LLM_API_KEY, LLM_MODEL | | Groq | LLM_PROVIDER=groq, LLM_API_KEY, LLM_MODEL | | Gemini | LLM_PROVIDER=gemini, LLM_API_KEY, LLM_MODEL | | Codex (ChatGPT subscription) | run npm run auth:codex once, then LLM_PROVIDER=codex | | Claude CLI (Claude Code subscription) | npm install @anthropic-ai/claude-agent-sdk (optional peer dependency, not installed by default — see note below), then run npm run auth:claude once and set LLM_PROVIDER=claude-cli | | Ollama | LLM_BASE_URL=http://localhost:11434/v1, LLM_MODEL | | LM Studio | LLM_BASE_URL=http://localhost:1234/v1, LLM_MODEL |

claude-cli is the only provider with an extra install step: @anthropic-ai/claude-agent-sdk is ~200MB and carries Anthropic's own licence (not OSI-approved — "SEE LICENSE IN README.md"), so it ships as an optional peer dependency rather than a hard one. Every other provider works with a plain npm install @m8i-51/shoal.

Codex (ChatGPT subscription)

Use your ChatGPT Plus / Pro subscription through the official Codex CLI login.

Prerequisites

• npx available (the official @openai/codex CLI is run via npx @openai/codex login, no separate install needed)
• A ChatGPT Plus / Pro (or equivalent) subscription

Setup

npm run auth:codex

This runs the official Codex CLI's OAuth login flow, verifies ~/.codex/auth.json was created, and writes LLM_PROVIDER=codex and a default LLM_MODEL to .env.

Run

Same as usual:

shoal
# or
npm start
shoal serve

Terms note

Unlike claude-cli (below), which only launches the official Claude Code CLI and never touches its OAuth tokens, codex reads the access and refresh tokens from ~/.codex/auth.json directly, refreshes them itself when they expire (rewriting that file), and calls ChatGPT's undocumented chatgpt.com/backend-api/codex endpoint using the official Codex CLI's own client id. This mirrors the OAuth flow the official @openai/codex CLI uses, but shoal — not that CLI — is the one making the API calls: an integration pattern OpenAI has not published or endorsed for third-party tools, and one that can break without notice if the endpoint or its authentication changes. For hosting or redistributing shoal, prefer an OpenAI API key (LLM_PROVIDER=openai) instead.

Troubleshooting

• Login fails → run npx @openai/codex login directly to see the underlying error
• ~/.codex/auth.json not found after login → the official CLI's login did not complete; retry it directly before re-running npm run auth:codex

Claude CLI (Claude Code subscription)

Use your Anthropic Free / Pro / Max subscription through the official Claude Code login (same pattern as xangi / OpenClaw). shoal never reads or stores OAuth tokens.

Prerequisites

• Claude Code installed with claude on your PATH
• A Claude Pro / Max (or Team / Enterprise) subscription

Setup

npm run auth:claude
# or:
#   claude auth login
#   # set LLM_PROVIDER=claude-cli and LLM_MODEL=claude-sonnet-4-6 in .env

auth:claude verifies login and writes LLM_PROVIDER=claude-cli to .env.

Run

Same as usual:

shoal
# or
npm start
shoal serve

API key vs subscription

• Prefer subscription quota: remove ANTHROPIC_API_KEY from .env and your shell (if set, Claude Code may prefer pay-as-you-go API billing)
• Prefer API key only (classic): set ANTHROPIC_API_KEY and leave LLM_PROVIDER unset or anthropic

Terms note

Anthropic does not allow third parties to intermediate Claude.ai login or Free/Pro/Max credentials in their own API clients. claude-cli only launches official Claude Code / the Agent SDK after you authenticate yourself. For hosting or redistributing shoal, prefer API keys or Bedrock.

Troubleshooting

• claude not found → install Claude Code and confirm PATH in a new shell
• auth status fails → run claude auth login again
• Permission prompts → shoal disables Claude Code built-ins and only allows shoal tools; update Claude Code and retry

Amazon Bedrock

Leave AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY unset to use the default AWS credential chain (shared credentials file, named profile, SSO cache, or instance role). You do not have to run aws sso login if the machine already has long-lived keys in the default profile — short-lived SSO login is often rejected in that situation. Use the existing keys (or a dedicated profile) instead.

Do not put empty AWS_ACCESS_KEY_ID= / AWS_SECRET_ACCESS_KEY= lines in .env. An empty value overrides the credential chain and authentication fails. shoal strips empty AWS keys at startup and logs a warning.

Set LLM_MODEL to a Bedrock model ID or an inference profile ID. Not every model generation is available for in-region invoke or for every geographic profile. If you need data to stay in-country, pick a generation that exists as a geo profile in that region:

| Scope | Example LLM_MODEL | Typical AWS_REGION | |---|---|---| | Foundation model (on-demand, when offered) | anthropic.claude-haiku-4-5-20251001-v1:0 (default) | region that hosts the model | | US cross-region | us.anthropic.claude-sonnet-4-5-20250929-v1:0 | us-east-1 | | EU cross-region | eu.anthropic.claude-sonnet-4-5-20250929-v1:0 | eu-central-1 | | APAC cross-region | apac.anthropic.claude-sonnet-4-5-20250929-v1:0 | ap-northeast-1 | | Japan (Tokyo + Osaka only) | jp.anthropic.claude-sonnet-4-5-20250929-v1:0 | ap-northeast-1 | | Japan | jp.anthropic.claude-haiku-4-5-20251001-v1:0 | ap-northeast-1 | | Japan | jp.anthropic.claude-sonnet-4-6 | ap-northeast-1 |

List profiles in your account with aws bedrock list-inference-profiles. Copy-paste examples live in .env.example.

License

MIT