@perf-labs/agent

The perflab host CLI. It runs the customer's own test command several times, collects the perf-lab data the app wrote during each run, turns it into metrics, and asks @perf-labs/stats for a verdict. v1 is local only and measure-only: no code changes in the customer app, no driver of our own (the test command is Maestro, Appium, XCUITest, anything that exits 0 on success).

Commands

bun install && bunx tsc --noEmit && bunx vitest run
bun run build                      # dist/src/cli.js (bin: perflab)

perflab measure --repeat 10 \
  --arms A=/builds/base.app,B=/builds/candidate.app \
  --platform ios-sim [--device <udid>] [--bundle-id com.x.app] [--out runs/s1] [--timeout 600] \
  [--ri-capture observations.ndjson] [--mem-window chat.open] \
  -- maestro test flows/open-chat.yaml
perflab verdict runs/s1 [--baseline A --candidate B]   # table + verdict.json
perflab extract runs/s1 --ri-capture observations.ndjson   # re-derive metrics.csv (adds ri.* / mem.*)
perflab aa runs/s1 [--reps 300 --boot 400]             # noise floor / MDE per metric
perflab doctor                                         # xcrun, simctl, devicectl, maestro, appium, node, booted sims, devices

• --arms: NAME=<.app/.ipa path> (installed with simctl install / devicectl device install app, only when the arm changes) or NAME=<bundle id> (already installed). Without --arms there is one arm A: plain repetition, usable for perflab aa.
• --arm-env 'A:KEY=VAL' (repeatable): extra env for the test command on that arm's runs, also exported as MAESTRO_KEY (Maestro forwards only MAESTRO_* to flows). Used for same-binary arms that differ only in knobs, e.g. --arms A=dev.perflab.reference,B=dev.perflab.reference --arm-env 'A:REF_KNOBS=…' --arm-env 'B:REF_KNOBS=…'. The arms with their env are written to <out>/arms.json. Order stays ABBA.
• --platform ios-device needs --device <udid>. The simulator default target is booted.
• The test command runs as argv (no shell) with PERFLAB_RUN_ID, PERFLAB_ARM, PERFLAB_RUN_DIR, PERFLAB_MARKER_URL and MAESTRO_PERFLAB_MARKER_URL / MAESTRO_PERFLAB_RUN_ID in its environment.
• Declared steps (Maestro): call maestro/perflab/step.yaml with STEP/PHASE (start|end) from your flow; measure adds step:<name>.* rows (source=declared). --no-markers turns the endpoint off, --clock-offset-ms overrides the clock estimate. See docs/research/28-step-markers-prototype.md.

perflab run and perflab.yml (customer entry point)

perflab run is the wrapper for the e2e command you already have (mode A in docs/research/23-execution-modes-and-data-flow.md). It runs the command N times, drops runs where it failed, pulls the app's perf-lab data after each run, derives metrics and writes the out dir. With --baseline <earlier out dir> it also says what changed. Nothing is built or installed by it: your CI supplies the build (with PERFLAB=1) and the device.

perflab run                                   # command built from perflab.yml
perflab run -- maestro test --include-tags perf e2e/flows/   # wrap your own command (config still gives bundle id, device, repetitions)
perflab run --baseline main-out --upload      # compare with the baseline run, then send results to the ingest

Flags: --config (default perflab.yml), --repeat N, --out DIR, --baseline DIR, --upload [--ingest-url URL], --wait, --fail-on-worse, --commit SHA / --branch NAME (default GITHUB_SHA / GITHUB_HEAD_REF on a pull request, else GITHUB_REF_NAME), --base-commit SHA, --device-model / --device-os (default: read from the booted simulator), --scenario NAME (flow name for a command given after --), --replay <out dir>[:ARM] (repeatable: take recorded runs instead of running anything), --no-markers, --clock-offset-ms.

Commit details: when a run set opens (perflab run --upload, perflab farm open), the CLI reads the subject, author name and committer date of --commit and --base-commit with git log in the checkout (argv only, 5 s timeout) and sends them, plus the pull request number and URL from GITHUB_EVENT_PATH on a pull_request event. The dashboard shows commits as title, author and time instead of the bare hash. Without git, a shallow clone that lacks the commit, or outside a pull request, the fields are left out and nothing fails. On a GitHub pull_request event GITHUB_SHA is the merge commit, so its title is "Merge … into …"; pass --commit ${{ github.event.pull_request.head.sha }} (with the head commit fetched) to get the branch's own commit.

Journey titles: on --upload the run set carries journeys[] — per flow its key (the file stem), source_path (the flow file relative to the git repo root) and a title and description: from the {path, title, description} form of flows.paths, else the flow's own name: (when it differs from the stem) and the first paragraph of the comment that opens the file. Given values update the journey on the dashboard; absent ones keep what is stored.

Flows: with runner: maestro every flow file is run on its own (one maestro test per file per repetition, tag filters applied) and its file stem is the journey name (meta.scenario, plan D1). A directory in flows.paths means the flow files directly in it (not config.yaml). A command after -- or flows.command is one flow, named by --scenario, its only .yaml argument, or e2e. With several flows, run ids get the flow as a suffix and runs.ndjson records scenario.

Annotated example: examples/perflab.yml. Every setting:

| Setting | Meaning | |---|---| | project.key_env | Name of the env var with the project key (default PERFLAB_KEY). Only read with --upload. | | project.ingest_url | PerfLab API base URL for --upload (or --ingest-url, or env PERFLAB_API). Defaults to the hosted PerfLab API. | | app.bundle_id | Required. App whose perf-lab log is read after each run. | | platform, device | ios-sim (default; device = UDID or booted) or ios-device (UDID required). One device per invocation in v1. | | repetitions | Runs of the whole flow set (default 5). | | baseline_branch | Branch the baseline comes from (default main). Recorded for CI; the CLI compares with the directory you pass to --baseline. | | flows.runner | maestro (build maestro test ...) or command (run flows.command). | | flows.tags, flows.exclude_tags | --include-tags= / --exclude-tags=. | | flows.paths | Files, directories or globs (*, ?, **); expanded by perflab (no shell). With only tags the path defaults to .. An entry can also be {path, title, description} (one flow file): how the dashboard names the journey. | | flows.command | Any command that exits 0 on success (Appium, Detox, ...); text or list, run without a shell. | | flows.maestro_version | Note only: printed at the start of the run. | | steps.rename | old: new names applied to metrics (old.dur_ms becomes new.dur_ms, step:old.* too), so history keeps stable names. | | steps.pin | Names always listed in the summary. | | output.dir | Out dir (default perflab-out): runs.ndjson, metrics.csv, <run>/test.log, <run>/perf-lab.log, <run>/markers.ndjson, and with a baseline compare/metrics.csv + compare/verdict.json. | | timeout_seconds | Per run of the command. |

Unknown settings and wrong types are errors, all listed together.

Results and exit codes: results never change the exit code by default (ruling D9, no blocking PR checks): 0 also for "More work". --fail-on-worse (needs --wait) makes "More work" exit 3. The CLI exits 2 for a configuration problem (bad or missing perflab.yml, flags, a glob matching nothing, --wait without a branch or key) and 1 for an infrastructure problem (cannot read the device log, baseline dir without metrics.csv, an API request failing; results stay in the out dir). Failed runs are marked invalid in runs.ndjson / metrics.csv and ignored by the statistics. The ending summary reads like:

7/8 runs valid (dropped: 1 test-failed: exit 1)
Compared with main-out: 42 metrics
Top changes:
  More work chat.open: renders  +100% (100 -> 200)
More work: 1  Less work: 0  No change: 38  Not enough data yet: 3

--upload (off by default) needs a project key with run scope in the env var named by project.key_env (default PERFLAB_KEY); the API URL defaults to the hosted PerfLab API. With a branch it opens a run set (POST /v1/run-sets) before measuring, so a wrong key fails before device time is spent; then sends each valid run: POST /v1/runs (header x-perflab-key, meta.scenario, meta.run_set_id), POST /v1/windows with the run's own PERF_LAB window log lines (run token, 200 per batch, deterministic batch ids so a retry is safe), POST /v1/markers with its declared step markers, POST /v1/runs/:id/finish. A 429 is retried after retry-after. Then it closes the run set:

• with --wait: POST /v1/run-sets/:id/complete computes the verdicts inline and the CLI prints them in plain words (More work / Less work / No change / Not enough data yet per journey and device, the top changed checks, "Few runs — result may be noisy" when the server flags it) plus a dashboard link that opens the comparison;
• without --wait: the set is closed with verdicts: on-read and the CLI exits 0 at once, printing the dashboard link; the verdicts are computed when someone looks.

On the baseline branch (main by default, set per app on the server) the runs grow each journey's history; on any other branch they are compared with it. Without a branch the runs are uploaded as before, without a run set and without a verdict. The CLI writes <out>/perflab-result.json (run_set_id, dashboard_url, state, summary) and <out>/perflab-run.json (device, branch), and on GitHub Actions appends the result to $GITHUB_STEP_SUMMARY. Ready workflow: examples/github-actions.yml.

perflab verdict <out> --baseline remote [--branch main] [--limit N] compares each flow's local runs with the server baseline of the same flow and device (GET /v1/baselines: the history of the baseline branch, inside the same window the dashboard uses), for the metrics the server judges.

perflab farm (device farm, the app pushes)

For a farm where the CLI cannot reach the device (BrowserStack App Automate; plan 36). The CLI only opens and closes; the app (SDK activated per iteration with a run token) pushes its windows to the API itself, and the step script posts its markers there directly. Ready workflow and Maestro flows: examples/farm/.

perflab farm open --api https://perflab-api-….run.app --branch main --commit "$APP_COMMIT" \
  --device-model 'iPhone 15 Pro' --device-os 17 --device-provider browserstack \
  --journeys e2e-community-and-live-chat --repeat 5 \
  --tokens-js farm-run/perflab-tokens.js --state farm-run/perflab-farm.json
# ... the farm session runs (browserstack-run.mjs --suite-overlay "flows/perflab/tokens.js:farm-run/perflab-tokens.js")
perflab farm close --state farm-run/perflab-farm.json --bs-build farm-run/browserstack.ios.json --settle-seconds 60 --wait

perflab farm run does the whole night in one command (docs/plans/54a): resolve the app (--app eas:<id>|file|bs://), farm open, an optional BrowserStack Local tunnel (--tunnel-for localhost:<port>, repeatable), the device run (--driver maestro with --suite-dir/--execute, or --driver appium --journey-module FILE for the built-in Appium driver, @perf-labs/appium-driver, docs/plans/54b; --driver-script FILE still runs your own driver), farm close, and cleanup (tunnel stopped, tokens file reset) whatever happened. --env KEY=VALUE values may use {tunnel:<port>}, {run_id}, {out}, {tokens_js}, {run_set_id}. Exit codes as farm close; a failed device run is 1. --dry-run prints the plan with no request. Needs PERFLAB_KEY, BROWSERSTACK_USERNAME, BROWSERSTACK_ACCESS_KEY (and EXPO_TOKEN for eas:).

Setup and completion (plan 51 stage 2): --completion-marker declares that the flow posts the step marker run:complete (end) at the end of every iteration, from the main flow and not from onFlowComplete; the API then marks a run without it invalid (flow-incomplete). --driver-version, --app-build-id, --harness-version, --flag-set go into every run's setup fingerprint; farm close adds the device farm's build id from --bs-build.

farm open also sends journeys[]: with --config perflab.yml and/or --flows <files|dirs|globs,...> each --journeys name is matched to the flow file named like it (title, description, source path as for perflab run); without them, keys only.

farm open makes a run set (source: farm, the device of --device-model/--device-os: the journey history is keyed by it, not by what the phone reports) and one run per journey per iteration, ahead of any data, with run tokens valid for --ttl-seconds (default 21600 = 6 h). The key (PERFLAB_KEY, or --key-env) needs run scope and never leaves the CI runner. It writes:

• --tokens-js: the only file holding run tokens. BrowserStack caps env values at 100 characters, so the tokens travel inside the uploaded suite; the dispatcher overlays this file at e2e/flows/perflab/tokens.js for one upload. ES5, for Maestro's runScript:

  output.perflabApi = "https://perflab-api-….run.app";
  output.perflabRunSet = "<run set id>";
  output.perflabTokens = {"<journey>": ["pl_rt_…", …one per iteration]};

  next-token.js (examples) hands out perflabTokens[JOURNEY][i] as output.perflabToken for iteration i.
• --state: JSON for farm close, no token and no key (safe to upload as an artifact): version, api, run_set_id, dashboard_url, branch, commit, base_commit, device, journeys, repeat, tokens_js, opened_at, expires_at, runs[] (order, journey, iteration, run_id, external_id), then closed_at and result.
• On GitHub Actions: ::add-mask:: for every token, and outputs run_set_id, dashboard_url, state, tokens_js, runs.

farm close reads the state and:

1. waits until no batch arrives for --settle-seconds (default 60), at most --max-wait-seconds (default 300);
2. reads --bs-build (the App Automate build detail browserstack-run.mjs writes). When the session failed, the iteration it stopped in is finished as invalid (device-session-<status>): the first run without data after the last run with data. A measured iteration sends nothing until its end-of-iteration flush, so a run with data completed and is never blamed; a Maestro failure stops the session, so later iterations have no data;
3. finishes every run not yet finished (a run without a single batch is reported as "no data", never judged), closes the set (--wait: verdicts inline; else on read) and prints the iterations and the same plain-words verdict as perflab run --upload --wait, plus $GITHUB_STEP_SUMMARY and <out>/perflab-result.json (--out, default the state's directory);
4. puts the empty stub back into the tokens file, also when a step above failed. The example workflow also deletes the file before uploading $OUT_DIR.

Running it again is safe: finished runs are left alone, the set is not reprocessed, the same result is printed. A batch that arrives after a run was finished is added by the server and the run re-processed at once. Exit codes: 0 for any result, 3 with --fail-on-worse on "More work", 2 for configuration, 1 when the API fails or nothing could be judged ("Blocked": no iteration sent any data).

Slack: set PERFLAB_SLACK_WEBHOOK (an incoming-webhook URL kept as a CI secret) and farm close --wait posts the result: a pull request only when something got worse (a table of the 3 biggest changes, main vs PR), main only when something changed (PERFLAB_SLACK_ALWAYS=1 also posts "no change"), and a measurement that sent no data always. The commit's title and author come from git log in the checkout. A Slack failure is printed and never changes the exit code.

Declared steps on the farm: maestro/perflab/step.yaml / step.js / send-markers.js (copy them into the suite). With a token set, step.js only records the marker; send-markers.js, run in end-run.yaml after each iteration, posts them to <output.perflabApi or PERFLAB_API_URL>/v1/markers with Authorization: Bearer <run token>, so the test never waits on the network between actions (and sends the host's clock round trips, plan 51 G1.4). A failed iteration never reaches end-run.yaml, so the example run-perflab.yaml also runs send-markers.js from onFlowComplete; such a run is invalid through its missing completion marker, and its markers stay for diagnosis. The example folder (examples/farm/flows/perflab/) ships its own copy of send-markers.js. Without a token, step.js posts to the local CLI endpoint at once; there a step started but never ended invalidates its run (unclosed-steps).

Detox (Jest): packages/detox (@perf-labs/detox) gives step(name, fn) / stepStart / stepEnd, posting the same markers: to the local endpoint from PERFLAB_MARKER_URL + PERFLAB_RUN_ID (set by perflab run/measure), or on the farm to PERFLAB_API_URL/v1/markers with PERFLAB_RUN_TOKEN as the bearer token. Optional per-test step names via @perf-labs/detox/auto-steps. See its README.

Farm e2e against a local API: PERFLAB_KEY=… PERFLAB_INGEST_KEY=… e2e/farm/run.sh <new out dir> (fake session using the SDK push client, one late batch, one empty iteration).

perflab findings and friends (read-only questions)

Ask PerfLab what it found, without running anything (plan 48 §4). Every answer has the same shape: what changed, where (journey, phone, branch, commit title and author, PR), how sure (runs per side, session spread, "measured differently — not blocking"), why (components that rendered longer and their cause, counts that moved), what to do (a prompt for a coding agent), and since when it is seen. The MCP server @perf-labs/mcp gives coding agents the same answers with the same words (packages/mcp/README.md).

export PERFLAB_KEY=pl_rk_…        # a project key with read scope (a CI key with run scope works too)
perflab findings --branch feat/perflab-integration     # what got worse or better on a PR, worst first
perflab findings --main --journey live-ui --phone "iPhone 15"
perflab findings --state all --json                    # also the resolved and not judged ones, as the API's JSON
perflab findings --state all --offset 200              # the next page (at most 200 per answer)
perflab finding f_0123456789abcdef                     # one finding with its evidence
perflab finding f_0123456789abcdef --prompt | pbcopy   # only the hand-off for a coding agent, ending with its id and page
perflab compare <runA> <runB>                          # two runs (ids or external ids), window by window, no statistics
perflab history live-ui "Tap: live chat send"          # one window on main, run by run (id or name)
perflab commits a1b2c3d c3d4e5f [--branch main]        # the measured commits between two points
perflab layers                                         # which layer answers which question, and its certificate

Flags: --api URL (or PERFLAB_API, default the hosted API), --key-env NAME (default PERFLAB_KEY), --json. A finding's id is stable: the same window, measure, journey and phone model keep the id across commits and PRs, so perflab finding <id> after a fix tells whether it is resolved. --state (default present) filters on the state on the asked branch: present (in the newest run set), resolved (the newest run set judged the measure: No change, or the other way), not-judged (the newest run set did not judge it: not measured, unstable, not enough data; it may still be there). A list starts with whether and when the branch was measured ("No PerfLab measurement on branch X", or the newest measured commit, its word and runs), so an empty list on an unmeasured branch does not read as clean. commits lists each measured commit once, with how many sessions measured it. Words come from the one word table (Worse / Warning / Better …), "Worse (not blocking)" when the branch was measured differently from main.

Output layout

<out>/arms.json              arms of a `measure` (name, target, per-arm env)
<out>/runs.ndjson            one record per run: run_id, arm, block, t0, t1, exit_code, valid, invalid_reason
<out>/<run_id>/test.log      stdout/stderr of the test command
<out>/<run_id>/perf-lab.log  Documents/perf-lab/log.txt pulled from the app
<out>/metrics.csv            long format: run_id,arm,block,metric,value,valid,invalid_reason
<out>/verdict.json           written by verdict / aa

Architecture

 cli.ts ──► runner.measure ──► schedule (ABBA)
               │  per run:
               ├─ AppInstaller ── IosSimulatorInstaller | IosDeviceInstaller
               ├─ Spawner.run(test command)            t0 .. t1, exit code
               ├─ DataSource.fetchLog ── IosSimulatorDataSource | IosDeviceDataSource
               └─ window-log (parse, slice by time) ► metrics (per-window medians) ► metrics.csv
 cli.ts ──► verdict / aa ──► @perf-labs/stats (compareRuns / aaRuns) ──► table + verdict.json

Pure core (schedule, window-log, metrics, table formatting) plus thin adapters. Process spawning, clock and filesystem come in through Deps (ports.ts), so every test runs on fakes and fixtures; nothing touches a simulator or device.

How measure-only works

1. The customer's build already contains the perf-lab SDK, which appends <wall ms> PERF_LAB window {json} lines to Documents/perf-lab/log.txt for every window it observes (screen opens, interactions).
2. After each run the log is pulled and the windows whose start (wall - durMs) lies in [t0, t1] of that run are kept. Anything older is leftover from earlier runs.
3. Per window name, the run's value is the median over its cycles of: dur_ms, commits, renders, mounts, longtask_ms, gc_count, gc_ms, alloc_mb, touch_max_ms; plus n, timeouts (windows ending in timeout are excluded from medians) and retained_mb_per_cycle (Theil-Sen slope of heap after GC across cycles, needs 3+ cycles). Metric names are <window>.<field>, kept as-is (e.g. screen:(app)/community/[handle]/(tabs).dur_ms) and quoted in the CSV when needed.
4. A failed test, an install or log-pull failure, or a run with no windows is written as invalid with a reason. Invalid runs stay in metrics.csv (a run.status row when they have no metrics) and the stats engine drops them whole.
5. block is the position of the run's pair in the ABBA order, so the stats engine can analyse paired blocks.

Realtime Inspector and memory metrics

--ri-capture <path> (on measure and extract) points at an RI ingest-sink capture, observations.ndjson, one JSON per line (envelope.batch.observations[]). A RiSource port (FileRiSource) re-reads it after every run; client_health.* metricSamples are sliced to the run's [t0, t1]:

• mean: ri.fps.ui, ri.fps.js, ri.cpu.process_cores, ri.cpu.main_thread_cores, ri.cpu.js_thread_cores, ri.thread.main_blocked_ratio, ri.thread.js_blocked_ratio, ri.sdk.collect_ms
• delta (last - first, it is a running total): ri.frames.dropped
• max: ri.memory.footprint_mb, ri.memory.used_mb, ri.thermal.state, ri.thread.main_stall_ms, ri.thread.js_stall_ms
• ri.samples: number of fps.ui samples in the interval

Memory slopes (Theil-Sen over the cycle index, 3+ cycles of --mem-window, default chat.open): mem.retained_mb_per_cycle from the window's heapAfterGc (js_allocatedBytes + js_externalBytes, MiB) and, with a capture, mem.footprint_mb_per_cycle from the last footprint sample at or before each window start. On ProMotion iPhones ri.fps.ui, ri.frames.dropped and ri.thread.main_blocked_ratio are artefacts: diagnostic only.

test/parity.test.ts runs tools/analysis/ios_series_metrics.py and the TS extraction on a synthetic series and requires equal values.

Limitations

• Simulator runs give mechanism metrics only (renders, commits, mounts, GC, allocations). Times (dur_ms, long tasks, touch latency) mean something only on a real device.
• The test driver's own overhead and timing noise are inside the measurement; A/B order cancels drift, not a driver that behaves differently per arm.
• Window slicing uses the host clock (t0/t1) against the app's wall clock; on a device with a skewed clock windows may be missed (the run then becomes no-windows).
• iOS only. Android is not in v1. BrowserStack only through perflab farm (the app pushes; the CLI never reads the device).
• perflab measure does not boot devices or launch the app; the test command does that.

Working on the CLI

bun run typecheck and bun run test first build the workspace packages the CLI imports (bun run deps: stats, derive, verdicts, profile-analysis, mcp), so both work from a clean checkout.