@ishlabs/cli

v0.36.0

Published

6 days ago

The command-line interface for ish

0High
0Medium
0Low

felixweiland

ish connect localhost testing

ish

CLI tool for Ish — run studies, send asks, and expose your localhost for AI participant sessions.

Install

Quick install (recommended)

macOS / Linux:

curl -fsSL https://ishlabs.io/install.sh | sh

Windows (PowerShell):

irm https://ishlabs.io/install.ps1 | iex

npm (all platforms)

npm install -g @ishlabs/cli

Homebrew (macOS / Linux)

brew install ishlabs/tap/ish

Authenticate

ish login              # browser-based auth, stores tokens in ~/.ish/config.json
ish logout             # clear saved credentials

The CLI resolves your auth token in this order:

--token CLI flag
ISH_TOKEN env var
Saved token from ish login

Testing

Test plan is available at /Users/felixweiland/ish-cli-test-plan.md.

Experiments

Durable records of engineering experiments (including reverted ones, so we don't re-run them) live in docs/experiments/.

Concepts

Two top-level research primitives, both consume reusable people:

Workspace (= product, top-level container)
│
├── People                 ← reusable personas
│     └── Audience Sources (images/PDFs/audio/video/text transcripts that seed generation)
│
├── Study  ─────────────── "structured research artifact"
│     ├── modality (interactive | text | video | audio | image | document)
│     ├── content-type (for non-interactive studies: email | social_post | ad | etc.)
│     ├── assignments (tasks the participant does)
│     ├── questionnaire (questions participants answer — text, slider, likert, choice)
│     └── Iterations       ← the unit of execution within a study
│           └── Participants ← instance of a Person inside this Iteration
│                 └── Interactions / results
│
└── Ask  ──────────────── "lightweight reaction artifact"
      ├── Audience (participants, fixed at creation, max 5 rounds per ask)
      └── Rounds           ← the unit of execution within an ask
            └── Responses (per-participant reactions to a variant)

The two primary run verbs:

| | ish study run | ish ask run | |---|---|---| | Default | Run the latest iteration on the active study | Append a round to the active ask | | Fresh setup | Create the iteration first: ish iteration create … | --new (creates ask + round 1 in one shot) | | Specific target | --iteration <id> | Pass the ask id as positional arg | | Audience | --person <ids>, --sample <N>, --all, or demographic filters (--country, --gender, --min-age, --max-age, --search, --visibility) — else reuse iteration's participants | --person <ids>, --sample <N>, --all-simulatable, or demographic filters (with --new) |

Commands

Auth & infra

ish login                          # browser auth
ish logout
ish connect <port>                 # Cloudflare tunnel exposing localhost (--detach, ish disconnect, ish connect status)
ish upgrade                        # self-update (single-binary installs only)
ish upgrade --release 0.8.1        # pin a specific release

ish upgrade only works on standalone-binary installs (curl/Homebrew). On npm-installed CLIs (npm install -g @ishlabs/cli) it refuses with a pointer to npm install -g @ishlabs/cli@latest — overwriting node would break every other Node tool.

Workspaces, studies, iterations, people, configs (CRUD groups)

ish workspace  list | create | get | update | delete | use | info
ish workspace  site-access status | basic-auth | cookie | login | affirm-public | hosts | clear
ish study      list | create | generate | get | results | update | delete | use
ish iteration  list | create | get | update | delete
ish person     list | create | generate | get | update | delete
ish source     upload | get | delete
ish config     list | create | get | schema | update | delete
ish chat       endpoint list | create | get | update | delete | use | init | test
ish secret     list | set | delete

ish workspace info reports studies_used / studies_max / participants_used / participants_max / tier so an agent can branch on plan caps before a destructive call returns error_code: usage_limit_reached.

ish chat endpoint configures HTTP-bot endpoints for chat-modality studies (auto-detect from a curl example, smoke-test, edit). ish secret is the per-workspace KV store referenced from chatbot endpoint headers via {{secret:KEY}} placeholders. Run ish docs get-page guides/chat for the end-to-end recipe.

Participants live as a nested group on a study (low-level — usually created via study run):

ish study participant <id>                              # show participant details and results
ish study participant create | batch-create | delete    # low-level escape hatches

use saves an active workspace/study/ask to ~/.ish/config.json so you don't repeat --workspace/--study/--ask on every command. Aliases like w-6ec, s-b2c, a-…, i-…, pt-… work anywhere an ID is expected.

Define a study — `ish study create`

A study locks in the persistent shape: modality, optional content-type taxonomy, the tasks participants do (--assignment), and the questionnaire they answer (--question or --questionnaire). The actual URL or file lives on an iteration (next step).

# Interactive study, one assignment + a single-question questionnaire:
ish study create --name "Onboarding UX" --modality interactive \
    --assignment "Sign up:Complete the signup flow" \
    --question "How easy was it?"

# Multiple assignments + a richer questionnaire from a file:
ish study create --name "Checkout" --modality interactive \
    --assignment "Browse:Find a product you like" \
    --assignment "Buy:Add to cart and checkout" \
    --questionnaire ./questionnaire.json

# Bulk assignments from a file (text/email study):
ish study create --name "Newsletter" --modality text --content-type email \
    --assignments-file ./assignments.json

--question adds a simple text question to the questionnaire (repeatable, defaults to type=text, timing=after). For richer types (slider, likert, single/multiple-choice, number) and custom timing, pass a JSON manifest via --questionnaire <file.json>. The two flags are mutually exclusive — pick one.

Configure a run — `ish iteration create`

Iterations carry the URL (interactive) or content (media) for a run. Create one before ish study run. Local files passed to --content-url, --image-urls, etc. are uploaded automatically; @filepath reads text from a file.

# Interactive — URL:
ish iteration create --study s-b2c --url https://example.com

# Interactive on mobile:
ish iteration create --url https://example.com --screen-format mobile_portrait

# Text/email (inline or @file):
ish iteration create --content-text @./email.html --title "Newsletter"

# Video (URL or local file):
ish iteration create --content-url ./video.mp4

# Image set:
ish iteration create --image-urls "./a.png,./b.png"

# Document (PDF):
ish iteration create --content-url ./report.pdf

# Video ad with copy text:
ish iteration create --content-url ./ad.mp4 --copy-text "Buy now — 50% off!"

# Social post with caption:
ish iteration create --image-urls ./post.png \
    --copy-text @./caption.txt --social-platform instagram

Configure site access — `ish workspace site-access`

For studies that target a gated URL (HTTP basic auth, a session-cookie wall like Vercel preview protection, or a login form), give the site its access once. A workspace can test several sites and each site holds its own access: every method binds to one site, and a run uses the access of the site its URL (or app) belongs to, never another site's. A site answers on one or more hosts (.com and .se, an auth subdomain, localhost:3000); hosts adds them, and www and bare are already the same host. A study's URL picks its site by host. Basic auth and the cookie are applied to the origin the run opens (the study URL's origin, or its tunnel), not to every host of the site. The login belongs to its site too. Credentials are encrypted at rest; the CLI never reads them back, only shows which methods each site has.

# One row per site: hosts, methods, cookie expiry:
ish workspace site-access status

# HTTP basic auth (e.g. staging gate); the site is created on first use:
ish workspace site-access basic-auth --origin https://staging.example.com \
    --username alice --password hunter2

# Session cookie (Vercel preview, Lovable, etc.) on an existing site:
ish workspace site-access cookie --site preview.example.app --name session --value abc123

# Login form — credentials the participant types into the page (web or app):
ish workspace site-access login --username demo --password demo
ish workspace site-access login --app com.example.app --username demo --password demo

# Mark a site as public (silences the "credentials needed?" check):
ish workspace site-access affirm-public

# The same site on more hosts (--add appends, --set replaces):
ish workspace site-access hosts --site shop.com --add shop.se,auth.shop.com

# Clear one method, or everything, from a named site:
ish workspace site-access clear cookie --site preview.example.app
ish workspace site-access clear all --origin https://staging.example.com

Name the site with --origin <url>, --site <name> (exact name from status; an unknown name exits 2 and lists the sites) or --app <artifact>. Without one, writes use the site for the workspace base_url (set via ish workspace update --base-url); clear and hosts need it named. Pass - for --password/--value to read from stdin and keep secrets out of shell history:

printf %s "$STAGING_PW" | ish workspace site-access basic-auth --username alice --password -

Run a study — `ish study run`

Picks the latest iteration on the study (or --iteration <id>), creates participants (explicit --person, demographic-filtered sample, or reusing the iteration's existing participants), and dispatches simulations. If the study has no iterations, run ish iteration create first.

# Run the latest iteration, reusing its participants (after `study use`):
ish study run -y

# Run with explicit people:
ish study run --person p-795,p-af2

# Sample 3 Swedish people aged 35–50 from the workspace pool:
ish study run --country SE --min-age 35 --max-age 50 --sample 3

# Run with every female person in the workspace:
ish study run --gender female --all

# Run a specific iteration:
ish study run --iteration i-d4e

# Override the simulation config (e.g. for a media study):
ish study run --config c-c3c

# Block until all simulations finish (or timeout):
ish study run --wait
ish study run --wait --timeout 600

# Local browser (no remote Browserbase):
ish study run --local --headed --slow-mo 500

Audience flags (shared with ish ask): --person <ids> for explicit IDs, or any of --country, --gender, --min-age, --max-age, --search, --visibility paired with --sample <N> or --all to seed from the workspace pool. Mutually exclusive with --person.

Other study verbs (low-level):

ish study poll --study <id>                                      # one-shot progress
ish study poll <participant_id>                                  # single participant status
ish study wait --study <id>                                      # block until all done
ish study wait --iteration <id>                                  # block on one iteration
ish study wait <participant_id> --timeout 600                    # block on one participant
ish study cancel <participant_id>                                # cancel a running sim

<participant_id> is a participant alias (pt-...) or UUID. Get them from ish study run --json's participant_aliases[] / participant_ids[] arrays:

ish study run --study s-b2c -y --json | jq -r '.participant_aliases[]'   # → pt-072, pt-1ed, ...

Run an ask — `ish ask run`

Smart wrapper around the ask flow. With --new, creates a fresh ask + round 1 (people from --person, --sample, --all-simulatable, or any demographic filter — --country, --gender, --min-age, --max-age, --search, --visibility). Without --new, appends a new round to the active or specified ask.

# Append a round to the active ask:
ish ask run --prompt "And now which?" \
    --variant text:"X" --variant text:"Y" --wait

# Append to a specific ask:
ish ask run a-6ec --prompt "Round 2" \
    --variant text:"A" --variant text:"B"

# Create a fresh ask with round 1, sample 30 people, wait for results:
ish ask run --new --name "tagline AB" \
    --prompt "Which sounds better?" \
    --variant text:"Short and punchy." \
    --variant text:"A longer, descriptive line." \
    --sample 30 --wants-pick --wait

# Demographic-filtered sample (Swedish people aged 35–50):
ish ask run --new --name "SE 35-50" \
    --prompt "Which sounds better?" \
    --variant text:"A" --variant text:"B" \
    --country SE --min-age 35 --max-age 50 --sample 10 --wants-pick

# Image comparison from local files (auto-uploaded), explicit people:
ish ask run --new --name "hero shots" \
    --prompt "Which feels premium?" \
    --variant image:./hero-a.png::label=A \
    --variant image:./hero-b.png::label=B \
    --person p-d4e,p-a17 --wants-ratings

# Text from a markdown file + JSON questionnaire:
ish ask run --new --name "newsletter" \
    --prompt "Would you read this?" \
    --variant text:@./body.md \
    --questions ./questions.json --sample 30

--questions takes a JSON array shaped [{"question": "...", "type": "open_ended"|"slider"|"choice"|"likert"}]. The server requires the key question (not text). Minimal example:

[
  { "question": "What stood out?", "type": "open_ended" },
  { "question": "Rate it 1-5", "type": "slider" }
]

People flags (--person, --sample, --all-simulatable, --country, --gender, --min-age, --max-age, --search, --visibility, --name, --description, --workspace) only apply with --new — the people are fixed at ask creation.

--visibility values (same set everywhere it's accepted):

| Value | Selects | |---|---| | workspace | People owned by your workspace (default scope). | | shared | Community-published people visible across workspaces. | | platform | Admin-curated people from the platform pool. |

Old values private (now workspace) and public (now platform) keep working until the next release; the server logs a deprecation warning and maps them to the new vocabulary.

Other ask verbs:

ish ask list [--archived]
ish ask get [id] [--round <n>]
ish ask results [id] [--round <n>]
ish ask wait [id] [--round <n>] [--timeout <s>]
ish ask add-round [id] --prompt … --variant …       # explicit form of `run`
ish ask add-questions [id] --round <n> --questions ./qs.json
ish ask add-people [id] --person …                   # extend people for one round
ish ask create …                                    # explicit form of `run --new`
ish ask update | archive | unarchive | delete | use

Generate people

ish person generate runs the same generation flow used in the web UI: an LLM reads your description and any uploaded sources (transcripts, customer records, audio interviews, screenshots) and returns either a single person or a group of people.

# 5 people from a written brief:
ish person generate \
  --description "Tech-savvy millennials in the US who use mobile banking" \
  --count 5

# One person from a transcript — the file is uploaded automatically:
ish person generate --source ./interviews/sarah.txt --count 1

# Audio call with a written brief, diarized:
ish person generate \
  --description "Voices behind support tickets" \
  --source ./call.mp3 --diarize --count 3

For explicit control over uploads — e.g. reusing the same source across multiple generate runs — upload first and pass the returned alias:

ish source upload ./call.mp3 --diarize
# → ps-3a4 (status: processed)

ish person generate --source ps-3a4 --propose-count
# → { proposed_count: 4, rationale: "..." }

ish person generate --source ps-3a4 --count 4

Person context

Build one existing person up from their real interactions: an email thread, a meeting transcript, a commented document, plus a note in your own words. ish reads them into episodes of the person's quoted words; nothing is used until you approve an episode, and only in this workspace.

ish person context add p-d4e ./thread.eml ./intake-v1.pdf \
  --note "Anna sent Peter the intake proposal." --role intake-v1.pdf=shown --wait --json
ish person context list p-d4e --status pending --json
ish person context show p-d4e <episode-id>
ish person context approve p-d4e <episode-id> --expected-version 1
ish person context withdraw p-d4e <episode-id> --expected-version 2
ish person context erase p-d4e <episode-id> --yes   # permanent
ish person context patterns show p-d4e

Files are ordered; a quoted * or ? in a file name is expanded. Supported: PDF, TXT, MD, CSV, TSV, XLSX, EML, VTT, SRT, PNG, JPG, JPEG, WEBP and recordings (MP3, WAV, M4A, OGG, MP4, MOV, WEBM); export Word documents as PDF first. At most 20 files, 100 MiB each, 200 MiB total. A question (needs_input) is answered with ish person context answer <person-id> <import-id> "<answer>" --wait; a wait that runs out exits 5 and resumes with ish person context status <person-id> <import-id> --wait.

A real third party (ish person update <id> --subject-kind real_third_party) is approved by a human at a terminal: approve asks y/N and exits 2 (human_approval_required) without a TTY. --yes does not apply. Read ish docs get-page concepts/person-context for the full lifecycle.

Chat-modality studies — `ish chat`

Configure a customer chatbot endpoint and run chat-modality studies against it.

# Author from a curl example (or hand-write the config)
ish chat endpoint init --from-curl ./bot.curl --name my-bot
ish chat endpoint create --endpoint-config ./bot-config.json --name "my-bot"

# CRUD on saved endpoints (every dialog edit reduces to one of these)
ish chat endpoint list
ish chat endpoint get ep-abc --verbose      # round-trippable {id, name, isTunnelBacked, config}
ish chat endpoint update ep-abc --name "Production support bot"
ish chat endpoint update ep-abc --url https://api.example.com/v2/chat --mode stateless
ish chat endpoint get ep-abc --verbose | jq '.config.outgoing.headers["X-API-Key"] = "{{secret:KEY}}"' \
  | ish chat endpoint update ep-abc --endpoint-config -
ish chat endpoint delete ep-abc
ish chat endpoint use ep-abc                 # set as the active chat endpoint

# Smoke test the connection (single turn; tunnel pre-flight when applicable)
ish chat endpoint test ep-abc -m "Hello"
ish chat endpoint test ep-abc -m "Tell me more" --conversation-id "$CID"   # stateful threading

# Run a chat-modality study using the saved endpoint (existing study verbs).
# Audience size lives on study run via --sample / --all / --person.
ish study create --modality chat --endpoint ep-abc --name "Sign-up Q1" --assignment "Sign up:Try to sign up"
ish study run --study stu-xyz --sample 5 --wait
ish study results stu-xyz --json | jq '.participants'

Local bots (localhost / 127.0.0.1 / 0.0.0.0) auto-flag is_tunnel_backed=true on init; pair with ish connect <port> in another shell. Override with --tunnel-backed / --no-tunnel-backed.

init returns confidence (high / medium / low) and a missingSignals: [...] array naming any inputs the inference couldn't observe (e.g. ["response_shape", "message_path"] when no response sample is provided). When confidence is low, verify with chat endpoint test before running a study.

Failures from chat endpoint test carry a structured error_kind: TunnelInactive (run ish connect <port> first), BotUnreachable (URL/port wrong or bot down), BotResponseError (non-2xx with a status code), BotEnvelopeError (200 OK with the bot's own error in the body — see raw_excerpt), BotInvalidResponseError (response doesn't match the parsing schema), BotAuthError, BotTimeoutError, BotRetryExhaustedError.

Full guide: ish docs get-page guides/chat.

Expose localhost

For interactive studies (and chat endpoints with is_tunnel_backed=true) that need to reach a service running on your machine:

ish connect 3000                          # foreground Cloudflare tunnel to :3000
ish connect 3000 --detach --json          # fork after first heartbeat; prints {pid, tunnel_url, registered}
ish connect status --json                 # {active, pid, tunnel_url, registered_at} or {active:false}
ish disconnect --json                     # graceful shutdown of an active tunnel
ISH_TOKEN=YOUR_TOKEN ish connect 8080

Foreground connect is long-running — keep it open while participants run. The tunnel URL prints prominently after "Connected"; pass --json for one-line machine-readable output ({"status":"connected","tunnel_url":"...","local_port":3000,"registered":true}). The --detach form forks after the first successful heartbeat and returns immediately, tracking PID + URL in ~/.ish/connect.lock so connect status and disconnect find it later.

Destructive verbs in --json mode (e.g. chat endpoint delete, study delete) require an explicit --yes; the rejection envelope carries error_kind: "ConfirmationRequired" and an example field with the same command + --yes appended, so an agent can recover without re-reading the help text.

Global flags

| Flag | Description | |------|-------------| | -t, --token <token> | Auth token (or set ISH_TOKEN env var) | | --api-url <url> | Backend API URL (default https://api.ishlabs.io or ISH_API_URL) | | --json | Output JSON (auto-enabled when piped) | | --fields <a,b,c> | Comma-separated fields to include in JSON output | | --verbose | Include full UUIDs and timestamps in JSON output | | -q, --quiet | Suppress progress messages on stderr | | -V, --version | Show CLI version |

All commands print machine-readable JSON when stdout is piped or --json is passed, so AI agents can chain them together without parsing tables.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

ish