@ishlabs/cli

v0.22.0

Published

12 hours ago

The command-line interface for ish

0High
0Medium
0Low

felixweiland

ish connect localhost testing

ish

CLI tool for Ish — run studies, send asks, and expose your localhost for AI participant sessions.

Install

Quick install (recommended)

macOS / Linux:

curl -fsSL https://ishlabs.io/install.sh | sh

Windows (PowerShell):

irm https://ishlabs.io/install.ps1 | iex

npm (all platforms)

npm install -g @ishlabs/cli

Homebrew (macOS / Linux)

brew tap ishlabs/tap
brew install ish

Authenticate

ish login              # browser-based auth, stores tokens in ~/.ish/config.json
ish logout             # clear saved credentials

The CLI resolves your auth token in this order:

--token CLI flag
ISH_TOKEN env var
Saved token from ish login

Testing

Test plan is available at /Users/felixweiland/ish-cli-test-plan.md.

Concepts

Two top-level research primitives, both consume reusable people:

Workspace (= product, top-level container)
│
├── People                 ← reusable personas
│     └── Audience Sources (images/PDFs/audio/video/text transcripts that seed generation)
│
├── Study  ─────────────── "structured research artifact"
│     ├── modality (interactive | text | video | audio | image | document)
│     ├── content-type (for non-interactive studies: email | social_post | ad | etc.)
│     ├── assignments (tasks the participant does)
│     ├── questionnaire (questions participants answer — text, slider, likert, choice)
│     └── Iterations       ← the unit of execution within a study
│           └── Participants ← instance of a Person inside this Iteration
│                 └── Interactions / results
│
└── Ask  ──────────────── "lightweight reaction artifact"
      ├── Audience (participants, fixed at creation, max 5 rounds per ask)
      └── Rounds           ← the unit of execution within an ask
            └── Responses (per-participant reactions to a variant)

The two primary run verbs:

| | ish study run | ish ask run | |---|---|---| | Default | Run the latest iteration on the active study | Append a round to the active ask | | Fresh setup | Create the iteration first: ish iteration create … | --new (creates ask + round 1 in one shot) | | Specific target | --iteration <id> | Pass the ask id as positional arg | | Audience | --person <ids>, --sample <N>, --all, or demographic filters (--country, --gender, --min-age, --max-age, --search, --visibility) — else reuse iteration's participants | --person <ids>, --sample <N>, --all-simulatable, or demographic filters (with --new) |

Commands

Auth & infra

ish login                          # browser auth
ish logout
ish connect <port>                 # Cloudflare tunnel exposing localhost (--detach, ish disconnect, ish connect status)
ish upgrade                        # self-update (single-binary installs only)
ish upgrade --release 0.8.1        # pin a specific release

ish upgrade only works on standalone-binary installs (curl/Homebrew). On npm-installed CLIs (npm install -g @ishlabs/cli) it refuses with a pointer to npm install -g @ishlabs/cli@latest — overwriting node would break every other Node tool.

Workspaces, studies, iterations, people, configs (CRUD groups)

ish workspace  list | create | get | update | delete | use | info
ish workspace  site-access status | basic-auth | cookie | login | affirm-public | clear
ish study      list | create | generate | get | results | update | delete | use
ish iteration  list | create | get | update | delete
ish person     list | create | generate | get | update | delete
ish source     upload | get | delete
ish config     list | create | get | schema | update | delete
ish chat       endpoint list | create | get | update | delete | use | init | test
ish secret     list | set | delete

ish workspace info reports studies_used / studies_max / participants_used / participants_max / tier so an agent can branch on plan caps before a destructive call returns error_code: usage_limit_reached.

ish chat endpoint configures HTTP-bot endpoints for chat-modality studies (auto-detect from a curl example, smoke-test, edit). ish secret is the per-workspace KV store referenced from chatbot endpoint headers via {{secret:KEY}} placeholders. Run ish docs get-page guides/chat for the end-to-end recipe.

Participants live as a nested group on a study (low-level — usually created via study run):

ish study participant <id>                              # show participant details and results
ish study participant create | batch-create | delete    # low-level escape hatches

use saves an active workspace/study/ask to ~/.ish/config.json so you don't repeat --workspace/--study/--ask on every command. Aliases like w-6ec, s-b2c, a-…, i-…, pt-… work anywhere an ID is expected.

Define a study — `ish study create`

A study locks in the persistent shape: modality, optional content-type taxonomy, the tasks participants do (--assignment), and the questionnaire they answer (--question or --questionnaire). The actual URL or file lives on an iteration (next step).

# Interactive study, one assignment + a single-question questionnaire:
ish study create --name "Onboarding UX" --modality interactive \
    --assignment "Sign up:Complete the signup flow" \
    --question "How easy was it?"

# Multiple assignments + a richer questionnaire from a file:
ish study create --name "Checkout" --modality interactive \
    --assignment "Browse:Find a product you like" \
    --assignment "Buy:Add to cart and checkout" \
    --questionnaire ./questionnaire.json

# Bulk assignments from a file (text/email study):
ish study create --name "Newsletter" --modality text --content-type email \
    --assignments-file ./assignments.json

--question adds a simple text question to the questionnaire (repeatable, defaults to type=text, timing=after). For richer types (slider, likert, single/multiple-choice, number) and custom timing, pass a JSON manifest via --questionnaire <file.json>. The two flags are mutually exclusive — pick one.

Configure a run — `ish iteration create`

Iterations carry the URL (interactive) or content (media) for a run. Create one before ish study run. Local files passed to --content-url, --image-urls, etc. are uploaded automatically; @filepath reads text from a file.

# Interactive — URL:
ish iteration create --study s-b2c --url https://example.com

# Interactive on mobile:
ish iteration create --url https://example.com --screen-format mobile_portrait

# Text/email (inline or @file):
ish iteration create --content-text @./email.html --title "Newsletter"

# Video (URL or local file):
ish iteration create --content-url ./video.mp4

# Image set:
ish iteration create --image-urls "./a.png,./b.png"

# Document (PDF):
ish iteration create --content-url ./report.pdf

# Video ad with copy text:
ish iteration create --content-url ./ad.mp4 --copy-text "Buy now — 50% off!"

# Social post with caption:
ish iteration create --image-urls ./post.png \
    --copy-text @./caption.txt --social-platform instagram

Configure site access — `ish workspace site-access`

For studies that target a gated URL (HTTP basic auth, a session-cookie wall like Vercel preview protection, or a login form), set credentials on the workspace once. Participants reuse them whenever a study points at a matching origin. Credentials are encrypted at rest; the CLI never reads them back, only checks whether each method is configured.

# Show what's configured:
ish workspace site-access status

# HTTP basic auth (e.g. staging gate):
ish workspace site-access basic-auth --username alice --password hunter2

# Session cookie (Vercel preview, Lovable, etc.):
ish workspace site-access cookie --name session --value abc123

# Login form — credentials the participant types into the page:
ish workspace site-access login --username demo --password demo

# Mark the site as public (silences the "credentials needed?" check):
ish workspace site-access affirm-public

# Clear one method, or everything:
ish workspace site-access clear cookie
ish workspace site-access clear all

--origin defaults to the workspace base_url (set via ish workspace update --base-url). Pass - for --password/--value to read from stdin and keep secrets out of shell history:

printf %s "$STAGING_PW" | ish workspace site-access basic-auth --username alice --password -

Run a study — `ish study run`

Picks the latest iteration on the study (or --iteration <id>), creates participants (explicit --person, demographic-filtered sample, or reusing the iteration's existing participants), and dispatches simulations. If the study has no iterations, run ish iteration create first.

# Run the latest iteration, reusing its participants (after `study use`):
ish study run -y

# Run with explicit people:
ish study run --person p-795,p-af2

# Sample 3 Swedish people aged 35–50 from the workspace pool:
ish study run --country SE --min-age 35 --max-age 50 --sample 3

# Run with every female person in the workspace:
ish study run --gender female --all

# Run a specific iteration:
ish study run --iteration i-d4e

# Override the simulation config (e.g. for a media study):
ish study run --config c-c3c

# Block until all simulations finish (or timeout):
ish study run --wait
ish study run --wait --timeout 600

# Local browser (no remote Browserbase):
ish study run --local --headed --slow-mo 500

Audience flags (shared with ish ask): --person <ids> for explicit IDs, or any of --country, --gender, --min-age, --max-age, --search, --visibility paired with --sample <N> or --all to seed from the workspace pool. Mutually exclusive with --person.

Other study verbs (low-level):

ish study poll --study <id>                                      # one-shot progress
ish study poll <participant_id>                                  # single participant status
ish study wait --study <id>                                      # block until all done
ish study wait --iteration <id>                                  # block on one iteration
ish study wait <participant_id> --timeout 600                    # block on one participant
ish study cancel <participant_id>                                # cancel a running sim

<participant_id> is a participant alias (pt-...) or UUID. Get them from ish study run --json's participant_aliases[] / participant_ids[] arrays:

ish study run --study s-b2c -y --json | jq -r '.participant_aliases[]'   # → pt-072, pt-1ed, ...

Run an ask — `ish ask run`

Smart wrapper around the ask flow. With --new, creates a fresh ask + round 1 (people from --person, --sample, --all-simulatable, or any demographic filter — --country, --gender, --min-age, --max-age, --search, --visibility). Without --new, appends a new round to the active or specified ask.

# Append a round to the active ask:
ish ask run --prompt "And now which?" \
    --variant text:"X" --variant text:"Y" --wait

# Append to a specific ask:
ish ask run a-6ec --prompt "Round 2" \
    --variant text:"A" --variant text:"B"

# Create a fresh ask with round 1, sample 30 people, wait for results:
ish ask run --new --name "tagline AB" \
    --prompt "Which sounds better?" \
    --variant text:"Short and punchy." \
    --variant text:"A longer, descriptive line." \
    --sample 30 --wants-pick --wait

# Demographic-filtered sample (Swedish people aged 35–50):
ish ask run --new --name "SE 35-50" \
    --prompt "Which sounds better?" \
    --variant text:"A" --variant text:"B" \
    --country SE --min-age 35 --max-age 50 --sample 10 --wants-pick

# Image comparison from local files (auto-uploaded), explicit people:
ish ask run --new --name "hero shots" \
    --prompt "Which feels premium?" \
    --variant image:./hero-a.png::label=A \
    --variant image:./hero-b.png::label=B \
    --person p-d4e,p-a17 --wants-ratings

# Text from a markdown file + JSON questionnaire:
ish ask run --new --name "newsletter" \
    --prompt "Would you read this?" \
    --variant text:@./body.md \
    --questions ./questions.json --sample 30

--questions takes a JSON array shaped [{"question": "...", "type": "open_ended"|"slider"|"choice"|"likert"}]. The server requires the key question (not text). Minimal example:

[
  { "question": "What stood out?", "type": "open_ended" },
  { "question": "Rate it 1-5", "type": "slider" }
]

People flags (--person, --sample, --all-simulatable, --country, --gender, --min-age, --max-age, --search, --visibility, --name, --description, --workspace) only apply with --new — the people are fixed at ask creation.

--visibility values (same set everywhere it's accepted):

| Value | Selects | |---|---| | workspace | People owned by your workspace (default scope). | | shared | Community-published people visible across workspaces. | | platform | Admin-curated people from the platform pool. |

Old values private (now workspace) and public (now platform) keep working until the next release; the server logs a deprecation warning and maps them to the new vocabulary.

Other ask verbs:

ish ask list [--archived]
ish ask get [id] [--round <n>]
ish ask results [id] [--round <n>]
ish ask wait [id] [--round <n>] [--timeout <s>]
ish ask add-round [id] --prompt … --variant …       # explicit form of `run`
ish ask add-questions [id] --round <n> --questions ./qs.json
ish ask add-people [id] --person …                   # extend people for one round
ish ask create …                                    # explicit form of `run --new`
ish ask update | archive | unarchive | delete | use

Generate people

ish person generate runs the same generation flow used in the web UI: an LLM reads your description and any uploaded sources (transcripts, customer records, audio interviews, screenshots) and returns either a single person or a group of people.

# 5 people from a written brief:
ish person generate \
  --description "Tech-savvy millennials in the US who use mobile banking" \
  --count 5

# One person from a transcript — the file is uploaded automatically:
ish person generate --source ./interviews/sarah.txt --count 1

# Audio call with a written brief, diarized:
ish person generate \
  --description "Voices behind support tickets" \
  --source ./call.mp3 --diarize --count 3

For explicit control over uploads — e.g. reusing the same source across multiple generate runs — upload first and pass the returned alias:

ish source upload ./call.mp3 --diarize
# → ps-3a4 (status: processed)

ish person generate --source ps-3a4 --propose-count
# → { proposed_count: 4, rationale: "..." }

ish person generate --source ps-3a4 --count 4

Chat-modality studies — `ish chat`

Configure a customer chatbot endpoint and run chat-modality studies against it.

# Author from a curl example (or hand-write the config)
ish chat endpoint init --from-curl ./bot.curl --name my-bot
ish chat endpoint create --endpoint-config ./bot-config.json --name "my-bot"

# CRUD on saved endpoints (every dialog edit reduces to one of these)
ish chat endpoint list
ish chat endpoint get ep-abc --verbose      # round-trippable {id, name, isTunnelBacked, config}
ish chat endpoint update ep-abc --name "Production support bot"
ish chat endpoint update ep-abc --url https://api.example.com/v2/chat --mode stateless
ish chat endpoint get ep-abc --verbose | jq '.config.outgoing.headers["X-API-Key"] = "{{secret:KEY}}"' \
  | ish chat endpoint update ep-abc --endpoint-config -
ish chat endpoint delete ep-abc
ish chat endpoint use ep-abc                 # set as the active chat endpoint

# Smoke test the connection (single turn; tunnel pre-flight when applicable)
ish chat endpoint test ep-abc -m "Hello"
ish chat endpoint test ep-abc -m "Tell me more" --conversation-id "$CID"   # stateful threading

# Run a chat-modality study using the saved endpoint (existing study verbs).
# Audience size lives on study run via --sample / --all / --person.
ish study create --modality chat --endpoint ep-abc --name "Sign-up Q1" --assignment "Sign up:Try to sign up"
ish study run --study stu-xyz --sample 5 --wait
ish study results stu-xyz --json | jq '.participants'

Local bots (localhost / 127.0.0.1 / 0.0.0.0) auto-flag is_tunnel_backed=true on init; pair with ish connect <port> in another shell. Override with --tunnel-backed / --no-tunnel-backed.

init returns confidence (high / medium / low) and a missingSignals: [...] array naming any inputs the inference couldn't observe (e.g. ["response_shape", "message_path"] when no response sample is provided). When confidence is low, verify with chat endpoint test before running a study.

Failures from chat endpoint test carry a structured error_kind: TunnelInactive (run ish connect <port> first), BotUnreachable (URL/port wrong or bot down), BotResponseError (non-2xx with a status code), BotEnvelopeError (200 OK with the bot's own error in the body — see raw_excerpt), BotInvalidResponseError (response doesn't match the parsing schema), BotAuthError, BotTimeoutError, BotRetryExhaustedError.

Full guide: ish docs get-page guides/chat.

Expose localhost

For interactive studies (and chat endpoints with is_tunnel_backed=true) that need to reach a service running on your machine:

ish connect 3000                          # foreground Cloudflare tunnel to :3000
ish connect 3000 --detach --json          # fork after first heartbeat; prints {pid, tunnel_url, registered}
ish connect status --json                 # {active, pid, tunnel_url, registered_at} or {active:false}
ish disconnect --json                     # graceful shutdown of an active tunnel
ISH_TOKEN=YOUR_TOKEN ish connect 8080

Foreground connect is long-running — keep it open while participants run. The tunnel URL prints prominently after "Connected"; pass --json for one-line machine-readable output ({"status":"connected","tunnel_url":"...","local_port":3000,"registered":true}). The --detach form forks after the first successful heartbeat and returns immediately, tracking PID + URL in ~/.ish/connect.lock so connect status and disconnect find it later.

Destructive verbs in --json mode (e.g. chat endpoint delete, study delete) require an explicit --yes; the rejection envelope carries error_kind: "ConfirmationRequired" and an example field with the same command + --yes appended, so an agent can recover without re-reading the help text.

Global flags

| Flag | Description | |------|-------------| | -t, --token <token> | Auth token (or set ISH_TOKEN env var) | | --api-url <url> | Backend API URL (default https://api.ishlabs.io or ISH_API_URL) | | --json | Output JSON (auto-enabled when piped) | | --fields <a,b,c> | Comma-separated fields to include in JSON output | | --verbose | Include full UUIDs and timestamps in JSON output | | -q, --quiet | Suppress progress messages on stderr | | -V, --version | Show CLI version |

All commands print machine-readable JSON when stdout is piped or --json is passed, so AI agents can chain them together without parsing tables.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

ish