moda-cli

CLI for Moda: production monitoring and evals for AI agents.

Query your trace analytics from the terminal. A trace is the full record of one agent run: every message, tool call, and step that shares one conversation_id (the trace ID).

Install

npm install -g @moda-ai/cli

Or use without installing:

npx -p @moda-ai/cli moda overview

Setup

Run the wizard to authenticate, pick an org, and provision an API key:

moda init

Or set the key manually:

export MODA_API_KEY="moda_sk_..."

Get your API key from moda.dev/settings.

Read-only keys

Turn on Read-only when you create a key in Settings → Ingestion keys. A read-only key works with every read command (overview, search, context, problems, ...) but the API rejects writes (false-positive, signals create, problem-feedback, ask, ...) with 403, and it cannot ingest data. Use one for agents and evaluations that should never change anything.

Platform admin keys

Platform admins can create moda_pk_ keys on the admin page (/admin/api-keys). They are not tied to an organization, so name one on each run:

export MODA_API_KEY="moda_pk_..."
export MODA_TENANT_ID="<organization-id>"
moda overview
moda users --tenant-id=<organization-id>   # tenant-scoped commands also take --tenant-id

They are read-only by default and stop working when the owner loses platform admin access. Usage is not counted in the organization's CLI analytics. Ordinary moda_sk_ keys cannot choose another organization: the API rejects them with 403.

Diagnostics

moda doctor --online checks configuration and cloud data flow. Experimental harness report citations, approval, and report sync do not affect normal doctor or status health; use the explicit harness commands to validate those reports. The legacy status.harnessReport JSON field is retained as skipped. Source folder presence is not evidence of SDK initialization, so doctor does not warn about instrumentation solely because a src or app directory exists. Verify your integration with a fresh application conversation and moda audit.

Local harness extraction

moda harness extract --path=<repo> --json scans source without an LLM or network calls. Its JSON preserves the traced call sites and roles, and adds graph.runtimeAgents and graph.artifacts: agent classes, multi-step tool loops, bound workflows, model-backed cron pipelines, and statically enabled framework/MCP tools alongside existing deterministic artifacts.

moda harness extract --path=. --json
moda harness candidates --path=. --promote=all --out=harness-components.json --yes

Every enumerated component includes a source citation and reachable or defined_only status. Production trace evidence can later promote it to in_prod. Dynamic MCP allow-all servers are one group; extraction does not connect to those servers. The repository-bound createAgentScopeResolver(result, ctx) callback maps CallSite.enclosing to emitted agent IDs, including inherited hooks and shared helpers; ambiguous or unknown scopes return [].

Component skeleton and diff

moda harness skeleton --path=. --out=/tmp/skeleton.json --json
moda harness skeleton-diff /tmp/before.json /tmp/after.json --json
moda harness skeleton-diff --from=/tmp/before.json --to=/tmp/after.json --json

skeleton produces harness-skeleton/v1, a component map of the harness: a HarnessGraph (agents, artifacts and relationships in the usual places, so sync and Ask read it unchanged) plus skeleton.callSites, counts, commit, content hash, coverage and a control-flow slot. It runs locally without an LLM or network calls and holds locations and owners, never bodies.

• Prompts: a named builder, template or prompt file a model call reaches is one unit, shared by every call that reaches it. Text assembled in the calling function is a call unit for that function, with the pieces as members. Constants inside a builder are not listed separately.
• Ids are stable across commits: they hold no line numbers, except the extractor's anonymous path#L<line> scopes. Repeated keys get ~2, ~3 in source order.
• liveKey is the key traces and the traced registry match on, normalised like the backend's trace status: agent class name, model-facing tool name, model id, skill name. Prompts and call sites have none.
• Map edges live in skeleton.edges, a compact table; decode them with decodeSkeletonEdges(graph.skeleton.edges) into { id: "<type>:<from>-><to>", type, from, to, source, evidence }. Types: triggers (entry point to agent), invokes_subagent and hands_off_to (agent to agent), uses_artifact (agent to prompt, tool or model), implemented_by (tool to code:<path>#<handler>), has_control_flow and guards (agent to control-flow item to call site), and covered_by_eval (agent or tool to eval suite). source is code for every extracted edge; consumers merge edges seen in traces as trace. relationships keeps only the agent-to-agent edges in the usual graph form, so sync and Ask read them unchanged.
• skeleton.coverage says per component type whether detection is complete or partial, with skeleton.knownGaps. Claim completeness only for complete types.
• Entry points: workflows and crons, exported route handlers of route agents (trigger: "http_route") and Durable Object bindings (trigger: "do_binding") of agent classes or of classes that construct one. Each lists the agents it starts in their entrypointIds (triggers edges); a workflow starts the function agents and sub-agent calls its code reaches.
• Tools are attributed to agents whose own model calls receive tools. A call that picks its model from a table at runtime has modelSource: "factory (computed)" and the table in modelCandidates; those candidates are not the call's or the agent's models, and each agent lists such calls in modelChoices (call site, location and candidates). A Workers eval suite that drives the deployed Worker covers the agents bound in its default Wrangler config.
• covered_by_eval is a static association (what the suite's own files import, through re-export barrels, and its own model calls), not execution or assertion coverage.

The JSON envelope's data is the skeleton; --out writes the bare skeleton. Incomplete scans return exit code 3 with warnings. skeleton-diff accepts either form, lists added, removed, moved and changed components and edges, ignores line-only shifts, and pairs unique same-name components across file renames as moved. Its identical flag compares content hashes. harness extract output is unchanged.

Library exports (src/harness-extract/skeleton/index.ts): buildHarnessSkeleton(rootDir, options?), skeletonFromExtract(result, ctx), diffHarnessSkeletons(before, after) and registerControlFlowDetector(detector). A detector receives {result, ctx, callSites, agents, callSiteIdOf} and returns control-flow items; items naming unknown call sites or agents are dropped, a throwing detector leaves controlFlowStatus: "failed" with a warning, and without a detector the status is not_run.

Usage

Find things by meaning, then drill in:

moda search "user wants a refund" --mode=semantic  # Semantic/keyword/hybrid message search
moda search "stripe.charges.create" --mode=keyword # Exact identifiers, incl. tool calls
moda search "checkout" --include-tool-io           # Search tool inputs/outputs too
moda context <conversation_id> --msg-index=5       # Read the exact turn
moda audit <conversation_id|trace_id>              # Raw span audit (trace ID or OTLP trace_id)
moda context <id>@12 <id2>@3 --window=2            # Batch: several traces in one call (max 25)
moda audit <id1> <id2>                             # Batch audit
jq -r '...' | moda context --ids-from=-            # Batch targets from stdin (or --ids-from=<file>)

Batch reads: moda context and moda audit take up to 25 targets (several positionals, or one per line with --ids-from=- / --ids-from=<file>). A context target is <id> or <id>@<msg_index>; ids may contain :, and a line may be JSON ({"id":"...","msg_index":N}) for an id that ends in @<digits>. Audit ids are verbatim. The output is { results: [{ target, ok, data | error }], succeeded, failed, has_more, remaining_targets }; a missing trace is a per-item not_found, and the exit code is 1 only when every target failed. One id keeps the single-trace output.

Production intelligence:

moda overview --days-back=30                     # Health briefing (KPIs, clusters, activity)
moda ask "what should I fix first?"              # Grounded answer with evidence
moda investigate                                 # Ranked issues worth fixing
moda problems                                    # Cross-signal Problems by root cause
moda problem <problem_id>                        # One Problem: full dossier
moda problem <problem_id> --evidence             #   ...attribution evidence (keyset paged)
moda problem <problem_id> --reports              #   ...investigation reports
moda problem <problem_id> --traces               #   ...affected traces
moda problem-feedback <problem_id> --action=mark_fixed   # Close the loop from the terminal
moda problem-linear <problem_id>                 # The Problem's Linear issue (key, state, URL)
moda problem-linear-file <problem_id>            # File it in Linear (needs Integrations -> Linear)

Signals:

moda emotions --family=confusion                 # Multi-family emotion detections
moda hallucinations --kind=contradicted          # Grounding: contradicted/verified outputs
moda frustrations --include-window --window=1    # Legacy single-family frustration view
moda tool-failures                               # Tool failure overview
moda tool-failure-detail <tool_name> --include-window

Traces, clusters, memory:

moda traces --search="error" --environment=production
moda cluster-traces <node_id>                    # Traces assigned to one cluster node
moda clusters                                    # Walk the topic hierarchy
moda clusters --search="billing disputes"        # Find a cluster by meaning
moda world-state <conversation_id>               # Agent memory (slots/threads/events)
moda world-state <id> --snapshot --msg-index=42  # What the agent believed at turn 42
moda world-state <id> --replay --message-count=50  # State evolution frame by frame

Live tail (one JSON line per new item — tail -f for your agent):

moda tail                                        # New traces, every 15s
moda tail --signal=all --interval=30             # Traces + emotion detections

Legacy aliases (same behavior, kept for existing scripts): moda conversations = moda traces, moda cluster-conversations = moda cluster-traces, --conversations = --traces (moda problem), --signal=conversations = --signal=traces.

Flag aliases: moda context --message-index=N = --msg-index=N, and moda users --limit=N = --page-size=N. Each command rejects a flag it does not take, and time windows differ by command: --days-back=N (a whole number, 1-90) on the overview, Problem and detection commands, --time-range on traces, search and clusters. The other window spelling is rewritten when it names the same span (--days-back=7 is --time-range=7d, --time-range=30d is --days-back=30); one with no exact match fails. Boolean flags accept true/false, 1/0, yes/no and on/off.

Prompt management:

moda prompts init                                # Create .moda/prompts.yml (version 2)
moda prompts status                              # Read-only local prompt status
moda prompts migrate --dry-run                   # Preview a version 1 -> 2 manifest rewrite
moda prompts sync                                # Upload changed prompt versions
moda prompts promote support.triage --label=prod --version=pver_abc123

Feedback:

moda feedback "cluster label looks wrong" --category=bad_cluster_label
moda feedback "trace belongs to another problem" --category=false_positive_attribution --problem-id=<id>

Output adapts to where it runs: piped/CI/agent environments get schema-versioned JSON (moda.agent.v1), an interactive terminal gets human prose. Force a mode with --json, --agent, or --human. In JSON modes, pipe to jq:

moda overview | jq '.data.metrics'
moda frustrations | jq '.data.frustrations[].primary_cause'

Every command supports moda <command> --help for scoped usage, and moda manifest --json describes the full machine protocol (commands, exit codes, event schema) for agents.

Prompt Management

Prompt management is code-first. Keep existing files and inline definitions in place. The CLI resolves selected sources without importing or executing your app. Existing prompts/**/*.prompt.md and prompts/**/*.prompt.json projects continue to work unchanged.

# Discover prompt sources and preview the sync without saving local state.
moda prompts sync --dry-run
# Run prompt-specific discovery, upload verified definitions, and save mappings.
moda prompts sync
# Optionally investigate unresolved sources with an installed coding agent.
moda prompts sync --analyst=claude
# Sync only configured sources, without discovery.
moda prompts sync --no-analyze

Every sync reuses the prompt-specific harness analyzer. It discovers candidates locally, then uses an installed Claude Code, Codex, or Cursor CLI to follow unresolved relationships and repair mappings. A separate agent session reviews new candidates. Source resolution reads the actual files; agent-written prompt text never becomes registry content. Uncertain candidates remain pending and are reported separately from sources eligible for sync.

Use --analyst=local-scan or --assist=none to disable agent calls, or --no-analyze to sync only configured sources. Existing mappings remain authoritative. Successful sync saves new accepted mappings in .moda/prompts.yml, preserving existing keys, legacy metadata, and negative prompt_paths patterns. Builders are stored as source definitions, not as claims about their rendered runtime text.

A plain --dry-run uses deterministic discovery. Add --analyst=claude, --analyst=codex, --analyst=cursor, or --analyst=auto to preview agent-assisted discovery without saving the manifest, lockfile, or registry versions. Agent investigations have no fixed wall-clock timeout and consume the installed agent's usage; interrupt the command to cancel. Sync prints scan and verification progress to stderr, including periodic analyst heartbeats while waiting. JSON results on stdout remain unchanged; --quiet suppresses progress and --agent --stream delivers progress as structured events.

--watch starts with analysis and then watches mapped definitions; re-run sync to discover new sources that do not affect an existing mapping. You can also run harness analyze --experimental --concern=prompts and import its report with prompts init --from-harness=<report>. Older reports are normalized and prompts without a promptSource are mapped from sourcePath when unambiguous; prompts that still do not resolve are returned in skipped with a reason, and the import fails only when none resolve. Discovery reports partial coverage: finding a prompt fragment does not prove that every composition or model call has been covered.

Prompt discovery keeps semantic role separate from mechanical extraction kind. Independent review can classify a source as an authored definition, builder, prompt transport, runtime context, non-prompt, or uncertain even when the resolver stored it as text, messages, template, or builder. Usage is tracked separately as active, conditional, display-only, or unknown, with an endpoint verification flag. Automatic code/configuration registration requires an authored definition or builder with independently reviewed active or conditional model use and a local consumer citation. Custom framework hooks can qualify without a recognized SDK binding. Consumer relationships are semantic reviewer judgments, not deterministic dataflow proofs; locally valid citations alone do not establish model use. The separate endpointVerified flag reports SDK provenance. Display-only sources stay visible as related; sources with unknown use remain pending. Helpers that forward prompt configuration or carry runtime context are retained as related discovery candidates and diagnostics, but are not synced as prompt sources. Builder provenance remains the source mapping and dependency evidence, so discovery does not claim complete whole-codebase classification.

The reviewer receives the complete parser-selected definition and can inspect surrounding relationships. Its definition citations, source hash, and source stability are checked locally. Malformed review JSON gets one format-only repair attempt; a repair that changes an already stated role or decision stays pending.

Before uploading, sync checks statically resolvable JavaScript/TypeScript lookups into the repository's .moda/prompts.lock.json. A missing key stops the upload and reports its source location. Preserve a stable runtime key by assigning it to the intended source mapping, or update the lookup to the selected mapping key. Dynamic lookups and other languages are not exhaustively checked; attribution coverage remains partial. A real sync with candidates but no eligible sources exits nonzero after printing discovery diagnostics; dry runs remain previews.

Configured mappings remain authoritative. To clean up sources previously registered in error, remove their entries from .moda/prompts.yml and sync with --allow-untrack. This removes local tracking, not historical registry versions.

You can also configure .moda/prompts.yml directly:

version: 2
prompt_paths:
  - instructions/**/*.txt
  - '!instructions/archive/**'
sources:
  - key: support.system
    path: src/agent.ts
    selector:
      symbol: SYSTEM_PROMPT
  - key: support.policy
    path: config/agents.yaml
    selector:
      pointer: /support/prompt
  - key: support.build
    path: src/prompt.py
    selector:
      symbol: build_prompt
    kind: builder
    dependencies:
      - path: config/policy.txt

Manifests from older CLIs say version: 1. Those CLIs matched .prompt.* files anywhere in the repository; patterns are now exact globs from the repository root. prompts status reports zeroMatchPatterns and legacyOnlyFiles (files only the old discovery found), and moda prompts migrate [--dry-run] [--discover] rewrites the manifest to version 2, adding those files as explicit paths so their keys and lockfile entries are unchanged. The lockfile contentHash is computePromptContentHash from @moda-ai/cli/prompt-hash (PROMPT_HASH_VERSION 1).

Selectors support named TypeScript/JavaScript/Python definitions and JSON Pointer selection in JSON/YAML. Python resolution requires python3; the helper parses ASTs without importing customer modules. Static literals and supported constant compositions become text or messages. Templates retain placeholders. Functions, unsupported expressions, and runtime composition remain builder definitions with source/dependency fingerprints and explicit incomplete coverage. Builders and templates are not plain prompt text; execute the application's builder and attribute the resulting model call to its registry version.

JSON definitions accept content, systemPrompt, or messages, with template as a fallback when none supplies usable text. Explicit mappings are required for ambiguous definitions. Exact file paths and globs are honored; duplicate keys, missing selectors, unsafe paths, and malformed manifests fail before upload.

status and diff are read-only. sync uploads changed versions and atomically writes .moda/prompts.lock.json only after validating the response and rechecking local sources. A failed sync preserves the prior lock. Removing a tracked source requires --allow-untrack; this removes its local lock entry, not remote versions. promote moves a remote dev, staging, or prod label.

Use the lock entry's { key, promptId, versionId } to attribute runtime calls. Actual rendered messages belong in traces; request data should not produce a new registry definition on each call. Plain renderable definitions can use the SDK:

const rendered = Moda.prompt("support.triage").render({
  ticket: { text: userMessage },
});

rendered = moda.prompt("support.triage").render({
    "ticket": {"text": user_message},
})

Register your harness

When the GitHub App or moda harness analyze isn't an option, have your coding agent write .moda/registry.json from the codebase and push it. Only what the manifest lists is uploaded. It needs just an API key.

moda registry prompt                                # brief to hand your coding agent
moda registry template --out=.moda/registry.json   # starter manifest (moda_registry.v2)
moda registry validate                              # local validation, no network
moda registry push --dry-run                        # server-side preview
moda registry push                                  # prompts, skills, tools, then agent wiring
moda registry status                                # registered harness and version
moda registry coverage --days=7                     # recent traces vs the registry

Prompts go to the prompt registry, skills to the skill library (pass --live or set "live": true to serve them), and tools to the tool registry as OpenAI function schemas. The v2 sections describe how the harness runs:

• agents: each agent's prompt, model and params, tools, skills, handoffs, and trace names.
• routing: how the harness switches prompts.
• context: what the runtime injects into prompts.
• runtime and mcpServers.

The format is in the bundled moda-cli skill. The analyzer's citation graph (the dashboard Harness map) still comes from moda harness analyze, the GitHub App, or an approved report.

Register a Replay Environment harness

moda harness register bundles the git repository a harness lives in for Replay Environments, prints what is included and excluded (secrets, env var values, and installs never leave the machine), and uploads it.

moda harness register --dry-run                     # collect and report; uploads nothing
moda harness register support-agent \
  --model-base-url-env=OPENAI_BASE_URL --entry -- node dist/server.js
moda harness registration list
moda harness registration status <id>

The collection policy, flags, statuses, and checklist are documented in Register a harness for Replay Environments (docs/harness/registration.mdx).

Anchors and message windows

Rows returned by moda frustrations and moda tool-failure-detail carry a top-level anchor block naming the exact turn the event fired on:

// frustrations
{ "kind": "frustration",  "conversation_id": "...", "msg_index": 12,
  "signal": "exasperation", "quote_preview": "...", "all_turns": [...],
  "no_anchor": false }

// tool-failure-detail examples
{ "kind": "tool_failure",  "conversation_id": "...", "msg_index": 7,
  "tool_name": "search", "tool_use_id": "...",
  "error_subtype": "SEARCH_NO_RESULTS", "no_anchor": false }

msg_index is the 0-indexed turn where the breakage happened. For frustrations it's derived from key_turns[0] (the same turn the API centers the embedded window on), falling back to user_quotes[0].turn when key_turns is empty. no_anchor: true means no anchor could be derived for that row.

Both commands accept two flags to inline a message window per row:

| Flag | Default | Description | |------|---------|-------------| | --include-window | off | Attach a window field with the message slice centered on the anchor | | --window=N | 1 | Half-width (1–5). --window=1 ⇒ 3 messages total (anchor ± 1) |

Environment Variables

| Variable | Required | Default | Description | |----------|----------|---------|-------------| | MODA_API_KEY | Yes | -- | Your Moda API key (moda_sk_...) | | MODA_BASE_URL | No | https://moda.dev | Base URL for the Data API. A loopback URL needs a non-prod profile (MODA_PROFILE=local); on a prod profile commands exit 5 before sending anything |

Documentation

• CLI Reference
• Data API Overview
• Quickstart

License

MIT

Local change impact (experimental)

moda harness impact --path=<checkout> --from=<old-commit> --to=<new-commit> --json follows changed JavaScript/TypeScript symbols through checker-confirmed references to model calls, agents, prompt units, and tool definitions. Alternatively pass --diff=<zero-context-diff> against --from. Control-flow additions use the old enclosing function. Single-line edits use changed columns to distinguish changed literals from unchanged siblings.

The command builds a disposable old source snapshot, preserves working-tree edits, returns only the reached slice, and makes no API calls or replays. It excludes assets and source files above 1 MiB; diff inputs have a 16 MiB aggregate source budget. Closure is bounded to 5,000 symbols and 100,000 references; returned units are bounded to 2,000 units / 2 MiB of literal text. Unchanged context reached through a changed symbol is capped at 20 units per call site and 400 in all (contextTruncated), so it never displaces changed text. Skips, closure budgets and a dropped changed-text unit (unitSliceTruncated) are reported as warnings and mean the result is incomplete. Changed styles, media and fonts are listed in ignoredFiles: they carry no prompt or tool text.

Optional --calls=<json> accepts {sourceCommit, candidate, sections, tools} records. sourceCommit must equal the resolved old commit, and candidate uses the frontend change-tests ChangeCase fields. Sections/tools supply text and sources: [{path, line, endLine}] from a same-agent source join. Optional agentIds rejects mismatched owners. Optional toolDefinitions ([{name, definition}]) joins recorded definitions to unique extracted tool names, owners and old source text. Ambiguous definitions remain unlinked. Templates require all static fragments in order; changed old literals cannot qualify through unchanged sibling text.

Keep call inputs and redirected output outside the repository: they can contain customer content. Python, unresolved dynamic dispatch and new files without old context remain unmodelled. This local result does not establish deployment attribution or production precision/recall.

Updating agent installations

The CLI checks npm at most daily. When a newer version is cached, --json commands also print an update notice on stderr without requiring a terminal, including on validation and authentication failures. JSON stdout is unchanged. With a writable home, notices are limited to once an hour for each running/latest version pair; --quiet, --no-update-check, MODA_CLI_UPDATE_CHECK=0, DO_NOT_TRACK=1, and CI suppress them. Read-only homes still receive cached guidance without a persisted cooldown. Agent envelopes continue to carry meta.update.

After updating, run moda skills install moda-cli from the project directory. It refreshes the installed guide and the init-managed Moda block in AGENTS.md, replacing legacy moda fixes instructions. Text outside that block stays intact; locally edited skill files are kept. Malformed or duplicated block markers and symlinked AGENTS.md files are left unchanged with a warning and recovery steps. Repeated installation is idempotent.

Retired fix, fixes, prompts ab, and prompts propose calls fail locally with migration guidance. Read problem evidence with moda problems get <id> --evidence, make changes in your project, and use moda prompts status, moda prompts diff, and moda prompts sync --dry-run to review prompt versions. The retired names stay out of help and the manifest.

Server warnings for legacy installations

Current buffered API requests send x-moda-cli-version; their output is unchanged. The Data API can prepend an upgrade line to an existing warnings array for an unversioned request whose authenticated tenant/key has recently reported an old CLI. The line describes an observed install on the key, not the current request.

This uses two nullable fields on the existing API-key row, not ClickHouse reads. There is no historical backfill: the first new legacy telemetry submission warms that key; subsequent reads can show the line, with up to 60 seconds of auth-cache delay on other instances. Observations expire after seven days. Current telemetry never clears evidence of another old install on the same key. Missing, invalid, future or expired observations are unknown and produce no hint. Platform keys are excluded. Version-header requests and x-moda-cli-update-check: 0 suppress it. Existing telemetry/update opt-outs are unchanged; old binaries cannot transmit new opt-out headers, so a prior observation expires normally if telemetry stops. No API keys or tenant content appear in the line or its failure logs.

Frozen 1.34.1 forwards this warning through JSON and agent warnings for frustrations, emotions, search, hallucinations, problem-impact, conversations and context. Overview is best-effort: its frozen wrapper nests the warning under rawOverview, does not promote it to envelope warnings, and can lose it under the 200 KB agent data limit. Response metrics, useful data and status codes are never changed to force delivery.

The offline boundary regression asserts that limitation as well as the seven positive cases. Build source aff167ef1f separately without modifying it, then run:

python3 scripts/verify-legacy-upgrade-warning.py \
  --frozen-cli /path/to/frozen/apps/moda-cli/dist/cli.js \
  --current-cli "$PWD/dist/cli.js"

Both builds use temporary local profiles, a loopback-only transport guard and telemetry disabled. No customer key, production request or paid evaluation is needed.