OpenIssue

OpenIssue turns existing AI-agent traces into evidence-cited incident packets, suspicious-cluster investigations, role-specific answers, and regression-eval suggestions.

The detection engine is deterministic and source-agnostic. OpenAI or Anthropic can be used only to explain and present those fixed results; neither provider can add, suppress, or modify detected issues, counts, severity, confidence, fingerprints, or evidence. The CLI is the source of truth today.

Install and verify

Requires Node.js 20 or newer and either an OpenAI or Anthropic API key. No OpenIssue account or hosted backend is required.

export OPENAI_API_KEY=sk_...
npx @sxohom/openissue --version
npx @sxohom/openissue check --json
npx @sxohom/openissue demo --out ./openissue-demo

check validates configuration and source parsing without making a model call. demo runs the complete detector-plus-synthesis path against a bundled reference incident. OpenAI remains the default provider with pinned model gpt-5.4-2026-03-05; override it with --model or OPENISSUE_MODEL.

Anthropic is also supported:

export ANTHROPIC_API_KEY=sk-ant-...
export ANTHROPIC_MODEL=claude-haiku-4-5-20251001
npx @sxohom/openissue demo --provider anthropic --out ./openissue-anthropic-demo

Set OPENISSUE_PROVIDER=anthropic to make Anthropic the environment default. Explicit --provider openai|anthropic takes precedence.

Every successful scan, trial, or demo writes run.json. Deterministic artifacts are written before synthesis, so a model failure leaves the evidence intact and records status: "failed" with stages.synthesis: "failed". Output directories must be empty unless --force is explicitly supplied.

Reference Lab

When production design-partner traces are unavailable, use the controlled seven-day ground-truth lab:

pnpm lab:reference -- --out .openissue-lab/reference-week
pnpm lab:external -- --out .openissue-lab/external-pyromind --limit 100

The first command must pass all 7 canonical days through both fixture and OTEL ingestion. The second is an opt-in network compatibility check over 100 trajectories from a pinned Apache-2.0 public corpus. See the reference-lab contract for the fault ledger, proof gates, limitations, and phase-exit criteria.

Demo

pnpm install
export OPENAI_API_KEY=sk_...
pnpm demo

The demo scans examples/probable-execution-worker-unavailable.json and writes:

• reports/demo/incident.md
• reports/demo/issues.json
• reports/demo/issue.json
• reports/demo/suggested-evals.json
• reports/demo/suggested-eval.md
• reports/demo/suggested-eval.json
• reports/demo/synthesis.json
• reports/demo/run.json

You can also run the CLI directly:

pnpm openissue -- scan --fixture examples/probable-execution-worker-unavailable.json --out reports/demo

Omit --out to print the incident report to stdout.

Additional detector demos:

pnpm demo:tool
pnpm demo:delegation
pnpm demo:investigate
pnpm demo:trial
pnpm demo:otel
pnpm openissue -- scan --fixture examples/multi-issue.json --out reports/multi

Read-only Langfuse scan:

export LANGFUSE_PUBLIC_KEY=pk_...
export LANGFUSE_SECRET_KEY=sk_...
export LANGFUSE_BASE_URL=https://cloud.langfuse.com
export OPENAI_API_KEY=sk_...

pnpm openissue -- scan --source langfuse --window "last 24h" --compare "previous 7d" --out reports/langfuse

LANGFUSE_HOST is also accepted as a base URL alias for repos that already use that name.

To distinguish "no incidents" from "no usable data", request diagnostics:

pnpm openissue -- scan --source langfuse --window "last 24h" --compare "previous 7d" --out reports/langfuse --diagnostics

This writes reports/langfuse/diagnostics.json with observation counts, latest timestamp, kind/status distribution, metadata coverage, and detector eligibility counts.

OTEL/OpenInference export scan:

pnpm openissue -- scan \
  --source otel \
  --otel-file examples/otel-openinference.json \
  --window "2026-06-22T00:00:00.000Z/2026-06-23T00:00:00.000Z" \
  --out reports/otel-scan \
  --diagnostics

OTEL input can be an array of spans, { "spans": [...] }, { "data": [...] }, or OTLP JSON with { "resourceSpans": [...] }.

Try OpenIssue on Langfuse in 5 minutes

export LANGFUSE_PUBLIC_KEY=pk_...
export LANGFUSE_SECRET_KEY=sk_...
export LANGFUSE_BASE_URL=https://cloud.langfuse.com
export OPENAI_API_KEY=sk_...

pnpm openissue -- trial --source langfuse --window "last 7d" --compare "previous 7d" --out reports/trial

Start with reports/trial/trial-summary.md. The trial pack includes doctor, investigation, scan, role-specific ask answers, suggested evals, and evidence bundles in one folder.

For a tight walkthrough, use docs/demo-script.md.

You can still run the individual steps when you want lower-level control:

pnpm openissue -- doctor --source langfuse --window "last 24h" --out reports/doctor
pnpm openissue -- investigate --source langfuse --window "last 24h" --out reports/investigation
pnpm openissue -- scan --source langfuse --window "last 24h" --compare "previous 7d" --out reports/langfuse --diagnostics

doctor checks whether the trace data is usable and assigns a trace-readiness score. investigate ranks suspicious evidence candidates, even below formal incident thresholds. scan emits confirmed incidents when deterministic detectors fire.

If no incident is detected, scan --diagnostics still reports whether OpenIssue saw tool failures, pending traces, workflow failures, or trace metadata gaps.

Ask Questions About Evidence

After generating an investigation, scan, or doctor report, teammates can ask evidence-grounded questions without reading raw trace JSON:

pnpm demo:investigate
pnpm openissue -- ask --report reports/demo-investigation --question "Why are users getting stuck?"
pnpm openissue -- ask --report reports/demo-investigation --preset support --thread support-triage
pnpm openissue -- threads list
pnpm openissue -- threads show support-triage

Ask first searches OpenIssue artifacts deterministically and maps evidence into signals like tool_failure, pending_without_result, and trace_quality_gap. OpenAI then presents that bounded answer. Returned citations are checked against the deterministic evidence allow-list; unknown references fail the command. If the loaded reports do not contain enough evidence, the limitation is preserved instead of inventing an answer.

Codex Skill

This repo includes a local workflow skill at skills/openissue-langfuse-triage. It guides Codex-style coding agents to run an OpenIssue trial, read trial-summary.md, inspect ask-engineering.md and evidence bundles, and avoid uncited fix claims.

The skill is a workflow wrapper over the CLI, not a separate runtime.

Web Trial Workspace

The repo also includes a Next.js web app that demonstrates the team-facing workflow over the same canonical reference corpus:

pnpm web:dev

Open http://localhost:3000.

The default workspace is reference day 5 (a controlled worker-outage scenario), not production data. Set OPENISSUE_REFERENCE_DAY=1 through 7 to render another day.

Current web surfaces:

• overview with trace-readiness, confirmed issue, and suspicious-cluster counts,
• trial page backed by the fixture engine and a read-only Langfuse trial API,
• issue inbox populated from confirmed issues and suspicious clusters,
• Ask page with cited answers over OpenIssue evidence,
• source diagnostics page,
• alert settings page with monitor and email-preview output,
• cron endpoint at /api/cron/monitor for hourly monitor runs.

Useful web APIs:

curl http://localhost:3000/api/workspace
curl http://localhost:3000/api/monitor/demo
curl http://localhost:3000/api/sources/demo
curl -X POST http://localhost:3000/api/ask \
  -H 'content-type: application/json' \
  -d '{"question":"Why are users getting stuck?"}'

Read-only Langfuse web trial:

curl -X POST http://localhost:3000/api/trial/langfuse \
  -H 'content-type: application/json' \
  -d '{"window":"last 24h","compare":"previous 7d"}'

OTEL/OpenInference web trial:

curl -X POST http://localhost:3000/api/trial/otel \
  -H 'content-type: application/json' \
  --data-binary @examples/otel-openinference.json

Required runtime env for Langfuse web trial:

• LANGFUSE_PUBLIC_KEY
• LANGFUSE_SECRET_KEY
• optional LANGFUSE_BASE_URL or LANGFUSE_HOST

Required runtime env for real email delivery:

• RESEND_API_KEY
• OPENISSUE_ALERT_FROM
• OPENISSUE_ALERT_TO
• optional OPENISSUE_CRON_SECRET to protect /api/cron/monitor

Without email env, monitor runs still produce deduped records and alert previews, but do not send mail.

Read-Only MCP Server

OpenIssue can expose generated report folders to coding agents through a local read-only MCP stdio server:

pnpm demo:trial
pnpm --silent mcp -- --report reports/trial

Available tools:

• openissue_list_issues
• openissue_get_issue
• openissue_ask_evidence

The MCP server reads local OpenIssue artifacts only. It does not write to Langfuse, Slack, GitHub, or the codebase.

Seeded Langfuse Sandbox Validation

Seeded Langfuse validation proves the real Langfuse API path and detector mechanics. It does not prove that Pocket or any other product has a real production incident.

Use a dedicated Langfuse sandbox project. The seed script refuses to write unless OPENISSUE_LANGFUSE_SEED_TARGET=sandbox is set.

export LANGFUSE_PUBLIC_KEY=pk_...
export LANGFUSE_SECRET_KEY=sk_...
export LANGFUSE_BASE_URL=https://cloud.langfuse.com
export OPENISSUE_LANGFUSE_SEED_TARGET=sandbox

pnpm test:langfuse-seeded

This seeds synthetic traces labeled with source: "openissue-seed" and synthetic: true, waits for ingestion, then writes reports/langfuse-seeded.

Validation Levels

• Fixture tests prove deterministic detector behavior against stable local contracts.
• The reference lab proves exact daily classification, OTEL parity, stable fingerprints, recovery suppression, and evidence integrity on a controlled seven-day corpus.
• The pinned public-corpus check proves bounded schema compatibility and a clean negative run; it is not an incident-recall benchmark.
• Seeded Langfuse sandbox runs prove OpenIssue can write test traces, read them back through Langfuse, normalize them, and report incidents through the real API path.
• Pocket or other real product traces are the only evidence that OpenIssue works on actual product behavior.

Current Detectors

execution_worker_unavailable detects the pattern where delegated execution is accepted, but the execution worker path cannot complete the task because the execution queue is unavailable or has no pollers.

tool_failure_cluster detects repeated failures from the same tool and error signature across users or sessions.

delegated_execution_no_result detects delegated execution that receives a pending user-visible response but never produces a final result.

The report includes:

• severity,
• affected users, sessions, conversations, and turns,
• earliest known start time,
• previous-window comparison,
• representative trace evidence,
• likely runtime layer,
• suspected cause,
• suggested fix direction,
• suggested regression eval,
• trace-quality warnings.

If no issue is detected, the CLI prints No OpenIssue incidents detected.

Project Boundary

This MVP now includes a local web trial workspace and a read-only MCP stdio server, but it is not yet a full hosted multi-tenant service. There is no Slack bot, Langfuse write-back, auto-fix loop, or LLM-based detector.

The Langfuse adapter is read-only. It queries observations, normalizes them into the existing OpenIssue trace model, and runs the deterministic detector registry. OpenAI requests use structured outputs and store: false; Anthropic requests use JSON-schema structured outputs. API keys are read only from environment variables.

Ask mode is a local CLI thread layer over minimized generated artifacts, not generic chat over raw logs.

The web app currently demonstrates the product loop and exposes API boundaries. Durable auth, team persistence, billing, and production database migrations are still separate SaaS hardening work.

Development

pnpm typecheck
pnpm test
pnpm build
pnpm web:build

The release gates, installed-tarball smoke test, live OpenAI demo, and read-only Langfuse sandbox proof are defined in docs/cli-release-gate.md.