checkstack

Shadow-test your LLM calls across 164+ models — without changing your code.

Checkstack is a local proxy that intercepts your OpenAI-compatible API calls, forwards them to your chosen provider, and silently runs the same prompt against shadow models for comparison.

Quick start

# 1. Authenticate (free $1 credit)
npx checkstack auth

# 2. Pick your shadow models
npx checkstack configure

# 3. Run your app through the proxy
npx checkstack run -- node app.js

The run command starts the proxy and launches your app with OPENAI_BASE_URL auto-injected. When your app exits, the proxy shuts down and prints a session summary.

Alternatively, start the proxy as a long-running server:

npx checkstack start

Then point your SDK at http://localhost:3456/v1.

Commands

| Command | Description | |---------|-------------| | npx checkstack auth | Authenticate via browser, activate free $1 credit | | npx checkstack start | Start the local proxy server | | npx checkstack run -- <cmd> | Start proxy + run a command with env vars injected | | npx checkstack configure | Open shadow model picker in browser | | npx checkstack rerun <id> | Re-run a past evaluation with different settings | | npx checkstack replay <id> | Re-queue evals for a stuck/stalled run | | npx checkstack wallet | Check credit balance and subscription status | | npx checkstack subscribe | Open subscription page |

start / run options

| Flag | Description | |------|-------------| | -p, --port <port> | Proxy port (default: 3456) | | --skip-auth | Skip auth check (start only) |

rerun options

| Flag | Description | |------|-------------| | -r, --repeat <n> | Repeats per model/row (1–5) | | -s, --suggest | Ask AI to suggest prompt improvements first | | -o, --optimize <n> | Auto-optimize prompts for N iterations (1–20) | | --system-prompt <text> | Override the system prompt | | --user-prompt <text> | Override the user prompt template ({{input}} placeholder) |

How it works

1. Your app sends LLM calls to the local proxy (port 3456)
2. The proxy forwards each call to your real provider (OpenAI, Anthropic, etc.)
3. In the background, the same prompt is sent to your configured shadow models
4. Results are compared and scored on the Checkstack dashboard

What gets sent where

• Your API keys — forwarded to the upstream provider you're calling
• Prompts & responses — sent to Checkstack for shadow comparison and scoring
• Shadow model calls — made by Checkstack servers using your wallet credits

Requirements

• Node.js 18+
• An OpenAI-compatible SDK (OpenAI, Vercel AI SDK, LangChain, etc.)

Links

• Dashboard
• Documentation