Promptjester

Promptjester is a TypeScript library for writing LLM evaluations that feel closer to Jest than to a one-off script. You write describe and test blocks, use runPrompt(...) to generate candidate responses through Promptjester when you want built-in model orchestration, and pass the result to evaluate(...). Promptjester then uses OpenRouter through the Vercel AI SDK to score the output against your rubric, persist the result locally, and compare it with the last passing run.

The evaluator itself runs with deterministic-friendly generation settings: temperature: 0, zero penalties, and a fixed seed.

What v1 does

• Gives you a dedicated promptjester run CLI for *.eval.ts files.
• Supports standard single-model evals and benchmark mode across multiple models.
• Lets tests use runPrompt(...) when Promptjester should own model execution.
• Scores responses against user-defined metrics on a 0-10 scale.
• Fails on threshold misses and on regressions against the last passing run.
• Stores run history locally in .cache/promptjester/history.jsonl.

This version is library-only. It does not implement hosted sync or a dashboard yet.

Install

pnpm add promptjester

Promptjester expects an OpenRouter API key in your environment because the evaluator model runs through OpenRouter.

export OPENROUTER_API_KEY=your-key

Minimal config

Create promptjester.config.ts at your project root:

import { defineConfig } from "promptjester";

export default defineConfig({
  evaluator: {
    model: "openai/gpt-4o-mini"
  },
  defaults: {
    subjectModel: "openai/gpt-4o-mini"
  },
  benchmark: {
    models: ["openai/gpt-4o-mini", "anthropic/claude-3.5-sonnet"]
  },
  promptRuns: {
    runs: 5,
    passStrategy: "most"
  }
});

Defaults:

• files: ["**/*.eval.ts"]
• storage.dir: .cache/promptjester
• evaluator.provider: openrouter
• evaluator.apiKeyEnv: OPENROUTER_API_KEY
• benchmark.models: []
• promptRuns.runs: 1
• promptRuns.passStrategy: every

Writing an eval

import { describe, test } from "promptjester";

describe("support bot", () => {
  test("stays polite and useful", async ({ runPrompt, evaluate }) => {
    const prompt = "My order is late and I need a refund.";
    const response = await runPrompt({ prompt });

    await evaluate({
      prompt,
      response,
      metrics: [
        {
          name: "helpfulness",
          rubric: "The answer should give the user a concrete next step.",
          passingScore: 8
        },
        {
          name: "tone",
          rubric: "The answer should stay calm and professional.",
          passingScore: 8
        }
      ]
    });
  });
});

response can be:

• The result of runPrompt(...)
• A plain string
• An array of plain strings or { text } objects when you want to sample multiple manual SDK responses in standard mode
• An object with a text field, such as the return value you get from generateText(...)

runPrompt(...) is the preferred path when Promptjester should control model execution. This becomes mandatory in benchmark mode because Promptjester needs to fan out the same prompt across multiple models.

Sampling multiple prompt runs

When you want to account for model variance, Promptjester can run the same prompt multiple times and gate pass/fail with a strategy.

Global defaults:

import { defineConfig } from "promptjester";

export default defineConfig({
  defaults: {
    subjectModel: "openai/gpt-4o-mini"
  },
  promptRuns: {
    runs: 5,
    passStrategy: "most"
  }
});

Per-test override:

const response = await runPrompt({
  prompt,
  runs: 3,
  passStrategy: "any"
});

Strategies:

• every: every sampled response must pass
• any: at least one sampled response must pass
• most: a strict majority must pass, such as 3/5

When sampling is enabled, Promptjester stores each sampled response separately in history and computes the overall test result from the configured strategy.

Benchmark mode

Benchmark mode runs the same eval against a model array and ranks the results by score. Enable it from the CLI:

npx promptjester run --benchmark

Override the model list from the command line:

npx promptjester run --benchmark --models openai/gpt-4o-mini,anthropic/claude-3.5-sonnet

Show the current benchmark responses side by side in a table:

npx promptjester run --benchmark --show-results

In benchmark mode:

• runPrompt({ prompt }) fans out across benchmark.models from config or the CLI override
• promptRuns.runs > 1 samples repeated responses per model before deciding pass/fail
• evaluate(...) compares each model result separately
• Promptjester reports a benchmark winner and per-model scores
• --show-results adds a side-by-side table with model scores and current responses
• The test passes when at least one model passes the required metrics

Manual single-model mode

Standard mode still supports manually generated single-model responses. If you already use the Vercel AI SDK yourself, you can pass its result object directly:

import { generateText } from "ai";
import { createOpenRouter } from "@openrouter/ai-sdk-provider";
import { test } from "promptjester";

const openrouter = createOpenRouter({
  apiKey: process.env.OPENROUTER_API_KEY
});

test("drafts a concise answer", async ({ evaluate }) => {
  const prompt = "Explain what a retry policy does in one sentence.";
  const result = await generateText({
    model: openrouter("openai/gpt-4o-mini"),
    prompt
  });

  await evaluate({
    prompt,
    response: result,
    subjectModel: "openai/gpt-4o-mini",
    metrics: [
      {
        name: "clarity",
        rubric: "The answer should be understandable to an engineer new to the topic.",
        passingScore: 8
      }
    ]
  });
});

If you want multiple manual SDK samples in standard mode, pass an array of SDK results and configure promptRuns:

const prompt = "Explain what a retry policy does in one sentence.";
const results = await Promise.all(
  Array.from({ length: 3 }, () =>
    generateText({
      model: openrouter("openai/gpt-4o-mini"),
      prompt
    })
  )
);

await evaluate({
  prompt,
  response: results,
  metrics: [
    {
      name: "clarity",
      rubric: "The answer should be understandable to an engineer new to the topic.",
      passingScore: 8
    }
  ]
});

Metrics and pass/fail

Each metric has:

• name
• rubric
• passingScore
• required (optional, defaults to true)

Scores are integers from 0 to 10. Promptjester treats 8 as a solid pass, reserves 9-10 for unusually strong answers, and uses lower scores for partial or clearly failing responses. A test passes only when all required metrics meet their threshold and none of those required metrics regress below the last passing run for the same test key.

Promptjester keys regression history by:

• Test identity
• Subject model
• Evaluator model
• Prompt hash
• Metrics hash
• Evaluator prompt version

If the prompt text or metric definitions change, the baseline resets automatically.

Running evals

Run the default discovery pattern:

npx promptjester run

Run a specific file or glob:

npx promptjester run "evals/**/*.eval.ts"

Run from CI:

OPENROUTER_API_KEY=$OPENROUTER_API_KEY npx promptjester run

Run benchmark mode:

OPENROUTER_API_KEY=$OPENROUTER_API_KEY npx promptjester run --benchmark

The process exits non-zero when:

• A required metric misses its threshold
• A required metric regresses against the last passing run
• The evaluator call fails
• A test never calls evaluate(...)

Show the previous and current evaluated responses for tests that already have a baseline:

npx promptjester run --show-response-comparison

Response comparison is opt-in because full answer bodies can be noisy in terminal output.

Local history

Promptjester stores append-only history in:

.cache/promptjester/history.jsonl

Stored fields include:

• Prompt text
• Response text
• Subject model
• Evaluator model
• Per-metric scores and rationales
• Pass/fail outcome
• Regression status

Benchmark runs store one history record per evaluated model, keyed by the subject model alongside the test and prompt hashes.

That stored response text is what Promptjester uses for the optional --show-response-comparison output.

Promptjester does not edit .gitignore. If .cache/ is not ignored in your repo, the CLI prints a warning and you can override storage.dir.

Examples in this repo

This repository includes example suites in examples/. They are part of the git repo but excluded from the published npm package.

After cloning the repo:

pnpm install
pnpm build
pnpm example:basic
pnpm example:vercel-ai
pnpm example:benchmark

See examples/README.md for what each sample demonstrates.

Codex Skill

This repo also includes a starter Codex skill in skills/promptjester/SKILL.md for agents that need concise Promptjester-specific authoring and CLI guidance.