Raindrop Vitest

Run reusable Raindrop eval suites as native Vitest rows. Your agent executes in your test process. Raindrop stores the candidate traces and evaluator results. Workshop is not required.

When evalTests creates its own Raindrop client, pass appGit in its options to inject the agent-under-test build identity: await evalTests(suite, { appGit: { commitSha: process.env.APP_BUILD_SHA } }). With an existing client, configure appGit on that client instead. Simulations report that candidate snapshot through the existing replay commit fields; pending automatic discovery is omitted, never awaited. Source/reference dataset Git properties are not candidate defaults.

This adapter requires raindrop-ai >=0.6.0 for dataset publication, evaluator pins, and the row-based suite contract. Input-only datasets also require the matching Raindrop backend.

Run against Raindrop

Install the SDK and adapter:

pnpm add -D raindrop-ai@^0.6.0 @raindrop-ai/vitest vitest

Before running, provide:

• RAINDROP_QUERY_API_KEY, an existing Query SDK organization API key. Browser login does not authenticate the SDK.
• RAINDROP_PROJECT_ID, the dataset's project slug. Omit it to use default.
• Any model credentials and test environment your agent requires.
• Dataset rows and evaluators defined locally, or their existing hosted slugs.

// eval/refunds.eval.ts
import { defineEvalSuite } from "raindrop-ai";
import { supportAgent } from "../src/support-agent";

export const refunds = defineEvalSuite({
  name: "refund responses",
  dataset: "refunds-golden",
  run: ({ input }) => supportAgent(input ?? ""),
  evaluators: [
    { evaluator: "refund-request-handled" },
  ],
});

// eval/refunds.eval.test.ts
import { evalTests } from "@raindrop-ai/vitest";
import { refunds } from "./refunds.eval";

await evalTests(refunds);

pnpm exec vitest run

evalTests(suite) defaults to hosted Raindrop. It uploads dataset rows and resolves existing hosted evaluator versions during async test collection. When the test runs, it starts a replay, calls your callback for each selected row, uploads candidate traces, waits for evaluation, and prints a receipt with evaluator run links and threshold results.

The callback is your integration point for environment setup and cleanup. Dataset inputs alone do not recreate databases, tool implementations, or side effects.

Publish an eval dataset

Create hosted evaluators in Raindrop and reference their slugs in await evalTests(suite). Dataset rows upload automatically before tests are registered. Use publishEvalSuite(client, definition) separately to upload the dataset without running or retain explicit version pins for before/after comparisons. Evaluator source is not uploaded. Hosted judges run on Raindrop, not in Vitest. See the complete author/run/compare flow.

For synthetic inputs, defineDataset accepts rows with just id and input. Row names default to their IDs; output and properties default to null and {}. Dataset versions are derived from content. To publish complete captured reference traces, use the lower-level publishEvalDataset before test collection:

import Raindrop, { publishEvalDataset } from "raindrop-ai";

const client = new Raindrop({
  apiKey: process.env.RAINDROP_QUERY_API_KEY,
  projectId: process.env.RAINDROP_PROJECT_ID,
  localWorkshopUrl: false,
});

try {
  await publishEvalDataset(client, {
    slug: "refunds-golden",
    name: "Refund examples",
    rows: [{
      id: "damaged-item",
      name: "Refund for a damaged item",
      input: "My item arrived damaged. Can I get a refund?",
      properties: { expected_action: "start_refund" },
    }],
  });
} finally {
  await client.close();
}

The SDK validates and fingerprints the rows, including any reference evidence. Omit expectedCurrentVersionId when creating the dataset; pass the current version ID when changing it. Conflicting updates throw EvalDatasetPublishConflictError.

defineDataset alone does not upload anything; evalTests publishes it. Automatic publication never generates missing reference traces or treats row outputs as reference evidence. Synthetic rows can run without a preliminary reference execution. Dataset updates still require expectedCurrentDatasetVersionId in the eval options. Edit hosted evaluators in Raindrop.

Evaluators that require a reference

Local evaluators can declare requiresReference: true in defineLocalEvaluator. Hosted evaluator programs declare export const requiresReference = true;. The SDK rejects selected rows without references before calling the agent. The backend also enforces the hosted declaration before executing the evaluator.

For candidate-only rows, reference is absent. Evaluators must guard this optional field; an absent reference is not a passing or failing grade. Existing reference traces remain unchanged and are not automatically treated as approved answers. The dashboard cannot grade an input-only dataset directly because it has no outputs yet; execute it through the SDK first.

Use an application-owned client

Keep one credential/project pair per Vitest worker process because tracing is process-global. Run suites with different credentials in separate workers or processes.

Use the explicit-client form when the application already owns its tracing provider:

await evalTests(client, refunds);

Construct that client with new Raindrop({ apiKey, projectId }). Eval control calls, dataset reads/publication, and replay trace uploads use the Query SDK apiKey. Production telemetry uses writeKey; it is not a replacement for eval authentication. The owned-client form closes its client after execution. Callers of the explicit form own their client's lifecycle.

Local evaluators, hosted results

An evaluator can run alongside your agent while its results still appear in Raindrop. Use defineLocalEvaluator when the verdict needs the agent's return value or local test state:

import { defineEvalSuite, defineLocalEvaluator } from "raindrop-ai";
import { evalTests } from "@raindrop-ai/vitest";
import { supportAgent } from "../src/support-agent";

const outputPresent = defineLocalEvaluator<string>({
  slug: "refund-output-present",
  name: "Refund output present",
  output: "boolean",
  judge: ({ result }) => ({ pass: result.trim().length > 0 }),
});

const refunds = defineEvalSuite({
  name: "refund responses",
  dataset: "refunds-golden",
  run: ({ input }) => supportAgent(input ?? ""),
  evaluators: [{ evaluator: outputPresent }],
});

await evalTests(refunds);

This example assumes supportAgent returns a string. For structured results, give the evaluator that result type and inspect its relevant field. Output presence is a minimal test, not an assessment of answer quality.

The SDK registers the local evaluator's identity and uploads its verdicts to Raindrop. Local callbacks receive { trace, row, result, reference }. row is the dataset input; reference holds the supplied reference snapshot and trace, when present. Choosing a local evaluator does not make the run local-only.

Test selection and results

When every evaluator is a local row callback, Vitest registers one test per dataset row and shares one replay run across selected rows. File and name filters apply before execution. Retries retain separate attempt traces and evaluator results.

A hosted evaluator or a batch-scoped evaluator makes the suite one aggregate native test. A name filter selects that test, not individual dataset rows within it.

pnpm exec vitest run
pnpm exec vitest run -t refund

The receipt includes values, thresholds, errors, and evaluator run links. A numeric evaluator without a threshold records a measurement. The SDK requires a verdict from each configured evaluator for every completed row; missing grades do not count as passes.

The full-service retry/concurrency rehearsal remains pending. Multi-file coordination and watch mode are outside the current release scope.

Run without Vitest

Use the same definition with runEvalSuite. It also defaults to Raindrop and publishes automatically:

import Raindrop, { runEvalSuite } from "raindrop-ai";
import { refunds } from "./eval/refunds.eval";

const client = new Raindrop({
  apiKey: process.env.RAINDROP_QUERY_API_KEY,
  projectId: process.env.RAINDROP_PROJECT_ID,
  localWorkshopUrl: false,
});

try {
  const result = await runEvalSuite(client, refunds);
  console.log(result);
} finally {
  await client.close();
}

Use createEvalSuiteRun only when you need to manage a session explicitly. Hosted row-scoped sessions require a pinned remote dataset selection and local evaluators.

Existing Workshop support

The SDK also contains an explicit { kind: "workshop", url } destination for local or pulled dataset snapshots and local or portable evaluators. This is separate from the hosted workflow above. It requires a Workshop build that implements the SDK's trace-read and eval-run endpoints. A running daemon alone does not prove compatibility.

Workshop results are stored in Workshop, not uploaded as Raindrop evaluator runs. Hosted evaluator slugs must be pulled before Workshop execution, and Workshop row callbacks do not receive golden reference pairs. Do not select this destination merely because your agent runs on your laptop.

defineDataset creates a local dataset object; running it with the default Raindrop destination publishes its rows. pullEval and loadEvalSnapshot retain local dataset/evaluator artifacts; pulling remote evaluator programs requires a Query SDK key. A pulled evaluator can be used with a hosted dataset slug, preserving its pinned evaluator identity. Local snapshots with non-null row outputs require explicit reference publication; automatic publication does not convert those outputs into reference traces.

No runs in Raindrop?

Creating a dataset or saving an evaluator does not execute your agent. A plain script that writes local JSON results does not upload them either. Confirm that the test calls evalTests or runEvalSuite and authenticates with a Query SDK key for the intended project. Raindrop is the default destination, and dataset rows upload automatically. Inspect the returned receipt or error. A Workshop endpoint error blocks that explicit destination, not hosted eval execution.