@gaussia/sdk

Core foundations for the Gaussia evaluation SDK — the TypeScript port of pygaussia. Ships the load-bearing abstractions (Gaussia base class, Retriever interface, LanguageModel adapter contract, public Zod schemas) that downstream branches — 002-generators, 003-gepa-optimizer, 004-tier-1-metrics — build on.

Status: Phase 0 (001-core-foundations). Statistical modes (FrequentistMode, BayesianMode) and their parity fixtures land in branch 004 alongside the Tier 1 metrics that consume them.

Supported runtimes

• Node ≥ 20 (LTS). CI matrix: Node 20 and Node 22.
• Modern browsers with native ESM, top-level await, fetch, and AbortController. No polyfills bundled.

Subpath exports

| Subpath | What you get | |---|---| | @gaussia/sdk | Gaussia base class, Retriever interface, LanguageModel interface, Logger interface + silentLogger, exception hierarchy. | | @gaussia/sdk/schemas | Zod schemas: Batch, Dataset, IterationLevel, Logprobs, SessionMetadata, StreamedBatch, Chunk, GeneratedQuery, GeneratedQueriesOutput, ConversationTurn, GeneratedConversationOutput, OptimizationResult, IterationResult, CandidateResult, FailingExample. | | @gaussia/sdk/adapters/ai-sdk | createAiSdkAdapter factory — wraps a Vercel-AI-SDK model so it satisfies LanguageModel. | | @gaussia/sdk/generators | BaseGenerator, StringContextLoader, LocalMarkdownLoader (Node), strategies, prompts — synthetic dataset generation. | | @gaussia/sdk/prompt-optimizer | GEPAOptimizer, BaseOptimizer, LLMEvaluator, Executor/Evaluator contracts — GEPA prompt optimization. |

No other subpaths are documented. A consumer importing only @gaussia/sdk resolves a bundle that contains zero bytes from the adapter subpath or the ai peer (verified by the bundle-inspection CI job).

Peer dependencies

pnpm add zod          # required at runtime
pnpm add ai           # optional — only if you use @gaussia/sdk/adapters/ai-sdk

zod is marked as a peer dependency (^3). ai is an optional peer (^5); install it only if you import the adapter subpath.

First-call example

import { Gaussia, type Retriever } from "@gaussia/sdk";
import { Batch, Dataset } from "@gaussia/sdk/schemas";
import { createAiSdkAdapter } from "@gaussia/sdk/adapters/ai-sdk";
import { openai } from "@ai-sdk/openai";

// 1. A trivial retriever.
class HelloRetriever implements Retriever {
  async loadDataset() {
    return [
      Dataset.parse({
        sessionId: "s-1",
        assistantId: "a-1",
        context: "Greeting evaluation",
        conversation: [
          Batch.parse({
            qaId: "q-1",
            query: "Say hello",
            assistant: "Hello!",
            groundTruthAssistant: "Hello!",
          }),
        ],
      }),
    ];
  }
}

// 2. A throwaway concrete Gaussia subclass.
class CountBatches extends Gaussia<number> {
  protected async batch({ batch }: { batch: Batch[] }) {
    this.metrics.push(batch.length);
  }
}

const totals = await CountBatches.run(HelloRetriever, {});
console.log(totals); // → [1]

// 3. Wire a real LLM.
const llm = createAiSdkAdapter(openai("gpt-4o-mini"));
const { text } = await llm.generateText({ prompt: "Say OK." });

Writing tests with the mock LLM helper

The repo ships a deterministic mock LanguageModel for downstream branches' unit tests at tests/mocks/createMockLanguageModel.ts. It is intentionally NOT exported on the public package surface in Phase 0.

import { z } from "zod";
import { createMockLanguageModel } from "../mocks/createMockLanguageModel.js";

const mock = createMockLanguageModel({
  responses: [
    { kind: "text", text: "Hi!" },
    { kind: "object", object: { score: 0.95 } },
  ],
});

const t = await mock.llm.generateText({ prompt: "greet" });
// t.text === "Hi!"

const o = await mock.llm.generateObject({
  prompt: "score",
  schema: z.object({ score: z.number() }),
});
// o.object.score === 0.95

Exhausting the queue raises MockQueueExhaustedError — never silent undefined.

Generators

The @gaussia/sdk/generators subpath turns context documents into validated Dataset[] for evaluation. A BaseGenerator loads context → selects chunk groups → calls your LanguageModel per chunk → maps the structured output into Dataset/Batch objects.

import { BaseGenerator, StringContextLoader } from "@gaussia/sdk/generators";

const generator = new BaseGenerator({ model }); // model: LanguageModel
const datasets = await generator.generateDataset({
  contextLoader: new StringContextLoader(),
  source: "Gaussia is a community-driven AI evaluation framework.",
  assistantId: "my-assistant",
  numQueriesPerChunk: 3,          // single-turn (default)
  // conversationMode: true,      // multi-turn instead
});

• Loaders: StringContextLoader (isomorphic) and the Node-only LocalMarkdownLoader/createMarkdownLoader (hybrid header-then-size chunking). In a browser build the markdown loader resolves a stub that throws a clear LoaderError — use StringContextLoader there.
• Strategies: SequentialStrategy (default) and RandomSamplingStrategy (seedable, reproducible within TS); custom { select } strategies plug in.
• Schemas/prompts: GeneratedQueriesOutput, ConversationTurn, GeneratedConversationOutput, plus the verbatim default prompts.

The generators bundle pulls in zero AI-SDK bytes, and the default (browser) bundle imports no node:* built-ins. Full walkthrough: specs/002-generators/quickstart.md.

Prompt optimizer (GEPA)

The @gaussia/sdk/prompt-optimizer subpath improves an underperforming system prompt against a dataset using GEPA (Generative Evolutionary Prompt Adaptation): evaluate the seed prompt → collect failing examples → ask the model for improved candidates → keep the best if it strictly improves → repeat until convergence or the iteration budget.

import { GEPAOptimizer } from "@gaussia/sdk/prompt-optimizer";

const result = await GEPAOptimizer.run(MyRetriever, retrieverConfig, {
  model,                       // any LanguageModel (e.g. createAiSdkAdapter(...))
  seedPrompt: "You are a helpful assistant.",
  objective: "Answer using only the provided context.",
});
console.log(result.optimizedPrompt, result.initialScore, result.finalScore);

• Default judge or your own: omit evaluator to use the built-in LLMEvaluator (structured-output scoring, clamped to [0, 1]), or pass a deterministic Evaluator function (exact match, regex, tolerance). For better-calibrated judging, opt into LogprobEvaluator — it scores from the model's Yes/No token log probabilities (a port of pygaussia's guardian scoring) and falls back to the structured judge when the provider returns no logprobs:

  import { LogprobEvaluator } from "@gaussia/sdk/prompt-optimizer";
  const evaluator = new LogprobEvaluator({ model, criteria }).evaluate;

• Optimize your real system: pass an Executor wrapping your RAG chain or agent instead of a bare model call.
• Tune & observe: iterations, candidatesPerIteration, failureThreshold, onProgress, concurrency (default 1; >1 is faster with identical results), and candidateRetries (default 2; retries a transient malformed candidate response before throwing OptimizerError). result.history records every iteration's candidates and the failing examples that drove generation.

Like the generators, the prompt-optimizer bundle contains zero AI-SDK bytes and no node:* built-ins (fully isomorphic); streaming retrievers are rejected. Full walkthrough: specs/003-gepa-optimizer/quickstart.md.

Development

pnpm install --frozen-lockfile
pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm publint
pnpm attw --pack .
# meta-script that runs all six:
pnpm validate

Integration tests against a real OpenAI model run under pnpm test:integration when OPENAI_API_KEY is set. They are automatically skipped on fork PRs in CI.

Releasing & documentation

Versioning and publishing are driven by Changesets. Every PR with a behavioral change MUST include a changeset:

pnpm changeset      # choose a bump level and write a summary; commit the generated file

On merge to main, the release workflow accumulates pending changesets into a "Version Packages" PR. Merging that PR runs pnpm validate, then publishes @gaussia/sdk to npm with provenance, pushes a git tag, and creates a GitHub Release.

Documentation lives in docs/ (Mintlify). Per the constitution's Documentation principle, any change to a public (subpath-exported) API updates its docs/ page in the same change. Preview locally with cd docs && mint dev, and check links with mint broken-links. Merged docs/** changes sync to the central docs site automatically. See docs/README.md for the full runbook (secrets, bootstrap, release, and rollback).

License

MIT — see LICENSE.