Vieval

Vitest-style evaluation framework for agents, models, and task pipelines.

vieval keeps eval authoring close to product code while giving you repeatable task discovery, matrix scheduling, live CLI output, JSON artifacts, and report commands.

Why Vieval

• Familiar eval files with describeTask, caseOf, casesFromInputs, and expect.
• Project, eval, and task matrix layers for model, scenario, rubric, and dataset variants.
• Built-in chat-model registration through ChatModels, plus custom project executors for non-chat workloads.
• Human-readable terminal output and machine-readable JSON/report artifacts from the same CLI.
• Importable runner, scheduler, assertion, config, plugin, and testing entrypoints for advanced integration.

Quick Start

Step 1. Create a config

// vieval.config.ts
import { defineConfig } from 'vieval'
import { chatModelFrom, ChatModels } from 'vieval/plugins/chat-models'

export default defineConfig({
  plugins: [
    ChatModels({
      models: [
        chatModelFrom({
          aliases: ['agent-mini', 'judge-mini'],
          inferenceExecutor: 'openai',
          model: 'gpt-4.1-mini',
        }),
      ],
    }),
  ],
  projects: [
    {
      name: 'default',
      root: '.',
      include: ['evals/*.eval.ts'],
      runMatrix: {
        extend: {
          model: ['agent-mini'],
          scenario: ['baseline'],
        },
      },
      evalMatrix: {
        extend: {
          rubric: ['default'],
        },
      },
    },
  ],
})

Step 2. Create an eval task

// evals/smoke.eval.ts
import { caseOf, describeTask, expect } from 'vieval'

describeTask('smoke', () => {
  caseOf('arithmetic-default', (context) => {
    expect(context.task.matrix.run.scenario).toBe('baseline')
    expect(2 + 2).toBe(4)
  }, {
    input: {
      prompt: 'Check simple arithmetic.',
    },
  })
})

Step 3. Run

pnpm -F vieval eval:run -- --config ./vieval.config.ts

The published binary form is:

vieval run --config ./vieval.config.ts

Authoring API

Use describeTask for the common Vitest-like authoring path:

import { caseOf, describeTask, expect } from 'vieval'

describeTask('prompt-language-ablation', () => {
  caseOf('resolves matrix axes', async (context) => {
    const selectedModel = context.model()
    const language = context.task.matrix.run.promptLanguage
    const scenario = context.task.matrix.run.scenario

    expect(selectedModel.id.length).toBeGreaterThan(0)
    expect(language).toBeDefined()
    expect(scenario).toBeDefined()
  }, {
    input: {
      prompt: 'Summarize the position in one sentence.',
    },
  })
})

Use builder style when loading a batch of inputs:

import { describeTask, expect } from 'vieval'

const arithmeticCases = [
  { name: 'addition-small', input: { a: 1, b: 2, expected: 3 } },
  { name: 'addition-large', input: { a: 20, b: 22, expected: 42 } },
]

describeTask('arithmetic-quality', ({ casesFromInputs }) => {
  casesFromInputs('arithmetic-case', arithmeticCases, ({ matrix }) => {
    const result = matrix.inputs.input.a + matrix.inputs.input.b
    expect(result).toBe(matrix.inputs.input.expected)
  })
})

describeEval remains exported as an alias of describeTask, but new examples should prefer describeTask because task/case semantics are the primary runtime model.

Matrix Model

vieval expands matrix scopes in this order:

1. Project config from vieval.config.*.
2. Eval definition from defineEval(...).
3. Task definition from defineTask(...).

Within each scope, matrix layers resolve in this order:

1. disable
2. extend
3. override

Both runMatrix and evalMatrix are supported at project, eval, and task scope. A flat object such as runMatrix: { scenario: ['baseline'] } is normalized to runMatrix.extend; layered form is preferred for new docs and examples.

Each scheduled task receives stable matrix metadata:

• task.matrix.run
• task.matrix.eval
• task.matrix.meta.runRowId
• task.matrix.meta.evalRowId
• task.matrix.inputs for caseOf(..., { input }) and casesFromInputs(...)

Config Example

import { defineConfig } from 'vieval'
import { chatModelFrom, ChatModels } from 'vieval/plugins/chat-models'

export default defineConfig({
  plugins: [
    ChatModels({
      models: [
        chatModelFrom({
          aliases: ['agent-mini', 'judge-mini'],
          inferenceExecutor: 'openai',
          model: 'gpt-4.1-mini',
        }),
        chatModelFrom({
          aliases: ['agent-large', 'judge-large'],
          inferenceExecutor: 'openai',
          model: 'gpt-4.1',
        }),
        chatModelFrom({
          aliases: ['agent-openrouter-mini'],
          inferenceExecutor: 'openrouter',
          model: 'openai/gpt-4.1-mini',
        }),
      ],
    }),
  ],
  projects: [
    {
      name: 'chat-evals',
      root: '.',
      include: ['evals/*.eval.ts'],
      runMatrix: {
        extend: {
          model: ['agent-mini', 'agent-large'],
          promptLanguage: ['en', 'zh'],
          scenario: ['baseline', 'stress'],
        },
      },
      evalMatrix: {
        extend: {
          rubric: ['strict', 'lenient'],
          rubricModel: ['judge-mini', 'judge-large'],
        },
      },
    },
  ],
})

Custom Executor

If a project provides no executor, vieval run still discovers eval files, schedules tasks, and executes module-defined task callbacks. Provide projects[].executor when a project needs custom execution for ASR, TTS, image, motion, hosted agents, or another domain runtime.

import { defineConfig } from 'vieval'

export default defineConfig({
  projects: [
    {
      name: 'motion-evals',
      root: '.',
      include: ['evals/*.eval.ts'],
      inferenceExecutors: [{ id: 'motion-engine' }],
      models: [
        {
          id: 'motion-engine:v2',
          aliases: ['motion-default'],
          inferenceExecutor: 'motion-engine',
          inferenceExecutorId: 'motion-engine',
          model: 'v2',
        },
      ],
      async executor(task, context) {
        const model = context.model({ name: 'motion-default' })
        const success = model.model === 'v2' && task.matrix.run.scenario === 'baseline'

        return {
          id: task.id,
          entryId: task.entry.id,
          inferenceExecutorId: task.inferenceExecutor.id,
          matrix: task.matrix,
          scores: [{ kind: 'exact', score: success ? 1 : 0 }],
        }
      },
    },
  ],
})

CLI

vieval run [--config <path>] [--project <name>] [--json] [--report-out <path>]
vieval compare [--config <path>] [--comparison <id>] [--output <path>] [--format table|json]
vieval report analyze <report-directory>
vieval report index <report-directory> [--output <path>] [--format table|json|jsonl]
vieval report cases <report-directory> [--where <key=value>] [--group-by <key>] [--format table|json|jsonl]
vieval report compare <left-report-directory> <right-report-directory> [--case-key <key>] [--score-kind <kind>] [--format table|json]

Common workspace commands:

pnpm install
pnpm -F vieval eval:run
pnpm -F vieval eval:run -- --config ./vieval.config.ts
pnpm -F vieval eval:run -- --config ./vieval.config.ts --project chess --project moderation
pnpm -F vieval eval:run -- --json
pnpm -F vieval eval:run -- --report-out .vieval/reports --workspace local --experiment prompt-v2 --attempt attempt-a
pnpm -F vieval exec tsx src/bin/vieval.ts compare --config ./vieval.config.ts --comparison agent-memory
pnpm -F vieval exec tsx src/bin/vieval.ts report analyze .vieval/reports/my-run
pnpm -F vieval eval:run -- --help

Concurrency flags are available on vieval run:

• --workspace-concurrency
• --project-concurrency
• --task-concurrency
• --attempt-concurrency
• --case-concurrency

Public Entrypoints

• vieval: defineConfig, loadEnv, requiredEnvFrom, describeTask, describeEval, caseOf, casesFromInputs, and expect.
• vieval/config: lower-level defineEval, defineTask, matrix types, task context types, model definitions, and plugin contracts.
• vieval/plugins/chat-models: ChatModels, ChatProviders, chatModelFrom, chatProviderFrom, chatModelMatrix, runtime config helpers, and chat telemetry helpers.
• vieval/core/runner: collection, scheduling, task context, cache runtime, scheduler runtime, execution, and aggregation utilities.
• vieval/core/assertions: assertion primitives and pipeline helpers.
• vieval/core/inference-executors: env helpers and remote provider executors.
• vieval/testing/expect-extensions: Vitest expect extensions for testing eval behavior.

Architecture

flowchart LR
  CLI["src/cli/index.ts\n(runTopLevelCli)"] --> RUN["src/cli/eval-run.ts\n(runEvalRunCli)"]
  CLI --> COMPARE["src/cli/compare.ts\n(runCompareCli)"]
  CLI --> REPORT["src/cli/report-*.ts\n(report commands)"]
  RUN --> ORCH["src/cli/run.ts\n(runVievalCli)"]
  ORCH --> CFG["src/cli/config.ts\n(loadVievalCliConfig)"]
  ORCH --> DISC["src/cli/discovery.ts\n(discoverEvalFiles)"]
  ORCH --> MODULES["src/cli/module-runtime.ts\n(load eval modules)"]
  MODULES --> DSL["src/dsl/task.ts\n(describeTask/caseOf/casesFromInputs)"]
  ORCH --> SCHEDULE["src/core/runner/schedule.ts\n(createRunnerSchedule)"]
  ORCH --> EXEC["src/core/runner/run.ts\n(runScheduledTasks)"]
  EXEC --> CTX["src/core/runner/task-context.ts\n(createTaskExecutionContext)"]
  EXEC --> AGG["src/core/runner/aggregate.ts\n(aggregateRunResults)"]
  ORCH --> REPORTERS["src/cli/reporters/*\nlive reporter + Vitest bridge"]
  ORCH --> ARTIFACTS["src/cli/report-artifacts.ts\nJSONL report artifacts"]
  CHAT["src/plugins/chat-models/*\nmodel/provider plugins"] --> CFG
  PROVIDERS["src/core/inference-executors/*\nprovider adapters + env"] --> CTX

Runtime Sequence

sequenceDiagram
  participant U as User
  participant C as src/cli/index.ts
  participant E as src/cli/eval-run.ts
  participant R as src/cli/run.ts
  participant L as src/cli/config.ts
  participant D as src/cli/discovery.ts
  participant M as src/cli/module-runtime.ts
  participant S as src/core/runner/*
  participant T as src/dsl/task.ts
  participant P as src/cli/reporters/*

  U->>C: vieval run --config ...
  C->>E: runEvalRunCli(argv)
  E->>R: runVievalCli(options)
  R->>L: loadVievalCliConfig()
  R->>D: discoverEvalFiles()
  R->>M: loadEvalModulesWithVitestRuntime()
  M->>T: register describeTask definitions
  R->>S: collectEvalEntries() + createRunnerSchedule()
  R->>P: createCliReporter(isTTY)
  R->>P: onRunStart + onTaskQueued
  R->>S: runScheduledTasks(...)
  S->>P: onTaskStart / onTaskEnd
  S->>T: task.run(context)
  T->>P: reporterHooks.onCaseStart / onCaseEnd
  S-->>R: aggregated run results
  R->>P: onRunEnd + dispose
  R-->>E: CliRunOutput
  E->>U: static summary or JSON

Examples In This Repository

• Define a custom eval task API
• Configure run/eval matrix combinations
• Load datasource records as task cases
• Use assertion helpers and Vitest expect extensions
• Compare reporters and experiment/attempt layering
• Bring your own agent execution pattern

Development

pnpm install
pnpm -F vieval test:run
pnpm -F vieval typecheck
pnpm lint

When To Use / Not Use

Use vieval when:

• you want evals close to app code with Vitest-like ergonomics;
• you need repeatable matrix experiments and stable run metadata;
• you want local diagnostics, CI JSON, and report artifacts from one runner;
• you need to evaluate product code or custom agent flows without moving them into a hosted eval system.

Do not use vieval when:

• you need hosted dataset management, annotation UI, or SaaS observability out of the box;
• you only need a one-off script without reusable eval definitions or matrix scheduling.

Acknowledgements

• Vitest
• LobeHub
• EvalSys

License

MIT