@ideonate/evals-viewer-server

Vite plugin and middleware exposing the filesystem-backed evals-viewer API. Pair with @ideonate/evals-viewer-core for the Vue 3 frontend.

Install

npm install @ideonate/evals-viewer-server

Quickstart

// vite.config.js
import { defineConfig } from "vite";
import vue from "@vitejs/plugin-vue";
import { resolve } from "path";
import { evalsViewerPlugin } from "@ideonate/evals-viewer-server";

export default defineConfig({
  plugins: [
    vue(),
    evalsViewerPlugin({
      resultsDir: resolve(__dirname, "tests/test-results/evals"),
    }),
  ],
});

That's it. The plugin serves the eval results JSON tree under /api/evals/* while the Vue app at / (built with @ideonate/evals-viewer-core) consumes it.

On-disk contract

The plugin reads files in this layout:

{resultsDir}/{run_id}/
├── run.json                       (timestamp, git_commit, tags)
└── {eval_name}/
    ├── summary.json               (aggregates + per-case scores)
    └── outputs/
        └── {case_name}.json       (full per-case output)

Plus optional inputs/, case-scores/, and a sidecar tags.json. See docs/data-layout.md for the full schema.

The Python evals-viewer-io package writes this layout from your test suite.

Loader helpers

Need to attach extra fields to the case-detail response (e.g. assessments, transcripts, persona files)? Compose a caseDataLoader from the included building blocks:

import {
  evalsViewerPlugin,
  resolveResultsDir,
  composeLoaders,
  staticCaseInputLoader,
  datasetCaseLoader,
} from "@ideonate/evals-viewer-server";

evalsViewerPlugin({
  resultsDir: resolveResultsDir({
    envVar: "TEST_RESULTS_FOLDER",
    fallback: "./tests/test-results/evals",
  }),
  evalsDir: "./tests/evals",
  caseDataLoader: composeLoaders(
    staticCaseInputLoader(),
    datasetCaseLoader({
      files: {
        assessment: "assessment.json",
        transcript: "transcripts/{person}.json",
      },
    }),
  ),
});

| Helper | Purpose | | --- | --- | | resolveResultsDir({ envVar, fallback }) | Resolve a path from an env var with a fallback | | composeLoaders(...loaders) | Chain multiple async loaders, shallow-merging results | | staticCaseInputLoader() | Loads {evalsDir}/{eval}/data/cases/{case}.json as caseInput | | datasetCaseLoader({ files }) | Parses {dataset}_{person} case names and loads per-dataset files (templates support {dataset} and {person} interpolation) |

You can also write a custom loader from scratch — it's just an async function ({ evalName, caseName, evalDir, evalsDir }) => extras.

API surface served

| Method | Path | Purpose | | ------ | ---- | ------- | | GET | /api/evals | List runs and their evals | | GET | /api/evals/:runId/:evalName/summary | Return summary.json | | GET | /api/evals/:runId/:evalName/case/:caseName | Composed case detail | | POST | /api/evals/:runId/tags | Add a user tag | | DELETE | /api/evals/:runId/tags/:tag | Remove a user tag | | DELETE | /api/evals/:runId | Delete a whole run from disk |

Lower-level usage

If you don't want the Vite plugin wrapper, use createEvalsApiMiddleware directly with any Connect-compatible server:

import { createEvalsApiMiddleware } from "@ideonate/evals-viewer-server";

const middleware = createEvalsApiMiddleware({ resultsDir: "..." });
app.use(middleware);

License

MIT