prompt-ceo

prompt-ceo is two things in one package:

1. A publishable APO (automatic prompt optimization) library on top of ceo-engine - generators, the provider abstraction, config to engine wiring, analytics, the session format, and visualization components. Other hosts (e.g. ChainForge) import these and supply only their own adapters. See Using prompt-ceo as a library.
2. A web dashboard that consumes that library to run and inspect experiments, adding concrete providers, a SQLite store, and a live streaming UI.

Where ceo-engine is a pure select/observe bandit core, prompt-ceo supplies the APO layer a real run needs: LLM provider calls, response scoring, prompt generators, and the visualization of the search.

Relationship to ceo-engine

prompt-ceo depends on ceo-engine as a local file dependency ("ceo-engine": "file:../ceo-engine"), so ceo-engine must be checked out as a sibling directory next to prompt-ceo. The postinstall script builds ceo-engine automatically, so a fresh install produces a working setup. Treat ceo-engine as the stable optimization core and prompt-ceo as the consumer that adapts it to a running application.

Prerequisites

• Node.js (version 20 or newer).
• API keys for whichever LLM providers you intend to call, supplied through the environment.

Install and run

The project works with npm, pnpm, yarn, or bun. Install (which also builds ceo-engine via the postinstall step) and start the dev server:

npm install   && npm run dev
pnpm install  && pnpm dev
yarn          && yarn dev
bun install   && bun run dev

The dev server runs a custom Node entry (server.ts) that starts Next.js together with a WebSocket server at the /ws path. The WebSocket channel streams engine events (selections, observations, posterior updates) to the browser as a run progresses.

Scripts (run with the run form for your package manager, for example npm run dev, pnpm dev, yarn dev, or bun run dev):

• dev: development server (Next.js plus WebSocket, under tsx).
• build: production Next.js build.
• start: run the production server.
• build:lib: build the publishable library subset (see below).

Tests and verification

The library code is covered by a vitest suite. The single verify gate runs the tests, the library build, and both packaging guards:

npm test                  # vitest (unit tests for generators, viewmodels, ui)
npm run verify            # test + build:lib + test:browser-clean + test:node-entries

• test:browser-clean asserts the browser-safe entries (./core, ./engine, ./types, ./generators, ./providers, ./viewmodels, ./ui) import no Node-only package or LLM SDK - the boundary that keeps the library usable in a browser host.
• test:node-entries asserts the node-only entries (./storage/sqlite, ./evaluator/vm) import cleanly under Node.

For interactive checks without spending provider tokens, the mock run path drives the full UI against synthetic distributions. The bandit algorithms themselves are covered by the ceo-engine test suite.

Architecture

The custom server in server.ts hosts the Next.js app and the WebSocket server in one process. A run is created through the REST API, executed by a shared run loop, persisted to SQLite, and broadcast to any connected browsers.

Library code lives under libs/:

• engine.ts: buildEngine(session) constructs a configured engine plus an in-memory event wrapper, resolving strategies, generators, and objectives from a session description.
• types.ts: the session and configuration types, including the discriminated union of generator kinds.
• core/: the runtime-agnostic run loop, evaluator, provider registry, local runtime, code evaluation, and the SQLite storage implementation.
• generators/: automatic prompt-optimization generators (for example pattern space, OPRO, APE, EvoPrompt, ProTeGi, PromptBreeder).
• providers/: Anthropic, OpenAI, Gemini, and Vertex callers registered into a default provider registry.
• db/: the better-sqlite3 schema and migrations for experiments, sessions, iterations, responses, observations, posterior snapshots, snapshots, templates, and breakpoints.
• ui/: presentational components (heatmap, tree visualization, inspector, winner, budget, control, snapshot graph) shared by the app and published by the library build.
• store.ts, ws.ts, client.ts, posteriors-snapshot.ts, eval.ts, mock/: the in-process session store, WebSocket forwarding, browser API client, posterior aggregation, evaluation glue, and a mock run path for UI development.

Application code lives under app/:

• api/: REST routes for starting and controlling runs, listing sessions, pattern lookups, combination validation, mock runs, and health.
• components/: setup, run, and results views plus configuration panels and hooks.
• mock/ and heatmap-mock/: sandbox pages for editing distributions and inspecting the heatmap component in isolation.

Using prompt-ceo as a library

npm run build:lib (tsup) emits the reusable pieces as separate subpath entry points. A host imports only what it needs.

Entry map

| Import | Contents | Environment | | --- | --- | --- | | prompt-ceo / prompt-ceo/core | run loop, evaluator, provider registry | isomorphic | | prompt-ceo/engine | buildEngine(session) config to engine wiring | isomorphic | | prompt-ceo/types | session and config types (CEOSession, GeneratorConfig, PatternGeneratorConfig) | isomorphic | | prompt-ceo/generators | generator classes + buildGenerator(config, ctx) factory + callMetaLLM | isomorphic | | prompt-ceo/providers | provider abstraction (registerProvider, resolveProvider, Provider) | isomorphic | | prompt-ceo/viewmodels | Mantine-free transforms: vsupColor/certainty, layoutSnapshots, snapshotsToTree/checkpointsToTree | isomorphic | | prompt-ceo/ui | Mantine components: Heatmap, SnapshotGraph, Inspector | browser (Mantine) | | prompt-ceo/storage/sqlite | better-sqlite3 storage adapter | Node only | | prompt-ceo/evaluator/vm | node:vm code evaluator | Node only |

Boundary rule

The browser-safe entries never import a host store, queryLLM, a UI framework beyond Mantine (for ./ui), a Node-only package, or a concrete LLM SDK. The provider SDKs and Node-only packages are optionalDependencies, so a browser host installs none of them. npm run test:browser-clean enforces this.

Integrating a host

1. Provider - register one provider for your prefix so generators can call a meta-LLM with registerProvider.
2. Generators - build from a declarative config with the shared factory: buildGenerator(config, { inputCols, rows, expectedCols }).
3. Visualization - feed the snapshot graph through an adapter rather than your own data shape by passing adapter output into SnapshotGraph (or snapshotsToTree); inject richer heatmap uncertainty via the certaintyFor prop. Non-Mantine hosts use prompt-ceo/viewmodels and render their own components on the same transforms.

Consuming it (file dependency)

prompt-ceo is consumed the same way it consumes ceo-engine: as a local file: dependency pointing at a prebuilt dist. A host adds, for example, "prompt-ceo": "file:../prompt-ceo", and builds the library (e.g. from its own postinstall, mirroring how prompt-ceo builds ceo-engine). Only dist, README.md, and CHANGELOG.md are shipped (files). React, Mantine, ceo-engine, and the optional Node/SDK packages are external peers, not bundled.

This package is pre-1.0; see CHANGELOG.md for API changes.

Persistence

State that is serializable lives in SQLite; live engine wrappers (the running engine, abort controllers, and WebSocket subscribers) stay in process memory and are not persisted. This split lets the dashboard restart and rehydrate completed work from the database while in-flight runs remain tied to their process.