ceo-engine

A bandit-algorithm-based prompt optimizer for large language models.

Given a set of models, prompt templates, and inputs, ceo-engine searches for the combination that maximizes a quality signal within a budget. It organizes the search space as a three-level tree (Model, then Template, then Prompt) and runs multi-armed bandit strategies over that tree.

The library is provider-agnostic: it never calls an LLM itself. The caller drives a select/observe loop, sends the selected prompt to whatever LLM client they use, scores the response, and reports the result back. This keeps the engine deterministic and testable, and keeps all network and key handling on the caller's side.

Installation

ceo-engine is published to the npm registry and installs with any of the common package managers:

npm install ceo-engine
pnpm add ceo-engine
yarn add ceo-engine
bun add ceo-engine

The package ships dual type declarations and ESM, so it works the same across all four toolchains. Within this workspace it is consumed locally as a file dependency instead:

{
  "dependencies": {
    "ceo-engine": "file:../ceo-engine"
  }
}

Local development

Clone the repository, install dependencies, and build the distributable (type declarations plus ESM into dist):

npm install   && npm run build
pnpm install  && pnpm build
yarn          && yarn build
bun install   && bun run build

dev runs the local entry under tsx for quick experiments, and clean removes the dist directory.

Tests

The suite runs under Mocha, loading TypeScript through tsx (configured in .mocharc.json); specs live in test/:

npm test          # pnpm test, yarn test, or bun run test
npm run test:coverage   # coverage report via nyc

Note for bun users: run scripts as bun run test rather than bun test, since bun test invokes bun's own test runner instead of this package's Mocha script.

Quick start

import {
  PromptCeoEngine, TokenBudget,
  ThompsonSamplingSelector, LevelTraversal,
  LeafPropagation, NoPruning,
} from 'ceo-engine';

const engine = new PromptCeoEngine({
  models:    [{ id: 'anthropic:claude-haiku-4-5-20251001' }],
  templates: [
    { id: 'direct', content: '{{question}}' },
    { id: 'cot',    content: 'Think step by step.\n\n{{question}}' },
  ],
  variables:        [{ question: 'What is 2+2?' }],
  budget:           new TokenBudget(10000),
  treeTraversal:    new LevelTraversal(),
  nodeSelector:     new ThompsonSamplingSelector(),
  rewardAssignment: new LeafPropagation(),
  pruneStrategy:    new NoPruning(),
  optimizationMode: { type: 'leaf' },
  costPenalty:      0,
});

while (!engine.budget.exhausted) {
  const sel = engine.select();
  const response = await myLLMClient(sel.promptText);
  const quality  = myScorer(response);
  engine.observe(sel.selected, {
    quality,
    cost: countTokens(sel.promptText, response),
    rawResponses: [response],   // optional, included in engine.history
  });
}

// engine.history is populated automatically, no manual tracking needed
const { history } = engine;

To silence logging:

import { setLogger, NoopLogger } from 'ceo-engine';
setLogger(new NoopLogger());

How it works

A run is configured by four pluggable algorithm slots plus a budget:

• nodeSelector: chooses which arm to pull at each tree level (the bandit policy).
• treeTraversal: decides how the tree is walked when selecting and expanding.
• rewardAssignment: propagates an observed reward through the tree.
• pruneStrategy: removes unpromising branches as evidence accumulates.

The budget is the stopping condition. optimizationMode is either leaf (evaluate prompt leaves directly) or structure (evaluate aggregated samples per structure node). costPenalty turns on cost-aware scalarization so the reward trades quality against token cost.

Public API

src/index.ts re-exports the full surface. The main groups:

• High-level prompt optimization: PromptCeoEngine, evaluation, generator interface, analytics, session, and controller (from prompt-ceo/).
• Algorithm classes for each slot (selectors, traversals, rewards, pruning).
• Posteriors, budgets, and scaling/scalarization strategies.
• factories/: preset strategy bundles.
• registry/: runtime plugin system, including createSelector, createPosterior, createBudget, and createEngine for building components from string ids.
• Pareto helpers (computeParetoFrontier, dominates, computeCrowdingDistance) and logging (setLogger, NoopLogger).

Module map

• prompt-ceo/: high-level public API on top of the core engine.
• engines/: the tree-based bandit engine and base interfaces.
• selectors/: ThompsonSampling, UCB, Random, SuccessiveHalving, Pareto, Active.
• traversals/: Level, Depth, Beam, MCTS.
• rewards/: Leaf, Full, Decay, Depth, Entropy propagation.
• pruning/: NoPruning, TopK, ConfidenceBound, SuccessiveHalving, SequentialHalving.
• posteriors/: Beta, NIG, multi-objective, and preference/feedback.
• budgets/: Token, Time, Api, Human, Composite.
• scaling/: CEOScaler, Chebyshev, Adaptive, QualityOnly, weighted scalarization.
• factories/: strategy preset bundles.
• registry/: runtime component registration and string-id construction.
• tree/, primitives/, adapters/, utils/: tree structure, shared types, integration adapters, and logging.

Reproducibility

Seeded runs are byte-exact across versions: under a fixed seed the numeric sequence the engine emits is preserved. Any change that alters the order or count of pseudo-random draws (the samplers in utils/random.ts, posteriors/nig.ts, Beta sampling, or Fisher-Yates shuffling) breaks this guarantee and is avoided.

Roadmap

• Multi-objective reward propagation (a vector of rewards up the tree) alongside the current single-scalar reward contract.
• Preset configuration bundles in configs/presets.ts (for example Presets.default(), Presets.fast(), Presets.explore()).
• Per-algorithm static metadata (id, name, description, hyperparameter schema) to enable config serialization, sweeps, and generated docs.
• Self-registering algorithms so importing a class is enough to make it available by string name, with a runtime-enumerable algorithm catalogue exported from the public API.
• Hyperband bracket scheduler wrapping Successive Halving.
• A headless adapter directory and a planned Python wrapper that bundles the compiled JS and drives the same select/observe loop over a subprocess.