@polebot/eval

@polebot/eval turns Pole requests into JSON fixtures and tests one layer of your bot against them. record folds the events of each request into one Recording. replay runs your own gate, normalize, input, or preview function on a recording and reports each field where the output differs from what was recorded.

The package depends on @polebot/protocol only. It never imports a trigger or a runtime. Your application passes the layer functions to replay.

The API reference is in the TSDoc of each export, so your editor shows it on hover.

Install

pnpm add @polebot/eval @polebot/protocol

Privacy

A recording holds message content: the snapshot of the arrival, the stored event and the request context, and any input you add to it. Store recordings where you would store the messages themselves, and delete them on the same schedule.

Recording is off unless you wire it. record does nothing until you pass its onEvent and drain to serve.

Record requests

import { appendFile } from 'node:fs/promises';
import { record } from '@polebot/eval';
import { serve } from '@polebot/protocol';

const recorder = record({
	sink: recording => appendFile('recordings.jsonl', `${JSON.stringify(recording)}\n`),
});

const served = await serve({
	runtime,
	triggers: [discord.trigger],
	onEvent: recorder.onEvent,
	drain: recorder.drain,
});

// On shutdown:
await served.stop();
await recorder.flush();

record keys each request by traceId and hands it to sink once, when the request ends. A request that ends with a wide event carries it as wide. A drop below the drain's level floor arrives without one. flush hands over the requests still open and waits for every sink call. It rejects with the error of a sink call that failed.

What a recording holds

| Field | Layer | Filled by | | --- | --- | --- | | traceId, requestId, conversationKey, trigger | | record | | snapshot | source | record, from the source of intake.received | | decision | gate | record: accepted, resumed, or the drop reason | | event | context | record, from request.accepted | | context | context | record, from request.accepted | | input | input | you | | preview | act | you, such as from the runtime's preview | | wide | record | record, from the drain |

request.accepted carries the stored event and the request context when the trigger reports them, as the Discord and scheduler triggers do. A resumed request has neither. No event carries the input or the preview, because the input can hold attachment bytes. Add them to a recording before you save it as a fixture, or rebuild the input with replay from event and context.

Replay one layer

import { readFile } from 'node:fs/promises';
import { replay, type Recording } from '@polebot/eval';

const recording: Recording = JSON.parse(await readFile('fixtures/deploy-question.json', 'utf8'));

const { actual, diffs } = await replay(recording, {
	from: 'gate',
	layers: { gate: async snapshot => myGate(snapshot) },
});

The layers run in this order: gate, normalize, input, preview. replay starts at from and runs each later layer you pass. Each layer reads the output of the layer before it, or the recorded value when that layer did not run. A gate that does not pass ends the replay.

| Layer | Reads | Writes | | --- | --- | --- | | gate | snapshot | decision | | normalize | snapshot | event | | input | event, context | input | | preview | requestId, conversationKey, input, context | preview |

A passing gate agrees with every decision that the trigger makes after its gate: accepted, resumed, duplicate, and stopped. A gate that throws decides predicate-failed, as the trigger does.

diffs has one entry per field whose JSON differs from the recording, with expected and actual. Key order does not count. A field the recording lacks diffs against null. replay rejects when a layer needs a field that the recording lacks, such as gate without a snapshot.

Known limits

• snapshot is the source of intake.received: what the trigger knew at arrival, not the full outside event.
• record drops a request that ends before it has a decision, such as one lost during intake.