@statsig/statsig-ai
v1.0.0
Published
Statsig is the on-stop data platform for product development. We help you move faster with feature gates (feature flags), experiments, and the world's most powerful stats engine. If you're new to Statsig, check out our product and create an account at [st
Readme
Statsig Node AI SDK
Statsig is the on-stop data platform for product development. We help you move faster with feature gates (feature flags), experiments, and the world's most powerful stats engine. If you're new to Statsig, check out our product and create an account at statsig.com.
The AI SDK provides convenient hooks for developers working with LLMs to iterate on their promps, run evals to test their efficacy, add observability to LLM usage.
Getting Started
Read through the Documentation to see code samples and how to get started.
Tracing model calls
Wrap your provider client to record each model call as an OpenTelemetry span. Calls made inside prompt.withLive(...) are tagged with the prompt name, version name and version ID.
import { wrapAnthropic, wrapGemini, wrapOpenAI } from '@statsig/statsig-ai';
const openai = wrapOpenAI(new OpenAI());
const anthropic = wrapAnthropic(new Anthropic()); // messages.create (incl. stream: true) and messages.stream()
const gemini = wrapGemini(new GoogleGenAI({})); // models.generateContent / generateContentStreamMessages are only recorded when you opt in with captureOptions (for example { captureOptions: { capture_all: true } }). All wrappers record gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions in the OpenTelemetry GenAI message format (role + parts such as text, tool_call and tool_call_response).
Tagging calls with a prompt version
Run the code that renders and sends a prompt inside prompt.withLive(...), or prompt.withVersion(version, ...) for a Candidate. Wrapped model calls and wrap spans inside it get statsig.llm.prompt_name, statsig.llm.prompt_version and statsig.llm.prompt_version_id, which Statsig uses to attribute online eval grades to a version.
The variables you pass to getPromptMessages are recorded as statsig.llm.prompt_template_variables (JSON), so the request can be re-run on another version. Model call spans only record them when input capture is on (capture_all or capture_input_messages). Workflow spans record the variables rendered inside them, so start the workflow inside the version scope.
const prompt = ai.getPrompt(user, 'support_bot');
const runTurn = wrap({ type: 'workflow', name: 'support_turn' }, (version) =>
openai.chat.completions.create({
model: version.getModel(),
messages: version.getPromptMessages({ input }),
}),
);
await prompt.withLive(runTurn);Marking the environment
The environment option of initializeTracing (defaults to NODE_ENV) is recorded on the resource as deployment.environment.name. To tag only some spans, for example evals running in the same process as production code, run that code inside withStatsigEnvironment. Spans started inside it get a statsig.environment attribute, which overrides the resource value for those spans.
Use production for live traffic. eval and online_eval_judge are reserved for spans Statsig generates: Eval() runs its tasks and scorers with statsig.environment=eval.
initializeTracing({ environment: 'production' });
await withStatsigEnvironment('staging_canary', () => handleRequest(req));Tracing evals
When tracing is initialized, Eval() gives each record its own trace. The root span is a workflow span marked statsig.environment=eval, with statsig.eval.name, statsig.eval.run_name and statsig.eval.record_index. LLM calls the task makes through wrapOpenAI, wrapAnthropic or wrapGemini are its children, and each scorer runs in its own child span. Each result's traceID and spanID are sent to Statsig with the scores, and Eval() flushes the spans before it returns.
Joining a Statsig eval trace
A workflow eval sends each dataset row to your endpoint as a POST request. To show your agent's tool and model calls under the eval row in the trace drawer, run the handler inside withIncomingTraceContext. It reads the W3C traceparent and tracestate request headers, so wrapped model calls and wrap spans become children of the eval row's span. It accepts Node request headers (req.headers) or a Fetch Headers object. If there's no valid traceparent, the handler runs as usual.
import { withIncomingTraceContext } from '@statsig/statsig-ai';
// Node / Express
app.post('/agent', (req, res) =>
withIncomingTraceContext(req.headers, async () => {
res.json({ output: await runAgent(req.body) });
}),
);
// Fetch-style handlers (Next.js route handlers, Hono, ...)
export async function POST(request: Request) {
return withIncomingTraceContext(request.headers, async () =>
Response.json({ output: await runAgent(await request.json()) }),
);
}If your server already uses OpenTelemetry HTTP instrumentation, its spans join the trace on their own. The helper keeps the active span when it's already in the incoming trace, so using both is safe.
What Statsig sends: workflow eval requests carry Content-Type: application/json, the headers you configured for the workflow, and two headers of their own. traceparent names the eval row's root span, and x-statsig-eval-run holds the eval run ID. A header you configure with the same name wins, matched case-insensitively. Test Connection doesn't send either header.
Spans from your endpoint get your app's environment, so eval requests land with your production traces, where online evals can grade them. To keep them out, check for x-statsig-eval-run and run the handler inside withStatsigEnvironment('eval', ...). You can use the same check to skip side effects.
import {
withIncomingTraceContext,
withStatsigEnvironment,
} from '@statsig/statsig-ai';
app.post('/agent', (req, res) =>
withIncomingTraceContext(req.headers, () => {
const handle = async () => {
res.json({ output: await runAgent(req.body) });
};
return req.headers['x-statsig-eval-run']
? withStatsigEnvironment('eval', handle)
: handle();
}),
);Running evals on a Statsig dataset
Pass statsigDataset(name, toRecord) as Eval()'s data to run on an eval dataset from the console. toRecord says which columns hold the input and the expected value. The run names the dataset and each result keeps its dataset row id, so the Eval page compares runs on the dataset row by row.
import { Eval, statsigDataset } from '@statsig/statsig-ai';
await Eval('support_bot_quality', {
data: statsigDataset('support_cases', (row) => ({
input: row.columns.question,
expected: row.columns.answer,
category: row.groups,
})),
task: (input) => answer(input),
scorer: ({ output, expected }) => output === expected,
});To post your own records for a dataset instead, set dataset and give each record the rowID of its dataset row.
Latency and tokens
Eval() times each record's task, from the call until its output is ready, and sends that with the result as latencyMs. Scorers aren't part of it.
The runner can't see your model calls, so the task reports its tokens with reportTokens from its second argument. Each call adds to the record's totals, so a task that makes several model calls can report each one, or its total once. The totals are sent as inputTokens and outputTokens.
await Eval('support_bot_quality', {
data,
task: async (input, { reportTokens }) => {
const { choices, usage } = await openai.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: input }],
});
if (usage) {
reportTokens({
inputTokens: usage.prompt_tokens,
outputTokens: usage.completion_tokens,
});
}
return choices[0].message.content ?? '';
},
scorer: ({ output, expected }) => output === expected,
});The run page shows a Latency column, and a Tokens column when the task reported tokens. Neither is graded. Eval() also returns them on each result. A record whose task throws sends neither, and a record whose task never calls reportTokens sends no tokens. reportTokens ignores a call whose counts aren't finite numbers of at least 0, and logs a warning.
Community
If you need any assistance or just have a question, feel free to reach out to us on Slack.
