@freesolo/sdk

TypeScript SDK surface for source applications that need Freesolo tracing or custom evaluations.

This npm package intentionally contains only:

• tracing helpers for exporting OpenTelemetry spans to Freesolo
• evaluation primitives and EvaluationClient

It does not include Freesolo training, datasets, GEPA, AutoSLM, or generated Python training-repo helpers.

Tracing

import { setupTracing, call, episode, span } from "@freesolo/sdk/tracing";

setupTracing({
  projectId: "project-id",
  scorerBundleId: "scorer-bundle-id",
});

const response = await call(
  "openai.chat.completions",
  {
    input: { messages },
    provider: "openai",
    model: "gpt-4o-mini",
  },
  async (trace) => {
    const result = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      body: JSON.stringify({ model: "gpt-4o-mini", messages }),
    }).then((r) => r.json());
    trace.setOutput(result);
    return result;
  },
);

setupTracing(...) is no-op safe when FREESOLO_API_KEY is absent. Trace export times out after 120 seconds.

For multi-step agents, create one root span for the bounded run with span(...), then nest tool, retrieval, validation, and call(...) spans inside it. Attach trace.episode to the root span only when a scorer needs whole-conversation evidence; it should not replace the child span tree.

const answer = await span(
  "support_agent.run",
  { input: { message }, kind: "agent" },
  async (runTrace) => {
    const context = await span(
      "support_agent.retrieve_context",
      { kind: "tool" },
      async (toolTrace) => {
        const result = await retrieveContext(message);
        toolTrace.setOutput(result);
        return result;
      },
    );

    const response = await call(
      "openai.chat.completions",
      {
        input: { message, context },
        provider: "openai",
        model: "gpt-4o-mini",
      },
      async (llmTrace) => {
        const result = await callModel(message, context);
        llmTrace.setOutput(result);
        return result;
      },
    );

    runTrace.setOutput(response);
    runTrace.setEpisode(episode({
      input: message,
      messages: [
        { role: "user", content: message },
        { role: "assistant", content: response },
      ],
      responseText: response,
    }));
    return response;
  },
);

Evaluation

import { BinaryResponse, CustomScorer, EvaluationClient } from "@freesolo/sdk/evaluation";

class NonEmpty extends CustomScorer<BinaryResponse> {
  name = "non_empty";

  score(row: Record<string, unknown>) {
    return new BinaryResponse({
      value: Boolean(String(row.actual_output ?? "").trim()),
      reason: "actual_output is non-empty",
    });
  }
}

await new EvaluationClient().run({
  name: "local-check",
  data: [{ actual_output: "hello" }],
  scorers: [new NonEmpty()],
});