Ashr Labs TypeScript SDK

A TypeScript client library for evaluating AI agents against Ashr Labs test datasets.

Documentation

• Testing Your Agent — start here
• Quick Start Guide
• Installation
• Authentication
• API Reference
• Error Handling
• Examples

Installation

npm install ashr-labs

Quick Start

import { AshrLabsClient, EvalRunner } from "ashr-labs";

// Only need your API key — baseUrl and tenantId are automatic
const client = new AshrLabsClient("tp_your_api_key_here");

// Fetch a dataset and run your agent against it
const runner = await EvalRunner.fromDataset(client, 42);
const run = await runner.run(myAgent);

// Inspect results
const metrics = run.build().aggregate_metrics as Record<string, unknown>;
console.log(`Passed: ${metrics.tests_passed}/${metrics.total_tests}`);
console.log(`Avg similarity: ${metrics.average_similarity_score}`);

// Submit results
await run.deploy(client, 42);

Your agent just needs two methods:

import type { Agent } from "ashr-labs";

const myAgent: Agent = {
  async respond(message: string) {
    // Call your LLM, return { text: "...", tool_calls: [...] }
    return { text: "response", tool_calls: [] };
  },

  async reset() {
    // Clear conversation history between scenarios
  },
};

See Testing Your Agent for a full end-to-end guide.

Observability — Production Tracing

Trace your agent in production. Captures LLM calls, tool invocations, and events. Never rejects — if the backend is unreachable, errors are logged silently.

// wrap() pattern — auto-end on completion, auto-capture errors
await client.trace("handle-ticket", { userId: "user_42" }).wrap(async (trace) => {
  const gen = trace.generation("classify", { model: "claude-sonnet-4-6",
    input: [{ role: "user", content: "help" }] });
  const result = await callLlm(...);
  gen.end({ output: result, usage: { input_tokens: 50, output_tokens: 12 } });

  await trace.span("tool:search", { input: { q: "..." } }).wrap(async (s) => {
    const data = await search(...);
    s.end({ output: data });
  });
});

// Analytics
const analytics = await client.getObservabilityAnalytics(7);
console.log(`Traces: ${analytics.overview.total_traces}`);
console.log(`Tool calls: ${analytics.overview.total_tool_calls}`);

See API Reference for full Trace/Span/Generation docs.

VM Stream Logs

Attach virtual machine session logs to test results for browser-based or desktop-based agents:

test = run.addTest("checkout_flow");
test.start();
// ... run agent, add tool calls and responses ...

// Kernel browser session (first-class support)
test.setKernelVm("kern_sess_abc123", {
  durationMs: 15000,
  logs: [
    { ts: 0, type: "navigation", data: { url: "https://app.example.com" } },
    { ts: 1200, type: "action", data: { action: "click", selector: "#login" } },
  ],
  replayId: "replay_abc123",
  replayViewUrl: "https://www.kernel.sh/replays/replay_abc123",
  stealth: true,
  viewport: { width: 1920, height: 1080 },
});

// Or use the generic setVmStream() for any provider
test.setVmStream("browserbase", {
  sessionId: "sess_abc123",
  durationMs: 45000,
  logs: [
    { ts: 0, type: "navigation", data: { url: "https://app.example.com" } },
    { ts: 1200, type: "action", data: { action: "click", selector: "#login" } },
  ],
});
test.complete();

Available Methods

All methods that accept tenantId auto-resolve it from your API key if omitted.

Datasets

| Method | Description | |--------|-------------| | getDataset(datasetId, ...) | Get a dataset by ID | | listDatasets(tenantId, limit, offset, ...) | List datasets |

Runs

| Method | Description | |--------|-------------| | createRun(datasetId, result, ...) | Create a new test run | | getRun(runId) | Get a run by ID | | listRuns(datasetId, tenantId, limit, offset) | List runs | | deleteRun(runId) | Delete a run |

EvalRunner

| Method | Description | |--------|-------------| | EvalRunner.fromDataset(client, datasetId) | Create a runner from a dataset | | runner.run(agent, { maxWorkers }) | Run agent against all scenarios, return RunBuilder | | runner.runAndDeploy(agent, client, datasetId, { maxWorkers }) | Run and submit in one call |

RunBuilder

| Method | Description | |--------|-------------| | new RunBuilder() | Create a new run builder | | run.start() | Mark the run as started | | run.addTest(testId) | Add a test and get a TestBuilder | | run.complete(status) | Mark the run as completed | | run.build() | Serialize to a result object | | run.deploy(client, datasetId) | Build and submit via the API |

TestBuilder

| Method | Description | |--------|-------------| | test.start() | Mark the test as started | | test.addUserFile(filePath, description) | Record a user file upload | | test.addUserText(text, description) | Record a user text input | | test.addToolCall(expected, actual, matchStatus) | Record an agent tool call | | test.addAgentResponse(expectedResponse, actualResponse, matchStatus) | Record an agent response | | test.setVmStream(provider, opts) | Attach VM session logs | | test.setKernelVm(sessionId, opts) | Attach Kernel VM session (convenience) | | test.complete(status) | Mark the test as completed |

Requests

| Method | Description | |--------|-------------| | createRequest(requestName, request, ...) | Create a new request | | getRequest(requestId) | Get a request by ID | | listRequests(tenantId, status, limit, offset) | List requests |

Observability

| Method | Description | |--------|-------------| | client.trace(name, opts?) | Start a production trace (returns Trace) | | trace.span(name, opts?) / trace.generation(name, opts?) | Add spans or LLM calls | | trace.wrap(fn) / span.wrap(fn) | Auto-end on completion, auto-capture errors | | await trace.end(opts?) | Flush trace to backend (never rejects) | | listObservabilityTraces(opts?) | List traces | | getObservabilityTrace(traceId) | Get trace with full observation tree | | getObservabilityAnalytics(days?) | Analytics: tokens, latency, errors, tool perf | | getObservabilityErrors(opts?) | Traces with errors | | getObservabilityToolErrors(opts?) | Traces with tool failures |

API Keys & Session

| Method | Description | |--------|-------------| | init() | Validate credentials and get user/tenant info | | listApiKeys(includeInactive) | List API keys for your tenant | | revokeApiKey(apiKeyId) | Revoke an API key | | healthCheck() | Check if the API is reachable |

Error Handling

import { AshrLabsClient, NotFoundError, AuthenticationError } from "ashr-labs";

const client = new AshrLabsClient("tp_...");

try {
  const dataset = await client.getDataset(999);
} catch (e) {
  if (e instanceof AuthenticationError) {
    console.log("Invalid API key");
  } else if (e instanceof NotFoundError) {
    console.log("Dataset not found");
  }
}

Configuration

// All defaults — just pass API key
const client = new AshrLabsClient("tp_...");

// From environment (reads ASHR_LABS_API_KEY)
const client = AshrLabsClient.fromEnv();

// Custom timeout
const client = new AshrLabsClient("tp_...", undefined, 60);

// Custom base URL (for self-hosted)
const client = new AshrLabsClient("tp_...", "https://your-api.example.com");

Requirements

• Node.js 18+
• TypeScript 5.4+ (recommended)

License

MIT