@withgateway/sdk

TypeScript SDK for Gateway — A/B experiments, tracing, and observability for LLM agents.

Installation

npm install @withgateway/sdk

# For tracing integration (required for tracing.init())
npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources

Quick Start

Option 1: One-Line Setup (Recommended)

import { tracing, experiments } from "@withgateway/sdk";

// 1. Initialize tracing - reads from env vars and sets up everything
tracing.init();

// 2. Initialize experiments
experiments.init({
  apiKey: process.env.GATEWAY_PUBLIC_KEY!,
  secretKey: process.env.GATEWAY_SECRET_KEY!,
  baseUrl: process.env.GATEWAY_HOST!,
});

// 3. Register and use experiment
await experiments.register({
  key: "prompt-test",
  variants: [
    { key: "control", weight: 50, config: { prompt: "Be helpful." } },
    { key: "treatment", weight: 50, config: { prompt: "Be concise." } },
  ],
});

const result = await experiments.getVariant("prompt-test", "user-123");

// 4. Shutdown on exit
process.on("SIGTERM", async () => {
  await tracing.shutdown();
  await experiments.shutdown();
});

Environment variables for tracing.init():

export GATEWAY_HOST="https://withgateway.ai"
export GATEWAY_PUBLIC_KEY="pk-lf-..."
export GATEWAY_SECRET_KEY="sk-lf-..."

Option 2: Explicit Configuration

import { tracing, experiments } from "@withgateway/sdk";

// Initialize with explicit config
tracing.init({
  host: "https://withgateway.ai",
  publicKey: "pk-lf-...",
  secretKey: "sk-lf-...",
  serviceName: "my-agent",
  environment: "production",
  debug: true, // Enable console logging
});

experiments.init({
  apiKey: "pk-lf-...",
  secretKey: "sk-lf-...",
  baseUrl: "https://withgateway.ai",
});

Option 3: Manual OpenTelemetry Setup (Advanced)

If you need full control over OpenTelemetry configuration:

import { experiments, ExperimentSpanProcessor } from "@withgateway/sdk";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

const provider = new NodeTracerProvider({
  spanProcessors: [
    new ExperimentSpanProcessor(), // Add BEFORE exporter
    new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: `${process.env.GATEWAY_HOST}/api/public/otel/v1/traces`,
        headers: {
          Authorization: `Basic ${Buffer.from(
            `${process.env.GATEWAY_PUBLIC_KEY}:${process.env.GATEWAY_SECRET_KEY}`,
          ).toString("base64")}`,
        },
      }),
    ),
  ],
});
provider.register();

Spans created after getVariant() are automatically tagged with:

• gateway.experiment.id — experiment key
• gateway.variant.key — assigned variant
• gateway.experiment.user_id — user ID

API Reference

Tracing API

import { tracing } from "@withgateway/sdk";

tracing.init(config?)           // Initialize OTel + Gateway exporter (call once)
tracing.flush()                 // Force flush pending spans
tracing.shutdown()              // Flush + shutdown (call on exit)
tracing.isInitialized()         // Check if initialized
tracing.isGatewayConfigured()   // Check if env vars are set

Config options:

tracing.init({
  host: string,              // Gateway server URL (or GATEWAY_HOST env var)
  publicKey: string,         // Auth key (or GATEWAY_PUBLIC_KEY env var)
  secretKey: string,         // Secret key (or GATEWAY_SECRET_KEY env var)
  serviceName?: string,      // Service name for traces (default: "gatewaysdk-tracing")
  environment?: string,      // Environment tag (e.g., "production")
  version?: string,          // Version tag (e.g., "1.2.3")
  debug?: boolean,           // Enable console logging (default: false)
  enableExperiments?: boolean, // Add ExperimentSpanProcessor (default: true)
});

Experiments API

import { experiments } from "@withgateway/sdk";

experiments.init(config)              // Initialize (call once)
experiments.register(input)           // Register an experiment
experiments.getVariant(key, userId)   // Get variant for user
experiments.fetch(keys?)              // Fetch experiments from backend
experiments.clearExperimentContext()  // Clear context (call after each request)
experiments.shutdown()                // Cleanup (call on exit)

Scores & Success Signals

Record evaluation scores and ROI-shaped success signals against a project. A success signal is a score tagged with gateway.success.signal, which the platform's Success Metrics feature aggregates into a metric's headline count / pass-rate (the HOOK metric source). The score shape is identical to the Python SDK's run.success(), so signals from either SDK aggregate the same.

import { ScoresClient, success } from "@withgateway/sdk/scores";

// Env-based (GATEWAY_HOST / GATEWAY_PUBLIC_KEY / GATEWAY_SECRET_KEY):
await success("resolved_ticket", true, {
  agentId: "support-triage",
  sessionId,
});
await success("handle_time_s", 42, { agentId: "support-triage", sessionId });

// Explicit keys:
const scores = new ScoresClient({ host, publicKey, secretKey });
await scores.success("resolved_ticket", true, {
  agentId: "support-triage",
  sessionId,
});
await scores.score("rubric", 0.9, { sessionId }); // a plain (non-signal) score

A boolean value is recorded as a 1/0 BOOLEAN score (pass/fail); a number is a NUMERIC score. metricName defaults to the signal name. Provide exactly one target — sessionId, traceId (+ optional observationId), or datasetRunId; when none is given the active OpenTelemetry trace id is used, and if there is none the call is a fail-soft no-op. Every score/success call is fail-soft: a bad key or transport failure warns and resolves, never throwing into your run. (postScore is the raw variant that DOES throw.)

Under tracing.initIsolated(...), the returned handle carries a scores client bound to that provider's project keys:

const handle = tracing.initIsolated({
  host,
  publicKey,
  secretKey,
  agent: "support-agent",
});
await handle.scores.success("resolved_ticket", true, {
  agentId: "support-agent",
  sessionId,
});

Security Wrapper

Use the security wrapper when you want Gateway to control tool visibility, authorization, credential bindings, and evidence while your agent framework keeps owning the native tool objects.

import { gateway } from "@withgateway/sdk/security";
import { streamText, tool } from "ai";

const gw = gateway.fromEnv({
  agent: {
    id: "support-triage",
    name: "Support Triage",
    version: "1.4.2",
  },
});

const tools = await gw.secureTools({
  toolsetId: "zendesk",
  provider: "zendesk",
  tools: {
    searchTickets: tool({
      description: "Search Zendesk tickets",
      inputSchema: searchSchema,
      execute: async (input) => zendesk.search(input),
    }),
    deleteTicket: tool({
      description: "Delete a Zendesk ticket",
      inputSchema: deleteSchema,
      execute: async (input) => zendesk.delete(input),
    }),
  },
  context: {
    userId: currentUser.id,
    teamId: "support",
    environment: "production",
    traceId,
    sessionId,
  },
});

await streamText({ model, prompt, tools });

secureTools calls /api/public/security/toolsets/resolve, hides blocked tools in filter/enforce, and calls /api/public/security/tool-calls/authorize immediately before execution. It also stamps Gateway security attributes on the active OpenTelemetry span when tracing is installed.

Class-Based API (Advanced)

import { ExperimentClient } from "@withgateway/sdk";

const client = new ExperimentClient({
  apiUrl: "https://withgateway.ai",
  apiKey: "pk-lf-...",
  secretKey: "sk-lf-...",
  projectId: "...",
});

Worlds & Live Sessions

A world is a versioned environment the platform hosts — tasks, tools, seed data and a grader, addressed as slug@ref. A session boots one task's container and holds it warm while your agent, running wherever it runs, calls its tools.

The distinction that matters: the agent is local, the world is not. Every tool call executes inside the platform's container, so you are testing against the environment as it actually is rather than a copy unwrapped on your own disk.

import { runSessions } from "@withgateway/sdk";

const report = await runSessions({
  world: "revenue-ops-sim@main",
  concurrency: 4, // one warm container per task
  failUnder: 0.7, // report.passed reads the mean
  agent: async ({ toolkit, instruction }) => {
    // The world's own tool schemas and callable implementations — nothing
    // to hand-write, nothing to keep in sync.
    await myAgent(toolkit.schemas, toolkit.impls, instruction);
  },
});

console.log(report.mean, report.passed);
process.exit(report.passed ? 0 : 1);

runSessions resolves the world once, so every task pins the same version and a push mid-run cannot split results across two worlds. A task that throws scores 0 and records why; the rest keep their results. Containers are handed back warm, so the next run starts hot.

Driving a single session directly, when you want the control:

import { openSession } from "@withgateway/sdk";

const session = await openSession(
  "revenue-ops-sim",
  "revops-mara-dedupe-contacts",
);
await session.ready(); // wait for the container
await session.call("crm_search", { entity: "deals" });
const { reward, rewards } = await session.grade(); // the task's own grader
await session.close({ keepWarm: true });

Sessions trace themselves from the platform — one trace per episode, one span per tool call — so an episode is readable even if your process dies mid-run.

Full worked example: examples/world-suite.ts is a complete local production harness — a real tool-calling agent loop, parallel execution, a pass gate, a JSON artifact and a CI exit code. The Python twin is gatewaysdk/examples/worlds/world_suite.py.

There is also a CLI for pushing, pulling and running worlds:

npx gateway bench --help

Cross-Language Compatibility

This SDK produces identical assignments to the Python SDK (gatewaysdk) for the same (experimentKey, userId) pair. You can run experiments across services in different languages.

Documentation

• Full A/B Experiments Guide
• TypeScript SDK Deep Dive

License

MIT