@withgateway/sdk

TypeScript SDK for Gateway — A/B experiments, tracing, and observability for LLM agents.

Installation

npm install @withgateway/sdk

# For tracing integration (required for tracing.init())
npm install @opentelemetry/api @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node

The gateway CLI installed with npm install -g @withgateway/sdk needs the same packages installed with npm install -g. Without them, runTask, runSet and gateway worlds task run|run-set still run, untraced, and print one warning per process with both install lines, pinned to the versions the SDK is tested with. --no-trace / trace: false turns tracing off and silences it.

Quick Start

Option 1: One-Line Setup (Recommended)

import { tracing, experiments } from "@withgateway/sdk";

// 1. Initialize tracing - reads from env vars and sets up everything
tracing.init();

// 2. Initialize experiments
experiments.init({
  apiKey: process.env.GATEWAY_PUBLIC_KEY!,
  secretKey: process.env.GATEWAY_SECRET_KEY!,
  baseUrl: process.env.GATEWAY_HOST!,
});

// 3. Register and use experiment
await experiments.register({
  key: "prompt-test",
  variants: [
    { key: "control", weight: 50, config: { prompt: "Be helpful." } },
    { key: "treatment", weight: 50, config: { prompt: "Be concise." } },
  ],
});

const result = await experiments.getVariant("prompt-test", "user-123");

// 4. Shutdown on exit
process.on("SIGTERM", async () => {
  await tracing.shutdown();
  await experiments.shutdown();
});

Environment variables for tracing.init():

export GATEWAY_HOST="https://withgateway.ai"
export GATEWAY_PUBLIC_KEY="pk-lf-..."
export GATEWAY_SECRET_KEY="sk-lf-..."

Option 2: Explicit Configuration

import { tracing, experiments } from "@withgateway/sdk";

// Initialize with explicit config
tracing.init({
  host: "https://withgateway.ai",
  publicKey: "pk-lf-...",
  secretKey: "sk-lf-...",
  serviceName: "my-agent",
  environment: "production",
  debug: true, // Enable console logging
});

experiments.init({
  apiKey: "pk-lf-...",
  secretKey: "sk-lf-...",
  baseUrl: "https://withgateway.ai",
});

Option 3: Manual OpenTelemetry Setup (Advanced)

If you need full control over OpenTelemetry configuration:

import { experiments, ExperimentSpanProcessor } from "@withgateway/sdk";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

const provider = new NodeTracerProvider({
  spanProcessors: [
    new ExperimentSpanProcessor(), // Add BEFORE exporter
    new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: `${process.env.GATEWAY_HOST}/api/public/otel/v1/traces`,
        headers: {
          Authorization: `Basic ${Buffer.from(
            `${process.env.GATEWAY_PUBLIC_KEY}:${process.env.GATEWAY_SECRET_KEY}`,
          ).toString("base64")}`,
        },
      }),
    ),
  ],
});
provider.register();

Spans created after getVariant() are automatically tagged with:

• gateway.experiment.id — experiment key
• gateway.variant.key — assigned variant
• gateway.experiment.user_id — user ID

API Reference

Tracing API

import { tracing } from "@withgateway/sdk";

tracing.init(config?)           // Initialize OTel + Gateway exporter (call once)
tracing.flush()                 // Force flush pending spans
tracing.shutdown()              // Flush + shutdown (call on exit)
tracing.isInitialized()         // Check if initialized
tracing.isGatewayConfigured()   // Check if env vars are set

Config options:

tracing.init({
  host: string,              // Gateway server URL (or GATEWAY_HOST env var)
  publicKey: string,         // Auth key (or GATEWAY_PUBLIC_KEY env var)
  secretKey: string,         // Secret key (or GATEWAY_SECRET_KEY env var)
  serviceName?: string,      // Service name for traces (default: "gatewaysdk-tracing")
  environment?: string,      // Environment tag (e.g., "production")
  version?: string,          // Version tag (e.g., "1.2.3")
  debug?: boolean,           // Enable console logging (default: false)
  enableExperiments?: boolean, // Add ExperimentSpanProcessor (default: true)
});

Experiments API

import { experiments } from "@withgateway/sdk";

experiments.init(config)              // Initialize (call once)
experiments.register(input)           // Register an experiment
experiments.getVariant(key, userId)   // Get variant for user
experiments.fetch(keys?)              // Fetch experiments from backend
experiments.clearExperimentContext()  // Clear context (call after each request)
experiments.shutdown()                // Cleanup (call on exit)

Scores & Success Signals

Record evaluation scores and ROI-shaped success signals against a project. A success signal is a score tagged with gateway.success.signal, which the platform's Success Metrics feature aggregates into a metric's headline count / pass-rate (the HOOK metric source). The score shape is identical to the Python SDK's run.success(), so signals from either SDK aggregate the same.

import { ScoresClient, success } from "@withgateway/sdk/scores";

// Env-based (GATEWAY_HOST / GATEWAY_PUBLIC_KEY / GATEWAY_SECRET_KEY):
await success("resolved_ticket", true, {
  agentId: "support-triage",
  sessionId,
});
await success("handle_time_s", 42, { agentId: "support-triage", sessionId });

// Explicit keys:
const scores = new ScoresClient({ host, publicKey, secretKey });
await scores.success("resolved_ticket", true, {
  agentId: "support-triage",
  sessionId,
});
await scores.score("rubric", 0.9, { sessionId }); // a plain (non-signal) score

A boolean value is recorded as a 1/0 BOOLEAN score (pass/fail); a number is a NUMERIC score. metricName defaults to the signal name. Provide exactly one target — sessionId, traceId (+ optional observationId), or datasetRunId; when none is given the active OpenTelemetry trace id is used, and if there is none the call is a fail-soft no-op. Every score/success call is fail-soft: a bad key or transport failure warns and resolves, never throwing into your run. (postScore is the raw variant that DOES throw.)

Under tracing.initIsolated(...), the returned handle carries a scores client bound to that provider's project keys:

const handle = tracing.initIsolated({
  host,
  publicKey,
  secretKey,
  agent: "support-agent",
});
await handle.scores.success("resolved_ticket", true, {
  agentId: "support-agent",
  sessionId,
});

Security Wrapper

Use the security wrapper when you want Gateway to control tool visibility, authorization, credential bindings, and evidence while your agent framework keeps owning the native tool objects.

import { gateway } from "@withgateway/sdk/security";
import { streamText, tool } from "ai";

const gw = gateway.fromEnv({
  agent: {
    id: "support-triage",
    name: "Support Triage",
    version: "1.4.2",
  },
});

const tools = await gw.secureTools({
  toolsetId: "zendesk",
  provider: "zendesk",
  tools: {
    searchTickets: tool({
      description: "Search Zendesk tickets",
      inputSchema: searchSchema,
      execute: async (input) => zendesk.search(input),
    }),
    deleteTicket: tool({
      description: "Delete a Zendesk ticket",
      inputSchema: deleteSchema,
      execute: async (input) => zendesk.delete(input),
    }),
  },
  context: {
    userId: currentUser.id,
    teamId: "support",
    environment: "production",
    traceId,
    sessionId,
  },
});

await streamText({ model, prompt, tools });

secureTools calls /api/public/security/toolsets/resolve, hides blocked tools in filter/enforce, and calls /api/public/security/tool-calls/authorize immediately before execution. It also stamps Gateway security attributes on the active OpenTelemetry span when tracing is installed.

Class-Based API (Advanced)

import { ExperimentClient } from "@withgateway/sdk";

const client = new ExperimentClient({
  apiUrl: "https://withgateway.ai",
  apiKey: "pk-lf-...",
  secretKey: "sk-lf-...",
  projectId: "...",
});

Worlds & Live Sessions

A world is a versioned environment the platform hosts — tasks, tools, seed data and a grader, addressed as slug@ref. A session boots one task's container and holds it warm while your agent, running wherever it runs, calls its tools.

The distinction that matters: the agent is local, the world is not. Every tool call executes inside the platform's container, so you are testing against the environment as it actually is rather than a copy unwrapped on your own disk.

import { runSessions } from "@withgateway/sdk";

const report = await runSessions({
  world: "revenue-ops-sim@main",
  concurrency: 4, // one warm container per task
  failUnder: 0.7, // report.passed reads the mean
  agent: async ({ toolkit, instruction }) => {
    // The world's own tool schemas and callable implementations — nothing
    // to hand-write, nothing to keep in sync.
    await myAgent(toolkit.schemas, toolkit.impls, instruction);
  },
});

console.log(report.mean, report.passed);
process.exit(report.passed ? 0 : 1);

runSessions resolves the world once, so every task pins the same version and a push mid-run cannot split results across two worlds. A task that throws scores 0 and records why; the rest keep their results. Containers are handed back warm, so the next run starts hot.

Driving a single session directly, when you want the control:

import { openSession } from "@withgateway/sdk";

const session = await openSession(
  "revenue-ops-sim",
  "revops-mara-dedupe-contacts",
);
await session.ready(); // wait for the container
await session.call("crm_search", { entity: "deals" });
const { reward, rewards } = await session.grade(); // the task's own grader
await session.close({ keepWarm: true });

Sessions trace themselves from the platform — one trace per episode, one span per tool call — so an episode is readable even if your process dies mid-run.

Request timeout. Since 0.2.9 every platform request the SDK makes — sessions, the worlds helpers and the CLI — is abandoned after 120 s without an answer and fails with one line naming the cause (connection refused, DNS lookup failed, timed out) instead of waiting forever. Set GATEWAY_TIMEOUT_MS to a positive whole number of milliseconds to change it (GATEWAY_TIMEOUT_MS=300000 for five minutes); any other value is refused. Sandbox requests wait 290 s, under the 300 s Node's fetch allows; an exec waits on its command in 40 s requests, however long the command runs.

Full worked example: examples/world-suite.ts is a complete local production harness — a real tool-calling agent loop, parallel execution, a pass gate, a JSON artifact and a CI exit code. The Python twin is gatewaysdk/examples/worlds/world_suite.py.

The gateway CLI

The package ships the gateway command — the full TypeScript port of the Python CLI, group for group, with the same command names, positionals and flags (npm i -g @withgateway/sdk, or npx -p @withgateway/sdk gateway …).

The package bundles the schema runtime (the SDK's stdlib-only Python gatewaysdk/environment directory, the same files the platform executes). With any python3 >= 3.12 on the machine the CLI runs it locally — worlds serve, connector capture, schema check|compile|describe, data import on a directory — with no pip and no login. Without one, those commands send the world's files to POST /api/public/worlds/runtime and the platform runs the same code. GATEWAY_RUNTIME=local|platform forces a side. The interpreter is GATEWAY_PYTHON when set, else the first of python3.13, python3.12, python3, python, then uv python find that is new enough — a Mac whose python3 is the system 3.9 gets one line naming GATEWAY_PYTHON (GATEWAY_PYTHON="$(brew --prefix [email protected])/bin/python3.12" or GATEWAY_PYTHON="$(uv python find 3.12)"). Only bench eval|install|harvest-harbor and the Python agent scaffolds (algorithm, deploy, init) stay Python-SDK-only.

gateway auth login --host https://withgateway.ai --public-key pk-lf-… --api-key sk-lf-…
gateway --help                 # every group
gateway bench --help           # every command in a group
gateway bench push --help      # a command's positionals and flags

Credentials resolve in this order: --host (and --api-key on secret, agent, run, dataset upload, release gate, traces import and worlds serve), then GATEWAY_HOST / GATEWAY_PUBLIC_KEY / GATEWAY_SECRET_KEY (GATEWAY_API_KEY is accepted for the secret key), then ~/.gateway/config.json written by gateway auth login (GATEWAY_CONFIG_DIR moves it). auth login needs all three of --host, --public-key and --api-key, each of which may come from its environment variable instead. GATEWAY_ENV_FILE=path loads a KEY=value file into the environment first, in the form people write it — KEY = value, quoted values, export KEY=value and # comments are all accepted, so the file need not be shell-sourceable; a variable already in the shell's environment wins. Commands print JSON where the Python CLI does and exit non-zero with the reason on stderr. A request with no answer in 120 s ends the command the same way; GATEWAY_TIMEOUT_MS changes that budget.

| Group | Commands | | --- | --- | | auth | login, logout, status | | secret | set, list, delete | | agent | list | | run | start, verify, list | | dataset | upload FILE --name … | | bench | push, pull, checkout, fork, diff, propose, status, log, branch, tag, dispatch, run, run-configs, secrets, validate, resolve, tasks, refs, proposals | | worlds session | open, status, list, state, reset, grade, browse, seed, export, pin, close, reopen, task save | | worlds sandbox | open, status, list, read, write, exec, check, commit [--rebase], import, close [--force] — edit a world server-side; nothing lands on this disk | | worlds describe | what a world holds and how each field is stored — entities, keys, relationships, row counts, the [redact] policy per field — for a directory or a platform slug | | worlds schema | templates, init, compile, check, describe (the schema runtime runs on the platform) | | worlds connector | validate, capture, conform, ingest, publish, list, pull, clone, skill | | worlds data | import, check, extract, status, batches, publish, show, query — rows into a world (streamed, chunked, resumable for a slug), out of vendor exports, the batch ledger, and rows back out by predicate (--where email=… --resolve finds a hashed identifier from its plaintext) | | worlds skill | [NAME] [--list] [--install DIR] [--force] — the packaged agent skills; --install DIR alone lands all of them | | worlds run / ci | a TypeScript agent module against hosted sessions | | environment | list, get, tasks, run, run-status, rollouts, samples, performance, link-task-set, create-task, agent-status, agent-trigger | | release | gate — exit 0 pass, 1 blocked, 2 untested |

Python-only, and said so by the command itself (exit 2 with the equivalent): bench eval (local engines → bench dispatch), bench install, bench harvest-harbor, bench push --build-wheel, deploy, init, algorithm list|new. worlds serve runs the bundled runtime with any python3 >= 3.12, and opens a hosted api session when there is none.

Rows files for worlds data import and worlds session seed are a JSON object {"<entity>": [row, …]}, JSONL of {"entity", "row"} lines, or bare rows with --entity; - reads stdin and a .gz path is gunzipped. worlds session export writes the session's call log as JSONL ({seq, kind, tool, args, result, error, completedAt} per line).

Rows in bulk: worlds data

The same commands and flags as the Python CLI. A platform slug streams the rows — never held whole — as gzipped JSONL chunks into a data batch the platform validates against the world's contract, applies onto the world's snapshot and turns into a data-only version; a local directory keeps the small inline import through the runtime bridge (at most 5000 rows / 4 MiB — larger inputs go to the platform world).

gateway worlds data import <slug|dir> <rows…|-> [--entity E] [--mode append|replace] [--merge upsert|fill]
    [--chunk-rows N] [--chunk-bytes SIZE] [--parallel N] [--resume] [--atomic|--per-chunk]
    [--publish now|later] [--wait|--no-wait] [--report PATH] [--dry-run] [-m MESSAGE]
gateway worlds data check <slug|dir> <rows…|-> [--entity E] [--mode …] [--merge …] [--report PATH]
gateway worlds data extract <dir|slug> <file|-> --shape tool-results|vendor-envelope|session-export
    --entity E [--out PATH|-] [--merge fill|upsert] [--no-redact]
gateway worlds data status <batchId> [--world SLUG]
gateway worlds data batches <slug> [--limit N]
gateway worlds data publish <slug> [-m MESSAGE]
gateway worlds data show <slug|dir> [--entity E]
gateway worlds data query <slug|dir> <entity> [--where FIELD=VALUE ...] [--resolve] [--limit N] [--offset N] [--json]

• import cuts chunks on row boundaries at --chunk-bytes (64MiB before gzip by default) or --chunk-rows, uploads --parallel (4) at a time to the presigned URLs the platform hands out, keeps the manifest at .gateway/imports/<batchId>.json after every chunk (the same schema as the Python CLI, so either CLI can --resume the other's upload), completes the batch and follows the job: one line per phase on a TTY, then the settled batch as JSON. Exit 0 applied or published, 2 refused (--report writes every refusal as JSONL), 1 error. --no-wait prints {batchId, state}; --publish later leaves the batch for data publish. --merge decides the same primary key inside one batch (upsert: last wins; fill: first wins, later rows add missing fields); --per-chunk applies the good chunks and reports the bad ones.
• check (and import --dry-run) is the dry run: a slug runs a dry-run batch on the platform, a directory the runtime bridge's import over a scratch copy; nothing is written. Exit 2 when any row is refused.
• extract runs here, natively, on files of any size: rows out of a tool-results dump (raw_output.results[]), the vendor's own responses ([api].envelope of connector.toml) or a session export (result.<envelope>[] of successful calls), with null and empty fields dropped, undeclared fields stripped and counted, rows deduped by primary key and the world's [redact] policy applied (hash fields become sha256:<hex> over <connector slug>:<value>, drop fields go, round fields round) — byte for byte what the Python CLI writes. The summary {"read", "rows", "unknownFields", "redacted"} goes to stderr.
• query reads one entity's rows back by predicate: --where field=value, repeated for more fields (all must match) or the same field again (any of). A value matches the store as given — a digest you already hold works — and with --resolve the values are plaintext the runtime maps through the world's [redact] policy first, so --where [email protected] --resolve finds the row whose stored email is the slug-salted sha256; a [drop] field resolves to nothing and matches no row, which the resolved <field> -> … lines on stderr make plain. One row per line on stdout (compact JSON, pipe it to jq), then <n> of <total> (offset <o>) on stderr; --json prints the whole answer as one document. A directory is queried by the schema runtime, a slug by GET /api/public/worlds/{slug}/data?entity=&where=&resolve= (its head; no @ref). The library twin is queryRows(slug, entity, {where, resolve, limit, offset}) from @withgateway/sdk/worlds.

Chunks and rows are serialised exactly as the Python CLI serialises them (json.dumps(row, ensure_ascii=False, separators=(",", ":")), key order kept, 1.0 kept a float), so extract output and chunk content digests match across the two CLIs, and gzip digests too wherever the two zlibs deflate alike (Node 20 and 24 bundle Chromium's zlib, which deflates differently from CPython's stock zlib); the shared fixtures under gatewaysdk/fixtures/world-data/ are asserted by both test suites. GATEWAY_WORLD_DATA_POLL_MS shortens the status poll (CI), GATEWAY_PROGRESS=1|0 forces the phase lines on or off.

The packaged skills (gateway worlds skill --list) are copied from the Python package into skills/ by npm run build (scripts/copy-skills.mjs) and ship with the npm package.

For agents

The CLI is built to be driven by a coding agent (Claude Code, Cursor, Codex): every command has --help with its positionals and flags, output is JSON where a program would read it, and failures exit non-zero with the reason on stderr. The guides an agent needs ship inside the package as skills:

npm i -g @withgateway/sdk                       # or: npx -p @withgateway/sdk gateway …
gateway auth login --host https://withgateway.ai --public-key pk-lf-… --api-key sk-lf-…
gateway worlds skill --list                     # the packaged guides
gateway worlds skill worlds-getting-started     # print one: auth → pick a path → contract → data → push → session → run
gateway worlds skill --install .                # land every guide (and its agent) in this repository's .claude/

| Skill | Use it when | | --- | --- | | worlds-getting-started | building a world or environment from scratch: templates, workspace connectors, vendor captures, custom contracts; publishing, sessions, running, iterating | | world-data-ingestion | putting rows into a world from any source: rows-file contract, worlds data import\|check\|extract, session seeding and export, refusals | | connector-schema-authoring | writing a connector schema world (gateway-world/1) by hand or from a vendor spec | | benchmarks-getting-started | building, dry-running, pushing and running a benchmark: a world with [[tasks]] or a harbor / verifiers container benchmark | | hooks-getting-started | rules around an agent's tool calls from one gateway-hooks.toml: the alias door, redaction, per-task overrides, gateway hooks check |

--install DIR writes the skills under DIR/.claude/skills/ (and the connector-template-author agent under DIR/.claude/agents/), so the agent reads them the way it reads any project skill; --force overwrites a copy that drifted. Every CLI step in the guides names its MCP twin, so an agent that has the MCP server attached can do the same work without a shell.

Cross-Language Compatibility

This SDK produces identical assignments to the Python SDK (gatewaysdk) for the same (experimentKey, userId) pair. You can run experiments across services in different languages.

Documentation

• Full A/B Experiments Guide
• TypeScript SDK Deep Dive

License

MIT