@tensorcost/sdk

One-line wrapper SDK for TensorCost — fire-and-forget LLM observability for OpenAI, Anthropic, and AWS Bedrock clients in Node.js.

Install

npm install @tensorcost/sdk
# or
pnpm add @tensorcost/sdk

Requires Node.js 18+ (uses native fetch).

Use

import OpenAI from "openai";
import { wrap } from "@tensorcost/sdk";

const client = wrap(new OpenAI({ apiKey: process.env.OPENAI_API_KEY }));

// Use the client exactly as before — TensorCost observes every call
// and ships fire-and-forget observations to the platform.
const resp = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "hi" }],
});

Anthropic is just as simple:

import Anthropic from "@anthropic-ai/sdk";
import { wrap } from "@tensorcost/sdk";

const client = wrap(new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }));

await client.messages.create({
  model: "claude-3-5-sonnet-20241022",
  max_tokens: 256,
  messages: [{ role: "user", content: "hi" }],
});

Configuration

wrap() reads from the environment by default:

| Variable | Purpose | Required | |---|---|---| | TENSORCOST_API_KEY | Long-lived API key from the TensorCost console | Yes | | TENSORCOST_BASE_URL | Backend base URL | No (default https://api.tensorcost.com) | | TENSORCOST_TENANT_ID | Optional explicit tenant id | No | | TENSORCOST_PROXY_URL | Proxy base URL for applied mode | Required when appliedMode: true | | TENSORCOST_ENVIRONMENT | Environment tag stamped on every observation | No | | TENSORCOST_CONNECTION_ID | Provider-connection identifier | No |

Or pass explicitly:

const client = wrap(new OpenAI(...), {
  apiKey: "tc_live_xxx",
  baseUrl: "https://api.tensorcost.com",
});

Applied mode (Layer 2)

When you enable applied mode, the SDK routes inference requests through the TensorCost inference-proxy instead of directly to the AI provider. The proxy decides per-request whether to re-route to an alternate provider or pass through unchanged — the SDK doesn't make that call.

const client = wrap(new OpenAI({ apiKey: process.env.OPENAI_API_KEY }), {
  apiKey: "tc_live_xxx",
  appliedMode: true,
  proxyUrl: "https://proxy.tensorcost.com", // or set TENSORCOST_PROXY_URL
});

// Use normally — the SDK redirects the client to the proxy transparently.
// Observe-only telemetry still fires after every call.
const resp = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "hi" }],
});

wrap() throws TensorCostConfigError at wrap() time if appliedMode: true and neither proxyUrl nor TENSORCOST_PROXY_URL is set. AWS Bedrock is not supported in applied mode — SigV4 signing happens inside the AWS SDK before the proxy layer can intercept it; passing a Bedrock client with appliedMode: true throws TensorCostConfigError immediately. Observe-only mode works fine for Bedrock.

See docs/features/inference-proxy-applied-mode.md for the proxy-side design.

Applied-mode hardening (v0.4.0)

When appliedMode: true, you can configure retry policy, timeouts, lifecycle hooks, and the fail-open circuit breaker.

Retries

import { wrap, RetryConfig } from "@tensorcost/sdk";

const client = wrap(new OpenAI(...), {
  appliedMode: true,
  proxyUrl: "...",
  retry: { maxAttempts: 5, baseDelayMs: 500, maxDelayMs: 30_000 },
});

5xx responses, network errors, and 429 are retried with jittered exponential backoff. A Retry-After header on 429 is honoured. 4xx errors (except 429) are never retried.

Timeouts

const client = wrap(new OpenAI(...), {
  appliedMode: true,
  proxyUrl: "...",
  timeoutMs: 30_000, // 30 seconds; default 60_000
});

Surfaces as TensorCostTimeoutError (with timeoutKind: "total") when exceeded.

Lifecycle telemetry hooks

import { wrap, LifecycleEvent } from "@tensorcost/sdk";

const client = wrap(new OpenAI(...), {
  appliedMode: true,
  proxyUrl: "...",
  onLifecycleEvent(event: LifecycleEvent) {
    // event.kind is one of:
    //   "before_request" | "after_response" | "on_retry"
    //   | "on_error" | "on_fallback"
    console.log(event.kind, event.model, event.attemptNumber);
  },
});

Events contain only request metadata — never prompt content, response bodies, or credentials. Errors thrown by the callback are swallowed.

Typed errors

Every failure from the proxy path is a subclass of TensorCostError. Import the classes from @tensorcost/sdk:

import {
  TensorCostError,
  TensorCostNetworkError,
  TensorCostTimeoutError,
  TensorCostProxyError,
  TensorCostQuotaError,
  TensorCostProviderError,
} from "@tensorcost/sdk";

try {
  await client.chat.completions.create({ model: "gpt-4o-mini", messages: [] });
} catch (err) {
  if (err instanceof TensorCostQuotaError) {
    console.log("rate limited; retry after", err.retryAfterMs, "ms");
  } else if (err instanceof TensorCostError) {
    console.log("tensorcost error", err.status, err.attempt);
  }
}

Circuit breaker

After 3 consecutive proxy 5xx responses the circuit opens and requests route directly to the provider (fail-open). The circuit closes after 5 consecutive probe successes.

Set failOpenEnabled: false if direct provider access is unavailable (e.g. in-VPC deployments) — proxy failures then surface as TensorCostProxyError instead of falling back.

const client = wrap(new OpenAI(...), {
  appliedMode: true,
  proxyUrl: "...",
  failOpenEnabled: false, // raise on proxy error instead of falling through
});

Fail-open guarantee

If TensorCost is slow, down, or misconfigured, your customer call STILL succeeds. The SDK swallows internal errors and surfaces them via console.warn. The one exception: a missing API key throws MissingConfigError at wrap() time — silently dropping every observation would be worse than a loud import-time failure.

What is captured

Only metadata — never prompt or completion content:

• Provider + model
• Operation (chat.completions, completions, messages)
• Request / response timestamps
• Input / output token counts (from the provider's usage field)
• Status (success / error) and error class/message on failure
• A correlation UUID

AWS Bedrock (observe-only)

import {
  BedrockRuntimeClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime";
import { wrap } from "@tensorcost/sdk";

// The only change: wrap() before passing the client to the rest of your code.
const client = wrap(
  new BedrockRuntimeClient({ region: process.env.AWS_REGION ?? "us-east-1" }),
);

// Everything else is unchanged.
const response = await client.send(
  new InvokeModelCommand({
    modelId: "anthropic.claude-3-haiku-20240307-v1:0",
    body: JSON.stringify({
      anthropic_version: "bedrock-2023-05-31",
      max_tokens: 64,
      messages: [{ role: "user", content: "What is 2 + 2?" }],
    }),
    contentType: "application/json",
    accept: "application/json",
  }),
);

The SDK registers a Smithy middleware on client.middlewareStack that intercepts every client.send() call and emits a fire-and-forget observation with the model id, token counts, latency, and status. Your code is otherwise unchanged.

Supported commands

| Command | Operation recorded | Token counts | |---|---|---| | InvokeModelCommand | bedrock.invoke_model | Parsed from response body (Anthropic-on-Bedrock and Nova/Titan shapes) | | ConverseCommand | bedrock.converse | Read from response.usage.inputTokens / outputTokens | | InvokeModelWithResponseStreamCommand | bedrock.invoke_model_stream | Not yet — placeholder observation with null tokens | | ConverseStreamCommand | bedrock.converse_stream | Not yet — placeholder observation with null tokens |

Per-chunk token accounting for streaming commands is a follow-up workstream. The placeholder observations record latency and success/error status, which is enough for alerting and anomaly detection. Token cost attribution will arrive once per-model pricing is wired in on the backend.

Applied mode

Applied mode ({ appliedMode: true }) is not supported for Bedrock. AWS Bedrock uses SigV4 request signing inside the AWS SDK before our middleware sees the request, so we cannot intercept and re-route it the way we do for OpenAI/Anthropic HTTP-Bearer clients. Calling wrap(bedrockClient, { appliedMode: true }) throws TensorCostConfigError immediately.

Installation

@aws-sdk/client-bedrock-runtime is an optional peer dependency — you only need it if you're using Bedrock. The SDK will not pull it in for OpenAI/Anthropic-only users.

npm install @aws-sdk/client-bedrock-runtime
# or
pnpm add @aws-sdk/client-bedrock-runtime

Currently supported

• OpenAI (openai >= 4.x) — chat.completions.create, completions.create
• Anthropic (@anthropic-ai/sdk >= 0.x) — messages.create
• AWS Bedrock (@aws-sdk/client-bedrock-runtime >= 3.x) — InvokeModelCommand, ConverseCommand (observe-only)

Coming soon

• Azure OpenAI, Vertex
• Streaming token accounting for Bedrock (InvokeModelWithResponseStreamCommand, ConverseStreamCommand)
• Tool / function calling (multi-call trace stitching)

License

Proprietary — TensorCost.