leewayllm

Typed client SDK for a LeewayLLM gateway — the LLM cost-optimization gateway whose whole value proposition is:

"We don't change your model. We remove the context your agent should not have sent."

The gateway speaks the standard chat-completions wire format and forwards your requests to your model provider unchanged (same model, same sampling parameters). All savings come from removing wasted context and structuring requests for native prompt caching. This package is a zero-dependency client for that gateway.

• Chat completions, including streaming (async iterator)
• Per-request optimization metadata (tokens before/after, estimated savings, pipeline time) on every result
• Audit API: fetch the full per-request record (token counts + per-category metadata; no request content is ever stored)
• Stats, models and health endpoints
• Typed errors, automatic retries on 429/5xx, timeouts, AbortSignal support
• Works on Node.js ≥ 18 and any runtime with WHATWG fetch + streams; ships ESM + CJS + type declarations

Install

npm install leewayllm

Quickstart

import { Leeway } from "leewayllm";

const leeway = new Leeway({
  // baseUrl defaults to the hosted gateway, https://api.leewayai.app
  // (self-host: baseUrl: "http://localhost:8787")
  apiKey: process.env.LEEWAY_API_KEY,     // your lwllm_ account key
  mode: "balanced",                       // default optimization mode
  sessionId: "support-agent-42",          // enables cross-request cache tracking
});

const result = await leeway.chat.completions.create({
  model: "<your-model-id>",
  max_tokens: 512,
  messages: [
    { role: "system", content: "You are terse." },
    { role: "user", content: "Say hello in one word." },
  ],
});

console.log(result.choices[0].message.content);
console.log(result.leeway);
// {
//   requestId: "req_…", mode: "balanced", provider: "…",
//   estInputTokensBefore: 12113, estInputTokensAfter: 1330,
//   estSavedUsd: 0.027, pipelineMs: 4.1
// }

Streaming

const stream = await leeway.chat.completions.create({
  model: "<your-model-id>",
  stream: true,
  messages: [{ role: "user", content: "Write a haiku about gateways." }],
});

console.log(stream.leeway.requestId); // metadata available before the first chunk

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}

const usage = await stream.finalUsage(); // provider-billed usage from the final chunk

Gateway-down failover (your app never blocks on Leeway)

Your provider keys remain yours — so if the gateway is ever unreachable, the SDK can replay the same request directly at your provider with the key from your own environment (it is never sent to the gateway):

const leeway = new Leeway({
  apiKey: process.env.LEEWAY_API_KEY,
  fallback: {
    baseUrl: "https://api.openai.com",          // any OpenAI-compatible base
    apiKey: process.env.OPENAI_API_KEY!,        // YOUR key, stays client-side
  },
});

Failover triggers only on availability failures (network errors, timeouts, 502/503/504). Intentional gateway answers — auth, plan or rate-limit errors, provider errors passed through — never fail over. Requests served directly carry result.leeway.mode === "off(direct_fallback)" so you can count them.

Modes

| mode | guarantee | |---|---| | off | byte-faithful passthrough — the gateway sends exactly what you sent | | safe | data-lossless: duplicate tool definitions removed, machine JSON re-serialized compactly, requests structured for native prompt caching | | balanced | safe + lossy trimming of oversized tool results (big arrays, binary payloads, raw page dumps) — everything removed is dropped, not stored | | aggressive | balanced + stubbing of tool results in old turns (never the most recent turns, never your own text) |

Set a default on the client (mode) or per call:

await leeway.chat.completions.create(request, { mode: "off", sessionId: "other-session" });

Human-authored content is never modified in any mode, and if the optimization pipeline ever fails, the gateway sends your original request untouched (result.leeway.mode reads off(fallback:…)).

Audit: see how much was removed (metadata only — no stored content)

Records carry token counts and per-category metadata only. Your prompts, tool results and attachments are optimized in memory and forwarded to the provider — they are never written to disk or DB, so there is no content to fetch back.

const record = await leeway.audit.get(result.leeway.requestId!);
console.log(record.totals);   // est tokens before/after, actual usage, est saved $
console.log(record.actions);  // one entry per transformation, with token deltas

const recent = await leeway.audit.list({ limit: 20, mode: "balanced" });

Ops

await leeway.stats();        // totals, by-mode savings, pipeline p50/p95
await leeway.models.list();  // models known to the gateway
await leeway.health();       // { ok: true, version: "…" }

Errors

All failures throw LeewayError:

import { LeewayError } from "leewayllm";

try {
  await leeway.chat.completions.create(request);
} catch (err) {
  if (err instanceof LeewayError) {
    err.status;         // gateway HTTP status (0 = network)
    err.providerStatus; // upstream status when the gateway passed an error through
    err.providerBody;   // raw upstream error body
    err.requestId;
  }
}

Retries: 429/500/502/503/529 and network errors are retried automatically (default 2 attempts, exponential backoff) before a stream is consumed. Configure with maxRetries; timeoutMs bounds the time to response headers (streams are unbounded after that — pass your own signal to cap them).

Options

| option | default | description | |---|---|---| | baseUrl | — (required) | gateway URL | | apiKey | – | upstream key, sent as Authorization: Bearer (BYOK) | | mode | gateway default | default optimization mode | | sessionId | derived by the gateway | cache-prefix tracking key | | timeoutMs | 120000 | time allowed until response headers | | maxRetries | 2 | automatic retries on 429/5xx/network | | fetch | global fetch | custom fetch implementation | | defaultHeaders | {} | extra headers on every request |

Releasing (maintainers)

This package ships from the leewayllm-sdk repo via the publish-sdk GitHub Action — pushing a commit does not publish; pushing a v* tag does.

npm version patch          # bumps package.json, commits, creates the v* tag
git push --follow-tags     # pushes the commit + tag → triggers the publish

The workflow checks the tag matches package.json "version", builds (ESM + CJS

• types) and runs npm publish. Auth is Trusted Publishing (OIDC) — no stored token (trusted publisher on npmjs.com: GitHub Actions · org Leeway-AI · repo leewayllm-sdk · workflow publish-sdk.yml · env empty). Provenance is attached automatically. Dry-run it from the repo's Actions tab (workflow_dispatch) first.

License

MIT