@render-lab/tasks-langsmith

⚠️ Experimental: proof of concept. This package is part of the Render Tasks POC and is published for testing only. It is not fully tested or production ready. Task names, inputs, outputs, and behavior can change or break in any release. Pin exact versions and expect breaking changes.

Durable LangSmith dataset, experiment, comparison, and human-review tasks for Render Workflows — plus a registration-free adapter for signed LangSmith Engine webhooks.

Why a hybrid SDK + REST client

This pack uses a single injectable LangSmithPort. Its default implementation is a hybrid:

• The langsmith SDK backs the stable-resource operations it exposes cleanly: datasets, examples, tracing-project lookup and creation, project stats, and feedback.
• Direct REST backs the surfaces the JavaScript client does not expose cleanly: bulk run ingestion (/api/v1/runs/batch), experiment close (PATCH /api/v1/sessions/{id}), comparative experiments (/api/v1/datasets/comparative), and annotation-queue lifecycle.

Either way, vendor retries are off. The SDK batches trace uploads but exposes no request-retry knob, and the REST calls use raw fetch, so the durable task retry policy is the single source of truth — every attempt is one observable run. No langsmith type ever crosses a task boundary; the port maps SDK objects and REST bodies to plain JSON DTOs and back.

A LangSmith experiment is a tracing project bound to a reference dataset. This pack uses "experiment" throughout; internally it maps to LangSmith projects/sessions.

import {
  ensureDataset,
  upsertExamples,
  listExamples,
  ensureExperiment,
  logExperimentRuns,
  closeExperiment,
  getExperimentStats,
  gateExperiment,
  addFeedback,
  createComparison,
  startReview,
  awaitReview,
} from "@render-lab/tasks-langsmith";

@renderinc/sdk is a peer dependency pinned to 1.0.0; install it alongside this pack in your workflow service. @render-lab/triggers is an optional peer dependency needed only for the ./webhooks subpath.

Tasks

| Task | Input | Output | Retry | | --- | --- | --- | --- | | langsmith.ensureDataset | EnsureDatasetInput | DatasetDTO | general (3× backoff) | | langsmith.upsertExamples | UpsertExamplesInput | UpsertExamplesResult | general (3× backoff) | | langsmith.listExamples | ListExamplesInput | ListExamplesResult | general (3× backoff) | | langsmith.ensureExperiment | EnsureExperimentInput | ExperimentDTO | general (3× backoff) | | langsmith.logExperimentRuns | LogExperimentRunsInput | LogExperimentRunsResult | general (3× backoff) | | langsmith.closeExperiment | CloseExperimentInput | CloseExperimentResult | general (3× backoff) | | langsmith.getExperimentStats | GetExperimentStatsInput | ExperimentStatsDTO | general (3× backoff) | | langsmith.gateExperiment | GateExperimentInput | GateExperimentResult | none (0 retries) | | langsmith.addFeedback | AddFeedbackInput | AddFeedbackResult | general (3× backoff) | | langsmith.createComparison | CreateComparisonInput | ComparisonDTO | general (3× backoff) | | langsmith.startReview | StartReviewInput | StartReviewResult | general (3× backoff) | | langsmith.awaitReview | AwaitReviewInput | ReviewProgressDTO | review (poll every 30s up to 7 days) |

Stable identities and idempotency

Every write is replay-safe because it carries a stable, caller-provided identity or is a convergent update — a retried attempt overwrites the same records rather than duplicating them:

• upsertExamples requires a stable non-blank id on each example (no duplicates); it partitions into create-vs-update by which IDs already exist.
• logExperimentRuns requires stable id and traceId on each run (no duplicate IDs, and endTime may not precede startTime).
• ensureDataset / ensureExperiment find-or-create by exact name. ensureExperiment rejects an existing experiment bound to a different reference dataset.
• closeExperiment is a convergent end-time update; re-closing converges.
• addFeedback uses a caller feedbackId, so (feedbackId, runId, key) is a convergent update.
• createComparison uses a caller comparisonId and needs at least two unique experiment IDs.
• startReview ensures one annotation queue by exact name, then assigns a unique, order-preserving set of run IDs. A run already in the queue (HTTP 409) normalizes to success.

Gate semantics

gateExperiment reads one getExperimentStats snapshot and evaluates release thresholds against it — it never polls or mutates vendor state. It returns { passed: true, stats } only when every threshold is met; otherwise it throws an error beginning with Release gate failed: and including the JSON-serialized GateViolation[]. It uses zero retries: an unmet quality threshold is a terminal release decision, not a transient vendor error.

Threshold mapping (GateThresholds):

• minimumScores[key] → stats.feedback[key].mean (operator >=). A missing mean yields a violation with actual: null.
• maximumErrorRate → stats.errorRate (operator <=).
• minimumReviewed → the smallest count among the requested feedback keys (operator >=). This requires minimumScores — those keys are the only source of a reviewed count — so minimumReviewed without minimumScores is rejected.

Review queue lifecycle

startReview ensures the annotation queue and assigns runs; awaitReview is a durable human-review wait. SDK 1.0 has no native sleep, so it polls by throwing: it reads review progress once per attempt and throws a polling message while any run is still missing a required feedback key, so the fixed-interval review retry (every 30s for up to seven days) re-runs it. It returns only once every requested feedback key exists for every requested run.

// 1. Seed the dataset (stable example IDs).
const dataset = await ensureDataset({ name: "qa-eval-set" });
await upsertExamples({
  datasetId: dataset.id,
  examples: [{ id: "ex-1", inputs: { q: "What is Render?" }, outputs: { a: "A cloud platform." } }],
});

// 2. Open an experiment and log runs under stable run/trace IDs.
const experiment = await ensureExperiment({ name: "release-2026-07-13", referenceDatasetId: dataset.id });
await logExperimentRuns({
  experimentId: experiment.id,
  runs: [{
    id: "run-1", traceId: "run-1", name: "answer", runType: "chain",
    inputs: { q: "What is Render?" }, outputs: { a: "A cloud platform." },
    startTime: "2026-07-13T00:00:00Z", endTime: "2026-07-13T00:00:01Z",
  }],
});

// 3. Close, then gate on quality.
await closeExperiment({ experimentId: experiment.id });
await gateExperiment({
  experimentId: experiment.id,
  thresholds: { minimumScores: { correctness: 0.8 }, maximumErrorRate: 0.05 },
});

// 4. Optional human review, durably awaited.
const review = await startReview({ queueName: "release-review", runIds: ["run-1"], feedbackKeys: ["correctness"] });
await awaitReview({ queueId: review.queueId, runIds: review.runIds, feedbackKeys: review.feedbackKeys });

Payload limits and the 4 MB cap

Every bulk task accepts 1 to 1,000 objects per call (upsertExamples, logExperimentRuns, startReview run IDs); listExamples bounds limit to 1..1000. Evaluation scores are constrained to [0, 1]. The port additionally rejects any serialized result at or above 4 MB — the Render Workflows task-boundary cap — with guidance to lower the limit or narrow the request. Lower your page limit if listExamples trips this.

Environment

Credentials are read lazily on first use, never at import — pulling in this pack for one task does not require another task's secret.

| Variable | Required | Purpose | | --- | --- | --- | | LANGSMITH_API_KEY | Yes | Authenticates LangSmith task calls (SDK and REST) | | LANGSMITH_ENDPOINT | No | Overrides https://api.smith.langchain.com | | LANGSMITH_WORKSPACE_ID | No | Selects a workspace when the key has access to multiple workspaces | | LANGSMITH_ENGINE_WEBHOOK_SECRET | For ./webhooks only | Verifies signed LangSmith Engine deliveries |

Webhooks (@render-lab/tasks-langsmith/webhooks)

The ./webhooks subpath is registration-free (ADR-0017): it imports only node:crypto, @render-lab/triggers types, and its own local types. It never imports the package root, calls task(), constructs a LangSmith client, or reads a credential at import time.

Scope: it verifies only signed LangSmith Engine events — deliveries carrying an X-LangSmith-Signature header. It does not verify unsigned LangSmith automation webhooks; route those through the generic trigger server instead.

import { langSmithEngineAdapter } from "@render-lab/tasks-langsmith/webhooks";

const adapter = langSmithEngineAdapter({
  // secret defaults to LANGSMITH_ENGINE_WEBHOOK_SECRET, read inside verify()
  onEvent: (event) => {
    if (event.type === "issue.created") {
      return { task: "engine.handleIssue", args: [event] };
    }
    return null; // ignore other event types
  },
});

Delivery semantics:

• Signature: X-LangSmith-Signature = sha256= + HMAC-SHA256 over the exact raw body, compared in constant time after an equal-length check.
• Retries: LangSmith makes up to four total delivery attempts.
• Deduplication: the event id is stable across retries — dedupe on it caller-side.
• Grouping: request_id groups coalesced events.
• Timeout: respond within 20 seconds.
• Event types: the three documented types are issue.created, issue.trace.added, and issue.agent_run.failed. Unknown types pass through to onEvent as an open shape so you can ignore or route future events — they are not claimed to be verified schemas.

Testing

pnpm -C packages/tasks-langsmith build
pnpm -C packages/tasks-langsmith test        # Tier 1: hermetic, no secrets/network
pnpm -C packages/tasks-langsmith typecheck

Tier 1 tests inject a fake LangSmithPort at the port boundary. The Tier 2 live suite (*.live.test.ts) is guarded by describe.skipIf(!process.env.RUN_LIVE) and hits real LangSmith — run it with RUN_LIVE=1 and a real LANGSMITH_API_KEY; it never gates pnpm -r test.