@langchain/sandbox-standard-tests

Shared integration test suites for deepagents sandbox providers. Run a single function call and get comprehensive coverage of the SandboxBackendProtocol — lifecycle management, command execution, file I/O, search, and more.

The package is framework-agnostic — it works with any test runner that provides describe, it, expect, beforeAll, and afterAll. A first-class Vitest sub-export is included for convenience.

Installation

npm install @langchain/sandbox-standard-tests

Quick start

With Vitest (recommended)

Import from @langchain/sandbox-standard-tests/vitest and the Vitest primitives are injected automatically:

import { sandboxStandardTests } from "@langchain/sandbox-standard-tests/vitest";
import { MySandbox } from "./sandbox.js";

sandboxStandardTests({
  name: "MySandbox",
  skip: !process.env.MY_SANDBOX_TOKEN,
  timeout: 120_000,
  createSandbox: (opts) => MySandbox.create({ ...opts }),
  closeSandbox: (sb) => sb.close(),
  resolvePath: (name) => `/tmp/${name}`,
});

With any test runner

Import from the root entry point and pass your runner's primitives via the runner config property:

import { sandboxStandardTests } from "@langchain/sandbox-standard-tests";
import { describe, it, expect, beforeAll, afterAll } from "bun:test";
import { MySandbox } from "./sandbox.js";

sandboxStandardTests({
  name: "MySandbox",
  runner: { describe, it, expect, beforeAll, afterAll },
  createSandbox: (opts) => MySandbox.create({ ...opts }),
  closeSandbox: (sb) => sb.close(),
  resolvePath: (name) => `/tmp/${name}`,
});

Run with your test runner of choice:

npx vitest run sandbox.int.test.ts

That single sandboxStandardTests() call registers 11 describe blocks covering every method on the sandbox protocol.

Configuration

sandboxStandardTests accepts a StandardTestsConfig<T> object:

| Option | Type | Required | Description | | ---------------------------- | ---------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | string | yes | Display name shown in the test runner (e.g. "ModalSandbox"). | | runner | TestRunner | yes* | Test-runner primitives (describe, it, expect, beforeAll, afterAll). *Optional when importing from /vitest. | | createSandbox | (opts?) => Promise<T> | yes | Factory that creates and returns a running sandbox. Receives an optional { initialFiles } map. | | resolvePath | (relativePath: string) => string | yes | Converts a relative filename (e.g. "test-file.txt") to the provider-specific absolute path (e.g. "/tmp/test-file.txt" or "/home/app/test-file.txt"). | | closeSandbox | (sandbox: T) => Promise<void> | no | Teardown function. If omitted the "close" lifecycle test is skipped. | | createUninitializedSandbox | () => T | no | Factory for a sandbox that has not been started yet. Enables the two-step initialization test. | | skip | boolean | no | Skip the entire suite (useful when credentials are missing). | | sequential | boolean | no | Run tests sequentially instead of in parallel (useful to avoid provider concurrency limits). | | timeout | number | no | Per-test timeout in ms. Defaults to 120_000 (2 min). |

TestRunner

The runner object must provide these five primitives from your test framework:

interface TestRunner {
  describe: SuiteFn;
  it: TestFn;
  expect: ExpectFn;
  beforeAll: HookFn;
  afterAll: HookFn;
}

The describe and it functions may optionally expose .skip, .skipIf(condition), and .sequential modifiers. When a modifier is not available the suite gracefully degrades (e.g. describe.sequential falls back to describe).

SandboxInstance

Your sandbox class must implement the SandboxInstance interface, which extends SandboxBackendProtocol from deepagents:

interface SandboxInstance extends SandboxBackendProtocol {
  readonly isRunning: boolean;
  uploadFiles(
    files: Array<[string, Uint8Array]>,
  ): MaybePromise<FileUploadResponse[]>;
  downloadFiles(paths: string[]): MaybePromise<FileDownloadResponse[]>;
  initialize?(): Promise<void>;
}

The key difference from the base protocol is that uploadFiles and downloadFiles are required (they are optional in SandboxBackendProtocol).

What gets tested

| Suite | What it covers | | --------------------- | -------------------------------------------------------------------------------------------- | | Lifecycle | create, isRunning, close, two-step initialize | | Command execution | echo, exit codes, multiline output, stderr, env vars, non-existent commands | | File operations | uploadFiles, downloadFiles, round-trip integrity | | write() | New files, parent directory creation, overwrite, special characters, unicode, long content | | read() | Basic read, non-existent path, offset, limit, offset + limit, unicode, chunked reads | | edit() | Single/multi occurrence, replaceAll, not-found handling, special chars, multiline, unicode | | lsInfo() | Directory listing, empty dirs, hidden files, large directories, absolute paths | | grepRaw() | Pattern search, glob filters, case sensitivity, nested directories, unicode | | globInfo() | Wildcards, recursive patterns, extension filters, character classes, deeply nested | | Initial files | Basic seeding, nested paths, empty files | | Integration | End-to-end write → read → edit workflows, complex directory operations, error handling |

Sandbox reuse strategy

To avoid spinning up too many sandbox instances (which can hit provider concurrency limits), the test suite uses a single shared sandbox for the majority of tests. Only two kinds of tests create temporary instances:

• Lifecycle tests that verify close and two-step initialization
• Initial files tests that require a fresh sandbox with pre-seeded content

These temporary sandboxes are torn down immediately, so the concurrent sandbox count never exceeds 2.

Retry helper

The package exports a withRetry utility for working around transient sandbox creation failures (e.g. provider concurrency limits):

import { withRetry } from "@langchain/sandbox-standard-tests/vitest";

const sandbox = await withRetry(
  () => MySandbox.create({ memoryMb: 512 }),
  5, // max attempts (default: 5)
  15_000, // delay between attempts in ms (default: 15 000)
);

Real-world examples

Remote provider (Modal)

import {
  sandboxStandardTests,
  withRetry,
} from "@langchain/sandbox-standard-tests/vitest";
import { ModalSandbox } from "./sandbox.js";

const hasCredentials = !!(
  process.env.MODAL_TOKEN_ID && process.env.MODAL_TOKEN_SECRET
);

sandboxStandardTests({
  name: "ModalSandbox",
  skip: !hasCredentials,
  timeout: 180_000,
  createSandbox: (opts) =>
    ModalSandbox.create({ imageName: "alpine:3.21", ...opts }),
  createUninitializedSandbox: () =>
    new ModalSandbox({ imageName: "alpine:3.21" }),
  closeSandbox: (sb) => sb.close(),
  resolvePath: (name) => `/tmp/${name}`,
});

Sequential execution (Deno Deploy)

import { sandboxStandardTests } from "@langchain/sandbox-standard-tests/vitest";
import { DenoSandbox } from "./sandbox.js";

sandboxStandardTests({
  name: "DenoSandbox",
  skip: !process.env.DENO_DEPLOY_TOKEN,
  sequential: true,
  timeout: 120_000,
  createSandbox: (opts) => DenoSandbox.create({ memoryMb: 768, ...opts }),
  createUninitializedSandbox: () => new DenoSandbox({ memoryMb: 768 }),
  closeSandbox: (sb) => sb.close(),
  resolvePath: (name) => `/home/app/${name}`,
});

Local provider (Node VFS)

import { sandboxStandardTests } from "@langchain/sandbox-standard-tests/vitest";
import { VfsSandbox } from "./sandbox.js";

sandboxStandardTests({
  name: "VfsSandbox",
  skip: process.platform === "win32",
  timeout: 30_000,
  createSandbox: (opts) => VfsSandbox.create(opts),
  closeSandbox: (sb) => sb.stop(),
  resolvePath: (name) => name,
});

Custom runner (Bun)

import { sandboxStandardTests } from "@langchain/sandbox-standard-tests";
import { describe, it, expect, beforeAll, afterAll } from "bun:test";
import { MySandbox } from "./sandbox.js";

sandboxStandardTests({
  name: "MySandbox",
  runner: { describe, it, expect, beforeAll, afterAll },
  createSandbox: (opts) => MySandbox.create(opts),
  closeSandbox: (sb) => sb.close(),
  resolvePath: (name) => `/tmp/${name}`,
});

Adding provider-specific tests

After calling sandboxStandardTests, you can add provider-specific tests in the same file using standard Vitest describe / it blocks:

sandboxStandardTests({
  /* ... */
});

describe("MySandbox Provider-Specific Tests", () => {
  it("should support custom image types", async () => {
    const sb = await MySandbox.create({ image: "python:3.12" });
    const result = await sb.execute("python --version");
    expect(result.exitCode).toBe(0);
    await sb.close();
  });
});

License

MIT