High performance client for Baseten.co - Node.js Bindings

This library provides a high-performance Node.js client for Baseten.co endpoints including embeddings, reranking, and classification. It was built for massive concurrent POST requests to any URL, also outside of baseten.co. The PerformanceClient is built on top of Rust (using napi-rs), reqwest and tokio and is MIT licensed.

Similar to the Python version, this client supports >1200 rps per client and was benchmarked in our blog.

Installation

npm install @basetenlabs/performance-client

Usage

Basic Setup

Since different endpoints require different clients, you'll typically need to create separate clients for embeddings and reranking deployments.

const { PerformanceClient, HttpClientWrapper } = require('@basetenlabs/performance-client');

const apiKey = process.env.BASETEN_API_KEY;
const embedBaseUrl = "https://model-yqv4yjjq.api.baseten.co/environments/production/sync";
const rerankBaseUrl = "https://model-abc123.api.baseten.co/environments/production/sync";

// Create separate clients for different endpoints
const embedClient = new PerformanceClient(embedBaseUrl, apiKey);
const rerankClient = new PerformanceClient(rerankBaseUrl, apiKey);

// Advanced setup with custom HTTP version and client wrapper
const httpWrapper = new HttpClientWrapper(2); // Use HTTP/2
const advancedClient = new PerformanceClient(baseUrl, apiKey, 2, httpWrapper);

Embeddings

const texts = ["Hello world", "Example text", "Another sample"];

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

try {
    const preference = new RequestProcessingPreference(
        8,         // maxConcurrentRequests
        2,         // batchSize
        30,        // timeoutS
        undefined, // maxCharsPerRequest
        undefined, // pinInitialEndpointOnce
        undefined, // hedgeDelay
        undefined, // totalTimeoutS
        undefined, // hedgeBudgetPct
        undefined, // retryBudgetPct
        undefined, // maxRetries
        undefined, // initialBackoffMs
        undefined, // cancelToken
        undefined, // primaryApiKeyOverride
        {"x-custom-header": "value"} // extraHeaders
    );
    const response = await embedClient.embed(
        texts,
        "text-embedding-3-small", // model
        null, null, null, // encoding_format, dimensions, user
        preference // preference parameter
    );

    console.log(`Model used: ${response.model}`);
    console.log(`Total tokens used: ${response.usage.total_tokens}`);
    console.log(`Total time: ${response.total_time.toFixed(4)}s`);

    if (response.individual_request_times) {
        response.individual_request_times.forEach((time, i) => {
            console.log(`  Time for batch ${i}: ${time.toFixed(4)}s`);
        });
    }

    response.data.forEach((embedding, i) => {
        console.log(`Embedding for text ${i} (original input index ${embedding.index}):`);
        console.log(`  First 3 dimensions: ${embedding.embedding.slice(0, 3)}`);
        console.log(`  Length: ${embedding.embedding.length}`);
    });
} catch (error) {
    console.error('Embedding failed:', error.message);
}

Reranking

const query = "What is the best framework?";
const documents = [
    "Machine learning is a subset of artificial intelligence",
    "JavaScript is a programming language",
    "Deep learning uses neural networks",
    "Python is popular for data science"
];

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

try {
    const preference = new RequestProcessingPreference(
        4,         // maxConcurrentRequests
        2,         // batchSize
        30,        // timeoutS
        undefined, // maxCharsPerRequest
        undefined, // pinInitialEndpointOnce
        undefined, // hedgeDelay
        undefined, // totalTimeoutS
        undefined, // hedgeBudgetPct
        undefined, // retryBudgetPct
        undefined  // maxRetries
    );
    const response = await rerankClient.rerank(
        query,
        documents,
        false, // raw_scores
        true,  // return_text
        false, // truncate
        "Right", // truncation_direction
        preference // preference parameter
    );

    console.log(`Reranked ${response.data.length} documents`);
    console.log(`Total time: ${response.total_time.toFixed(4)}s`);

    response.data.forEach((result, i) => {
        console.log(`${i + 1}. Score: ${result.score.toFixed(3)} - ${result.text?.substring(0, 50)}...`);
    });
} catch (error) {
    console.error('Reranking failed:', error.message);
}

Classification

const textsToClassify = [
    "This is great!",
    "I did not like it.",
    "Neutral experience."
];

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

try {
    const preference = new RequestProcessingPreference(
        4,         // maxConcurrentRequests
        2,         // batchSize
        30,        // timeoutS
        undefined, // maxCharsPerRequest
        undefined, // pinInitialEndpointOnce
        undefined, // hedgeDelay
        undefined, // totalTimeoutS
        undefined, // hedgeBudgetPct
        undefined, // retryBudgetPct
        undefined, // maxRetries
        undefined, // initialBackoffMs
        undefined, // cancelToken
        undefined, // primaryApiKeyOverride
        {"x-custom-header": "value"} // extraHeaders
    );
    const response = await rerankClient.classify(
        textsToClassify,
        false, // raw_scores
        false, // truncate
        "Right", // truncation_direction
        preference // preference parameter
    );

    console.log(`Classified ${response.data.length} texts`);
    console.log(`Total time: ${response.total_time.toFixed(4)}s`);

    response.data.forEach((group, i) => {
        console.log(`Text ${i + 1}:`);
        group.forEach(result => {
            console.log(`  ${result.label}: ${result.score.toFixed(3)}`);
        });
    });
} catch (error) {
    console.error('Classification failed:', error.message);
}

General Batch POST

The batch_post method is generic and can be used to send POST requests to any URL, not limited to Baseten endpoints:

const payloads = [
    { "model": "text-embedding-3-small", "input": ["Hello"] },
    { "model": "text-embedding-3-small", "input": ["World"] }
];

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

try {
    const preference = new RequestProcessingPreference(
        4,         // maxConcurrentRequests
        2,         // batchSize
        30,        // timeoutS
        undefined, // maxCharsPerRequest
        undefined, // pinInitialEndpointOnce
        undefined, // hedgeDelay
        undefined, // totalTimeoutS
        undefined, // hedgeBudgetPct
        undefined, // retryBudgetPct
        undefined, // maxRetries
        undefined, // initialBackoffMs
        undefined, // cancelToken
        undefined, // primaryApiKeyOverride
        {"x-custom-header": "value"} // extraHeaders
    );
    const response = await embedClient.batchPost(
        "/v1/embeddings", // URL path
        payloads,
        undefined, // custom headers
        preference // preference parameter
    );

    console.log(`Processed ${response.data.length} batch requests`);
    console.log(`Total time: ${response.total_time.toFixed(4)}s`);

    response.data.forEach((result, i) => {
        console.log(`Request ${i + 1}: ${JSON.stringify(result).substring(0, 100)}...`);
    });

    // Access response headers and individual request times
    response.response_headers.forEach((headers, i) => {
        console.log(`Response ${i + 1} headers:`, headers);
    });

    response.individual_request_times.forEach((time, i) => {
        console.log(`Request ${i + 1} took: ${time.toFixed(4)}s`);
    });
} catch (error) {
    console.error('Batch POST failed:', error.message);
}

Advanced Features

RequestProcessingPreference

The RequestProcessingPreference class provides a unified way to configure all request processing parameters. This is the recommended approach for advanced configuration as it provides better type safety and clearer intent.

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

// Create a preference with custom settings
const preference = new RequestProcessingPreference(
    64,        // maxConcurrentRequests (default: 128)
    32,        // batchSize (default: 128)
    30.0,      // timeoutS (default: 3600.0)
    undefined, // maxCharsPerRequest
    undefined, // pinInitialEndpointOnce
    0.5,       // hedgeDelay
    undefined, // totalTimeoutS
    0.15,      // hedgeBudgetPct (default: 0.10)
    0.08,      // retryBudgetPct (default: 0.05)
    3,         // maxRetries (default: 5)
    250        // initialBackoffMs (default: 125)
);

// Use with any method
const response = await embedClient.embed(
    ["text1", "text2"],
    "my_model",
    undefined, undefined, undefined, // encodingFormat, dimensions, user
    preference // preference parameter
);

Budget Percentages:

• hedgeBudgetPct: Percentage of total requests allocated for hedging (default: 10%)
• retryBudgetPct: Percentage of total requests allocated for retries (default: 5%)
• Maximum allowed: 300% for both budgets

Retry Configuration:

• HTTP status-code retries are controlled by maxRetries, not by retryBudgetPct.
• Retryable status codes by default: 408, 409, 429, and 500 through 599.
• Use nonRetryableStatusCodes=[529] to opt specific statuses out of the default retry policy.
• maxRetries: Maximum HTTP status-code retries per request (default: 5, max: 6). Set to 0 to disable these retries.
• retryBudgetPct: Budget for timeout and network-error retry paths (default: 5%, max: 300%).
• initialBackoffMs: Initial backoff duration in milliseconds (default: 125, range: 50-45000).
• Backoff multiplies by 4 after each retry, caps at 45000ms, and adds 0-99ms jitter. With defaults, the retry sleeps are about 125ms, 500ms, 2000ms, 8000ms, and 32000ms; a sixth retry sleeps about 45000ms.

Request Hedging

The client supports request hedging for improved latency:

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

const preference = new RequestProcessingPreference(
    8,          // maxConcurrentRequests
    2,          // batchSize
    30,         // timeoutS
    100000,     // maxCharsPerRequest
    undefined,  // pinInitialEndpointOnce
    0.5,        // hedgeDelay
    60,         // totalTimeoutS
    0.1,        // hedgeBudgetPct
    0.05,       // retryBudgetPct
    3,          // maxRetries
    250,        // initialBackoffMs
    undefined,  // cancelToken
    undefined,  // primaryApiKeyOverride
    {"x-custom-header": "value"} // extraHeaders
);
const response = await embedClient.embed(
    texts,
    "text-embedding-3-small",
    null, null, null, // encoding_format, dimensions, user
    preference // preference parameter
);

Retry Configuration

Configure retry behavior and backoff settings:

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

// Configure for more aggressive retrying
const preference = new RequestProcessingPreference(
    32,        // maxConcurrentRequests
    16,        // batchSize
    60.0,      // timeoutS
    undefined, // maxCharsPerRequest
    undefined, // pinInitialEndpointOnce
    undefined, // hedgeDelay
    undefined, // totalTimeoutS
    undefined, // hedgeBudgetPct
    0.10,      // retryBudgetPct (10% for retries)
    6,         // maxRetries (maximum allowed)
    500,       // initialBackoffMs (start with 500ms backoff)
    undefined, // cancelToken
    undefined, // primaryApiKeyOverride
    {"x-custom-header": "value"} // extraHeaders
);

const response = await embedClient.embed(
    texts,
    "text-embedding-3-small",
    null, null, null, // encoding_format, dimensions, user
    preference // preference parameter
);

Custom Headers

Use custom headers with all requests using RequestProcessingPreference:

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

const preference = new RequestProcessingPreference(
    4, // maxConcurrentRequests
    undefined, // batchSize
    30, // timeoutS
    undefined, // maxCharsPerRequest
    undefined, // pinInitialEndpointOnce
    undefined, // hedgeDelay
    undefined, // totalTimeoutS
    undefined, // hedgeBudgetPct
    undefined, // retryBudgetPct
    undefined, // maxRetries
    undefined, // initialBackoffMs
    undefined, // cancelToken
    undefined, // primaryApiKeyOverride
    { "x-custom-header": "value" } // extraHeaders
);

// Use with any method (embed, rerank, classify, batchPost)
const response = await client.embed(
    texts,
    "text-embedding-3-small",
    null, null, null, // encoding_format, dimensions, user
    preference // preference parameter
);

const batchResponse = await client.batchPost(
    "/v1/embeddings",
    payloads,
    preference // preference parameter (headers come from preference)
);

HTTP Version Selection

Choose between HTTP/1.1 and HTTP/2:

// HTTP/1.1 (default for compatibility)
const clientHttp1 = new PerformanceClient(baseUrl, apiKey, 1);

// HTTP/2 (better performance for multiple requests)
const clientHttp2 = new PerformanceClient(baseUrl, apiKey, 2);

HTTP Proxy Support

Route all HTTP requests through a proxy (e.g., for connection pooling with Envoy):

const { HttpClientWrapper } = require('@basetenlabs/performance-client');

// Create wrapper with HTTP proxy
const wrapper = new HttpClientWrapper(
    1,  // http_version
    "http://envoy-proxy.local:8080"  // proxy
);

// Share the wrapper across multiple clients
const client1 = new PerformanceClient(
    "https://api1.example.com",
    "your_key",
    undefined,  // http_version
    wrapper
);
const client2 = new PerformanceClient(
    "https://api2.example.com",
    "your_key",
    undefined,  // http_version
    wrapper
);
// Both clients will use the same connection pool and proxy

You can also specify the proxy directly when creating a client:

const client = new PerformanceClient(
    "https://api.example.com",
    "your_key",
    undefined,  // http_version
    undefined,  // client_wrapper
    "http://envoy-proxy.local:8080"  // proxy
);

API Reference

Constructors

PerformanceClient

new PerformanceClient(baseUrl, apiKey?, httpVersion?, clientWrapper?, proxy?)

• baseUrl (string): The base URL for the API endpoint
• apiKey (string, optional): API key. If not provided, will use BASETEN_API_KEY or OPENAI_API_KEY environment variables
• httpVersion (number, optional): HTTP version to use (1 for HTTP/1.1, 2 for HTTP/2). Default: 2
• clientWrapper (HttpClientWrapper, optional): Custom HTTP client wrapper for advanced configuration
• proxy (string, optional): Proxy URL to route all HTTP requests through (e.g., "http://proxy:8080")

RequestProcessingPreference

new RequestProcessingPreference(maxConcurrentRequests?, batchSize?, timeoutS?, maxCharsPerRequest?, pinInitialEndpointOnce?, hedgeDelay?, totalTimeoutS?, hedgeBudgetPct?, retryBudgetPct?, maxRetries?, initialBackoffMs?, cancelToken?, primaryApiKeyOverride?, extraHeaders?, nonRetryableStatusCodes?)

• maxConcurrentRequests (number, optional): Maximum number of parallel requests (default: 128)
• batchSize (number, optional): Number of items per batch (default: 128)
• timeoutS (number, optional): Per-request timeout in seconds (default: 3600.0)
• maxCharsPerRequest (number, optional): Character-based batching limit (default: undefined)
• pinInitialEndpointOnce (boolean, optional): Route all initial requests in one operation to one selected endpoint (default: false)
• hedgeDelay (number, optional): Request hedging delay in seconds (default: undefined)
• totalTimeoutS (number, optional): Total timeout for the entire operation in seconds (default: undefined)
• hedgeBudgetPct (number, optional): Hedge budget percentage (default: 0.10, range: 0.0-3.0)
• retryBudgetPct (number, optional): Retry budget percentage (default: 0.05, range: 0.0-3.0)
• maxRetries (number, optional): Maximum number of HTTP status-code retries per request (default: 5, max: 6)
• initialBackoffMs (number, optional): Initial backoff duration in milliseconds (default: 125, range: 50-45000)
• cancelToken (CancellationToken, optional): Token for cancelling operations
• primaryApiKeyOverride (string, optional): Override API key for requests
• extraHeaders (Record<string, string>, optional): Custom headers to include with all requests
• nonRetryableStatusCodes (number[], optional): Status codes that should not be retried even if the default policy would retry them

Methods

embed(input, model, encodingFormat?, dimensions?, user?, preference?)

• input (Array): List of texts to embed
• model (string): Model name
• encodingFormat (string, optional): Encoding format
• dimensions (number, optional): Number of dimensions
• user (string, optional): User identifier
• preference (RequestProcessingPreference, optional): Advanced configuration preference object

rerank(query, texts, rawScores?, model?, returnText?, truncate?, truncationDirection?, preference?)

• query (string): Query text
• texts (Array): List of texts to rerank
• rawScores (boolean, optional): Return raw scores (default: false)
• model (string, optional): Model name for reranking
• returnText (boolean, optional): Return text in response (default: false)
• truncate (boolean, optional): Truncate long texts (default: false)
• truncationDirection (string, optional): "Left" or "Right" (default: "Right")
• preference (RequestProcessingPreference, optional): Advanced configuration preference object

classify(inputs, rawScores?, model?, truncate?, truncationDirection?, preference?)

• inputs (Array): List of texts to classify
• rawScores (boolean, optional): Return raw scores (default: false)
• model (string, optional): Model name for classification
• truncate (boolean, optional): Truncate long texts (default: false)
• truncationDirection (string, optional): "Left" or "Right" (default: "Right")
• preference (RequestProcessingPreference, optional): Advanced configuration preference object

batchPost(urlPath, payloads, preference?)

• urlPath (string): URL path for the POST request
• payloads (Array): List of JSON payloads
• preference (RequestProcessingPreference, optional): Advanced configuration preference object (headers come from preference.extraHeaders)

Error Handling

The client throws standard JavaScript errors for various failure cases:

const { RequestProcessingPreference } = require('@basetenlabs/performance-client');

try {
    const preference = new RequestProcessingPreference();
    const response = await embedClient.embed(texts, "model", null, null, null, preference);
} catch (error) {
    if (error.message.includes('cannot be empty')) {
        console.error('Parameter validation error:', error.message);
    } else if (error.message.includes('HTTP')) {
        console.error('Network error:', error.message);
    } else {
        console.error('Other error:', error.message);
    }
}

Testing

Run the test suite:

npm test

The tests use a simple built-in test framework and validate parameter handling, constructor behavior, and error conditions.

Development

To build the native module:

# Install dependencies
npm install

# Build release version
npm run build

# Build debug version
npm run build:debug

Releasing

To release a new version of the Node.js bindings:

1. Update the version in Cargo.toml - This is the source of truth for versioning
2. Sync versions with NAPI - Run the version sync command to update package.json and regenerate code:

   napi version

3. Build the project - This regenerates the index.js file with the correct version checks:

   npm run build

4. Commit the changes - Include both Cargo.toml and package.json updates:

   git add Cargo.toml package.json
   git commit -m "chore: bump version to x.y.z"

5. Publish - The CI will automatically publish when run via workflow dispatch and setting "release" or "next" as the publish type

Important Notes

• Always update Cargo.toml first, then run napi version to sync to package.json
• The napi version command ensures version consistency between Rust and Node.js
• Rebuilding after version sync is crucial to update hardcoded version checks in the generated index.js file
• The CI will fail if package.json version doesn't match the built-in version checks

Benchmarks

Like the Python version, this Node.js client provides significant performance improvements over standard HTTP clients, especially for high-throughput embedding and reranking workloads.

License

MIT License

Acknowledgements:

Venkatesh Narayan (Clay.com) for the prototpe of this here https://github.com/basetenlabs/truss/pull/1778 and Suren (Baseten) for getting a PoC and protyping the release pipeline. https://github.com/suren-atoyan/rust-ts-package