wynd-dataingest-ts

High-performance TypeScript client for wynd data ingest.

It is built for Node.js services that want:

• connection reuse through undici
• small API surface for single events and batches
• producer-style internal batching with linger-based flushing
• bounded retries with exponential backoff and jitter
• clean packaging for npm distribution

Install

npm install wynd-dataingest

Quick start

import { WyndDataIngestClient } from "wynd-dataingest";

const client = new WyndDataIngestClient({
  url: "http://127.0.0.1:8686/events",
  connections: 16,
  tableName: "orders",
  tableOperation: "insert",
  compression: {
    gzip: true,
  },
  headers: {
    authorization: "Basic my-token",
  },
  retry: {
    maxAttempts: 5,
    initialDelayMs: 200,
    maxDelayMs: 5000,
    factor: 2,
    jitterRatio: 0.2,
  },
});

await client.send([
  {
    service: "api",
    level: "info",
    message: "order accepted",
  },
  { service: "api", message: "batch item 1" },
  { service: "api", message: "batch item 2" },
]);

await client.close();

This client is NDJSON-only, so send() always expects an array and always sends newline-delimited JSON.

API

new WyndDataIngestClient(options)

options supports:

• url: full endpoint URL
• connections: max pooled HTTP connections; defaults to 16
• tableName: base table name for the table-name header
• tableOperation: "insert" | "new" | "update" | "delete"; defaults to "insert" when tableName is set
• headers: default request headers
• timeoutMs: per-request timeout; defaults to 30000
• userAgent: override the default user agent
• agent: undici agent options for connection tuning
• dispatcher: inject a custom undici dispatcher
• retry: retry configuration
• compression: optional body compression settings
• batching: optional internal batching controls for producer-style buffering

Sending data

The client exposes one public send method:

• send(payload, options)
• close()

send() behavior:

• payload must be an array
• payload is serialized as newline-delimited JSON
• content-type is always application/x-ndjson
• when batching is enabled, multiple send() calls can be coalesced into one HTTP request
• batches can be flushed concurrently when maxInflightBatches is greater than 1

Internal batching

You can buffer multiple send() calls similarly to a Kafka producer linger.ms setting:

const client = new WyndDataIngestClient({
  url: "http://127.0.0.1:8686/events",
  batching: {
    lingerMs: 1000,
    maxRecords: 500,
    maxBytes: 5 * 1024 * 1024,
    maxInflightBatches: 4,
  },
});

Flush happens when the first of these limits is hit:

• lingerMs: max time to wait before sending the buffered records
• maxRecords: max number of buffered records in one request
• maxBytes: max raw NDJSON bytes before compression
• maxInflightBatches: max number of batch requests that can be in flight at the same time

Default batching values:

• lingerMs: 1000
• maxRecords: 500
• maxBytes: 5MB
• maxInflightBatches: 4

This keeps latency bounded while allowing the client to combine nearby sends into larger requests.

When maxInflightBatches is greater than 1, later batches do not wait for earlier batches to finish sending, but strict batch ordering is no longer guaranteed.

When batching is enabled, the client also does a best-effort flush during common Node.js shutdown paths such as beforeExit, SIGINT, SIGTERM, uncaught exceptions, and unhandled rejections. This helps reduce data loss during app shutdown, but it cannot guarantee delivery for hard kills like SIGKILL or abrupt process termination outside Node's lifecycle hooks.

Table routing header

When tableName is configured, the client automatically adds a table-name header:

• tableOperation: "insert" or "new" -> table-name: <tableName>
• tableOperation: "update" -> table-name: <tableName>-updates
• tableOperation: "delete" -> table-name: <tableName>-deletes

Compression

You can enable gzip with built-in defaults:

const client = new WyndDataIngestClient({
  url: "http://127.0.0.1:8686/events",
  compression: {
    gzip: true,
  },
});

That uses the default gzip threshold of 100KB raw bytes.

You can also customize gzip based on the original payload size before compression:

const client = new WyndDataIngestClient({
  url: "http://127.0.0.1:8686/events",
  compression: {
    gzip: {
      minBytes: 4096,
      level: 6,
    },
  },
});

When enabled:

• gzip is only applied when the raw request body size is greater than or equal to minBytes
• the client sets Content-Encoding: gzip
• if you already set content-encoding manually, the client does not override it

Connection tuning

You can tune the connection pool directly:

const client = new WyndDataIngestClient({
  url: "http://127.0.0.1:8686/events",
  connections: 64,
});

If you also pass agent.connections, that lower-level agent value takes precedence because it is spread after the default pool settings.

Retry behavior

Retries are attempted for transport failures and these response codes by default:

• 408
• 425
• 429
• 500
• 502
• 503
• 504

Backoff uses:

• exponential growth
• optional jitter
• Retry-After header when present on retryable responses
• a hard maxAttempts limit

Publish

npm run build
npm test
npm publish --access public

If you want to publish under a scope, update the name field in package.json first.