Snowflake Streaming Ingest Node.js SDK

A high-performance Node.js SDK for streaming data ingestion into Snowflake, built with Rust for optimal performance and reliability.

• Overview
• Supported Platforms
• Installation
• Quick Start
• Features
• Architecture
• Dependencies
• Support
• License

Overview

The Snowflake Streaming Ingest Node.js SDK provides a Node.js interface for real-time data streaming into Snowflake tables. It leverages a Rust core for high performance while providing a familiar JavaScript API for easy integration.

Supported Platforms

| OS | Architecture | C Library | Notes | |----|-------------|-----------|-------| | Linux | x64 | glibc (>= 2.26) | | | Linux | x64 | musl | Alpine Linux, etc. | | Linux | arm64 | glibc (>= 2.26) | | | Linux | arm64 | musl | Alpine Linux, etc. | | macOS | arm64 | — | Apple Silicon, macOS 11.0+ | | Windows | x64 | MSVC | Windows 10+ |

Node.js

• Node.js 20 or later

Installation

npm install snowpipe-streaming

The correct native binary for your platform is installed automatically via optionalDependencies.

Quick Start

const { createClient } = require("snowpipe-streaming");

// Create a client
const client = await createClient({
  clientName: "my-client",
  dbName: "MY_DB",
  schemaName: "MY_SCHEMA",
  pipeName: "MY_PIPE",
  properties: {
    account: "your_account",
    user: "your_user",
    private_key: "-----BEGIN PRIVATE KEY-----\n...",
  },
});

// Open a channel
const { channel } = await client.openChannel({ name: "my_channel" });

// Insert data
await channel.insertRows([
  { id: 1, name: "John Doe", timestamp: "2024-01-01T00:00:00Z" },
  { id: 2, name: "Jane Smith", timestamp: "2024-01-02T00:00:00Z" },
]);

// Wait for data to be committed to Snowflake
await channel.waitForCommit();

// Close resources
await channel.close();
await client.close();

Features

• High Performance: Rust-based core for optimal throughput and low latency
• Memory Efficient: Minimal memory footprint with efficient data handling
• Automatic Retries: Built-in retry logic for transient failures
• Backpressure Handling: Intelligent backpressure management
• Comprehensive Logging: Detailed logging for debugging and monitoring
• Cross-Platform: Native support for Linux, macOS, and Windows
• Zero Dependencies: No JavaScript runtime dependencies

Architecture

The SDK uses a hybrid Rust-Node.js architecture:

• Rust Core: High-performance data processing, networking, and Snowflake communication
• Node.js Bindings: JavaScript API using napi-rs for seamless integration
• Platform Packages: Architecture-specific native binaries distributed as npm optional dependencies

Dependencies

• None: The SDK has zero JavaScript runtime dependencies
• Native Extensions: Rust-based binary modules (automatically installed for your platform)

Support

• Documentation: Snowpipe Streaming Overview

License

This project is licensed under the Apache License 2.0.