@spark-connect-js/node

TypeScript client for Apache Spark Connect. Talks to Spark over gRPC and decodes results with Apache Arrow. No JVM in your Node process.

Note: This project is in early development (v0.4.0) and is not recommended for production usage, but feedback is very welcome on GitHub.

Install

npm install @spark-connect-js/node

Quick example

import { connect, col, sum, desc } from "@spark-connect-js/node";

const spark = connect("sc://localhost:15002");

const result = await spark
  .table("employees")
  .filter(col("age").gt(30))
  .groupBy("dept")
  .agg(sum("salary").alias("total"))
  .sort(desc("total"))
  .collect();

await spark.stop();

DataFrame methods build a logical plan locally. The plan is sent to the server only when an action runs (collect, count, show, first, head, take, toLocalIterator, or a DataFrameWriter save method).

Compatibility

| Component | Required | | -------------------- | --------------------------------------------------- | | Node.js | 22 LTS or 24.x | | Spark Connect server | Spark 3.4+ (recommended: 4.0+) | | Connection scheme | sc://host:port[/;use_ssl=true][;token=...][;...] |

Package layout

This package re-exports the full public API from @spark-connect-js/core and adds the gRPC transport (via @grpc/grpc-js) and the Arrow IPC decoder (via apache-arrow). For application code, this is the only package you install. @spark-connect-js/connect is pulled in transitively as an implementation detail.

Documentation

Full docs at prustic.github.io/spark-connect-js.

• Quickstart: start a local Spark Connect server, run a query
• SQL and DataFrame guide: transformations, actions, the Column DSL
• Comparison to PySpark: differences between PySpark and this client
• I/O: reading and writing Parquet, ORC, CSV, JSON
• Error handling: typed error hierarchy and gRPC status codes
• Roadmap: what's shipped, what's planned

Runnable example scripts in examples/.

License

Apache-2.0