🚀 DFScript

DFScript is a lightweight, high-performance, and zero-dependency data analysis library for TypeScript and JavaScript. Heavily inspired by modern dataframe libraries like Polars and Pandas, DFScript brings a robust, expression-based columnar data processing engine directly to the JavaScript ecosystem.

With optimized columnar storage under the hood, DFScript enables you to build clean, maintainable, and type-safe data pipelines using a declarative expression API.

💡 Why DFScript?

Like many developers, I spent years working on frontend codebases filled with deeply nested, unoptimized data processing loops. We've all seen them: chained .map(), .filter(), .sort(), and .forEach() loops processing arrays of objects. Each step in the chain allocates new intermediate arrays, triggers garbage collection thrashing, and operates on row-based layouts that slow down browser main threads.

When you need to perform complex analytical transformations on the client-side—handling missing values, formatting timezone-aware datetimes, parsing decimals, or computing rolling averages—relying on standard JavaScript array methods quickly becomes a performance and maintenance nightmare. There had to be a better, cleaner, and faster way to express these pipelines. I realized that if nobody else had built a lightweight, zero-dependency columnar expression engine for the JavaScript ecosystem yet, why not roll up my sleeves and build it myself?

DFScript was born from that frustration. It brings Polars-like columnar execution, strict domain namespaces (.str, .dt, .arr, .struct), and lazy/declarative expression compilation to JavaScript. Under the hood, it avoids intermediate array allocations and uses highly optimized loops with cached lengths, so your frontend code stays clean, type-safe, and incredibly fast.

🗺️ Table of Contents

• ✨ Key Features
• ⚙️ Compatibility & Design Principles
• 🤝 Contributing & Collective Wisdom
• 📦 Installation
• 🚀 Quick Start
• 📖 Core Concepts
• 🛠️ DataFrame API Reference
• 📂 File / Data I/O
• 🧮 Expressions API Reference
• 📂 Namespaces
• 🪟 Window & Rolling Expressions
• 🛡️ Typing and Schema Registry
• 🧑‍💻 Contributing & Development
• 📄 License

✨ Key Features

• 📦 Zero Dependencies — Extremely lightweight with zero runtime overhead.
• ⚡ Columnar Execution — Operates on efficient columnar arrays, minimizing allocations and speed bottlenecks.
• 🔗 Expression-Based API — Compose complex calculations, mappings, and filters using fluent, Polars-like expressions.
• 📂 Strict Namespaces — Clear API organization for specific domains:
  • .str for advanced string manipulations.
  • .dt for microsecond-precision datetimes, timezones, and duration calculations.
  • .arr for robust array/list column operations.
  • .struct for nested object and struct manipulation.
• 🪟 Analytical Window Functions — Windowing (over()), cumulative aggregations (cum_sum(), cum_max()), and rolling metrics (rolling_mean(), rolling_std()).
• 🛠️ Relational Operations — Rich, high-speed joins, pivots, unpivots, vertical/horizontal concatenations, and group-by aggregations.
• 🛡️ Defensive & Type-Safe — Native type-coercion, robust null-safety, and strict schema validation.

⚙️ Compatibility & Design Principles

DFScript is designed with a low-abstraction, zero-dependency philosophy to guarantee maximum compatibility, predictability, and runtime performance:

• 📦 Zero External Dependencies — Lightweight footprint with zero runtime overhead or supply chain vulnerabilities.
• 🌐 Universal Compatibility — Works out-of-the-box in any JavaScript/TypeScript environment, including Node.js, Deno, Bun, web browsers, and cloud/edge workers.
• 🧱 Built-in Standards — Prioritizes native, built-in APIs (like standard Date, Intl formatting, and TextEncoder) and standard arrays rather than custom wrappers or heavy runtime abstractions.
• ⚡ Optimized Execution Paths — Under the hood, performance-critical code avoids higher-level array iterators and short-lived intermediate allocations in favor of simple, fast for and while loops with cached lengths, keeping garbage collection overhead to an absolute minimum.
• 🔄 Easy Transpilation — Relies strictly on low-level native operations, making it fully compatible with older environments (like ES6 or even ES5) without requiring complex polyfills or modern engine-specific features.

🤝 Contributing & Collective Wisdom

We don’t pretend to have encountered every localized date format, database quirk, or environment-specific edge case. DFScript is built on the belief that software correctness is a collective endeavor.

If you run into an unsupported edge case or unexpected behavior in any of our functions, we want to hear about it. Help us harden this engine by opening a GitHub Issue or submitting a PR—every report makes the library more robust for everyone.

📦 Installation

Install DFScript using your favorite package manager:

npm install df-script

Or with Yarn/PNPM:

yarn add df-script
pnpm add df-script

🚀 Quick Start

Here is a quick example showing how to load data, run expressions, perform aggregations, and compute rolling statistics.

import { $df } from "df-script";

// 1. Create a DataFrame with structured data and automatic schema inference
const df = $df.data([
  { id: 1, name: "Alice", join_date: "2026-01-15", sales: 1200.50, tags: ["sales", "east"] },
  { id: 2, name: "Bob", join_date: "2026-02-20", sales: 850.00, tags: ["support", "west"] },
  { id: 3, name: "Charlie", join_date: "2026-03-05", sales: 2300.00, tags: ["sales", "north"] },
  { id: 4, name: "David", join_date: "2026-03-12", sales: null, tags: ["marketing"] },
]);

// 2. Select columns, transform strings, format dates, and fill missing values
const processedDf = df.select(
  $df.col("id"),
  $df.col("name").str.upper().alias("NAME_UPPER"),
  $df.col("join_date").str.to_datetime().dt.year().alias("join_year"),
  $df.col("sales").add(500).alias("sales_adjusted"),
  $df.col("tags").arr.lengths().alias("tag_count")
);

console.log(processedDf.to_dicts());
/* Output:
[
  { id: 1, NAME_UPPER: 'ALICE', join_year: 2026, sales_adjusted: 1700.5, tag_count: 2 },
  { id: 2, NAME_UPPER: 'BOB', join_year: 2026, sales_adjusted: 1350, tag_count: 2 },
  { id: 3, NAME_UPPER: 'CHARLIE', join_year: 2026, sales_adjusted: 2800, tag_count: 2 },
  { id: 4, NAME_UPPER: 'DAVID', join_year: 2026, sales_adjusted: null, tag_count: 1 }
]
*/

📖 Core Concepts

The $df Entry Point

DFScript uses the $df namespace to bootstrap DataFrames, refer to columns, build expressions, and access data types.

• $df.data(dataRowsOrCols, schema?): Instantiates a new DataFrame.
• $df.read_json(content, options?): Reads JSON/NDJSON content into a new DataFrame.
• $df.read_csv(content, options?): Reads CSV content into a new DataFrame.
• $df.col(name): Creates a column reference expression.
• $df.all(): Selects all columns in the DataFrame.
• $df.exclude(columns): Creates an expression matching all columns except the specified ones.
• $df.coalesce(...exprs): Returns the first non-null value among columns or literal expressions.
• $df.lit(val): Explicitly wraps a raw value into a literal expression.
• $df.duration(options): Constructs a Duration expression supporting days, hours, minutes, seconds, milliseconds, weeks, and timeUnit precision.
• $df.struct(fields): Constructs a nested Struct object column expression from an object of named expressions or sibling columns.
• $df.when(predicate).then(value)...otherwise(value): Constructs a conditional expression (when-then-otherwise chain).
• $df.implode(column): Aggregates a column's rows (or grouped values) into a list.
• $df.seq_range(value, options?): Generates a sequence range of values.
• $df.element(): References the current array element within an .arr.eval(...) expression.
• $df.DataType: Direct access to the DataTypeRegistry for schema specification.

DataFrames vs. Columns

• DataFrame holds data in a columnar-oriented object: columns: Record<string, any[]>.
• ColumnExpr represents an evaluation sequence over rows. Operations (arithmetic, strings, lists, date-time, comparisons) are chained to build a tree of computations evaluated lazily.

🛠️ DataFrame API Reference

1. Transformations & Projection

• select(...exprs): Projects columns. Supports strings, raw column names, $df.col(...) expressions, $df.all(), and $df.col("struct").struct.unnest().
• with_columns(...exprs): Adds or overrides columns. Accepts expressions, strings, or options objects mapping keys to values/expressions.
• drop(...names): Drops one or more columns from the DataFrame.
• rename(mapping): Renames columns using a { oldName: newName } object.
• explode(columns): Unnests list-like columns into multiple rows, replicating other columns per list element.
• implode(columns): Groups values in specified columns back into a single list element per column.

2. Filtering & Row Selection

• filter(...predicates): Filters rows where all predicate expressions evaluate to true (or non-null truthy values).
• find(predicate): Evaluates a predicate expression and returns the first matching row record object (or undefined if no match is found).
• unique(columns?): Returns unique rows. If a subset of columns is provided, deduplicates based on those columns.
• limit(n, options?): Returns the first n rows. Options include offset and direction from: "start" | "end".
• head(n) / tail(n): Shortcuts for limit from the start or end of the DataFrame.
• slice(start, end?): Extract a subset of rows using standard index slicing.
• gather(indices, options?): Gathers rows at specified indices. Supports single index, arrays of indices, and negative indexing. Options include { null_on_oob?: boolean }.

3. Sorting & Structural Operations

• sort({ by, descending?, nullsLast?, custom? }): Sorts rows. Supports single or multiple columns/expressions, custom descending configurations per column, custom null sorting rules, and custom comparator functions.
• clone(): Performs a complete deep copy of the DataFrame, replicating all underlying column arrays and schema metadata.

4. Grouping & Aggregations

• groupby(keys): Groups the data by one or more columns, returning a GroupedData object.
• GroupedData.agg(...exprs): Run aggregations on grouped data (e.g. $df.col("sales").sum()).

5. Reshaping & Joining

• join(other, onOrOptions, how?, suffixes?): Merges two DataFrames. Supports:
  • Join modes (how): "inner" | "left" | "right" | "outer" | "semi" | "anti" | "cross".
  • Heterogeneous key names (leftOn, rightOn).
  • Key coalescing (coalesce: boolean).
  • Row order preservation (maintain_order: "none" | "left" | "right" | "left_right" | "right_left").
• join_asof(other, options): Performs inexact time-series or nearest-neighbor joins on sorted key columns.
  • Parameters: on, leftOn, rightOn, grouping parameters (by, leftBy, rightBy), matching strategy ("backward" | "forward" | "nearest"), numeric/duration tolerance, and allow_exact_matches.
• pivot(index, columns, values): Pivots the table, converting unique values in columns into column headers.
• unpivot(idVars, valueVars, varName?, valueName?): Melts/unpivots the table, converting wide columns into long format name-value pairs.
• concat(items, options?): Concatenates multiple DataFrames. Supported concat strategies: "vertical" | "horizontal" | "diagonal".

📂 File / Data I/O

DFScript provides helpers to serialize and parse data formats like JSON and CSV.

Reading Data

• $df.read_json(content, options?): Reads a JSON array or Newline Delimited JSON (NDJSON) string and loads it into a new DataFrame.

  import { $df } from "df-script";
  
  // Read standard JSON array
  const df = $df.read_json('[{"id": 1, "name": "Alice"}]');
  
  // Read Newline Delimited JSON (NDJSON)
  const dfNdjson = $df.read_json('{"id": 1}\n{"id": 2}', { format: "ndjson" });

• $df.read_csv(content, options?): Reads a CSV string and loads it into a new DataFrame, with automatic data type inference.

  import { $df } from "df-script";
  
  const csvContent = "id,name,active\n1,Alice,true\n2,Bob,false";
  const df = $df.read_csv(csvContent, {
    separator: ",",
    hasHeader: true,
    inferSchema: true
  });

Writing Data

• df.write_json(file?, options?): Serializes a DataFrame into a JSON or NDJSON string. If a file path or writable stream/object (with a .write method) is provided, writes/streams the content as a side-effect. Always returns the serialized string.

  // Write to a file and get the string
  const jsonStr = df.write_json("output.json");

• df.write_csv(file?, options?): Serializes a DataFrame into a CSV string. Supports options for headers, custom separators, quote styles, float precision, and BOM. If a file path or writable stream/object (with a .write method) is provided, writes/streams the content as a side-effect. Always returns the serialized string.

  // Serialize to a CSV string
  const csvStr = df.write_csv();
  
  // Write to a file with custom separator
  df.write_csv("output.csv", { separator: ";" });

🧮 Expressions API Reference

All column expressions inherit from ExprBase and support standard operators.

➕ Arithmetic Expressions

Chained mathematical functions execute cleanly with built-in null-safety (Kleene logic).

• .add(val), .sub(val), .mul(val), .div(val), .floordiv(val), .mod(val), .pow(val)
• .abs(), .sqrt(), .cbrt(), .exp(), .expm1(), .log(base?), .log1p()
• .ceil(), .floor(), .trunc(), .round(decimals), .clip(lower, upper), .sign(), .negate()
• .sin(), .cos(), .tan(), .sinh(), .cosh(), .tanh(), .asin(), .acos(), .atan(), .asinh(), .acosh(), .atanh(), .degrees(), .radians(), .hypot(val)

🔍 Comparison Expressions

• .eq(val), .ne(val) — Strict value equivalence (null values return null).
• .eq_missing(val), .ne_missing(val) — Equality checking that treats null/undefined values as equal.
• .gt(val), .ge(val), .lt(val), .le(val)
• .is_null(), .is_not_null()
• .is_finite(), .is_infinite(), .is_nan(), .is_not_nan()
• .is_in(arrayOrExpr), .not_in(arrayOrExpr)

⚡ Aggregations

• .sum(), .product(), .avg() / .mean(), .median(), .mode(), .variance(), .std(), .skew(), .kurtosis(), .entropy(base?, normalize?)
• .min(), .max(), .nan_min(), .nan_max(), .min_by(by), .max_by(by), .arg_min(), .arg_max()
• .count(options?) — Option { includeNulls: boolean }.
• .first(), .last()
• .any(), .all(), .any_null(), .all_null(), .n_unique(), .null_count()
• .bitwise_and(), .bitwise_or(), .bitwise_xor()

🔀 Control Flow & Conditionals

Construct dynamic CASE WHEN branches using the $df.when API:

import { $df } from "df-script";

df.select(
  $df.col("sales"),
  $df.when($df.col("sales").gt(2000)).then("High Performance")
     .when($df.col("sales").gt(1000)).then("Standard Performance")
     .otherwise("Low Performance")
     .alias("sales_category")
);

• $df.when(predicate).then(value): Starts a conditional evaluation.
• .when(predicate).then(value): Chains additional conditions.
• .otherwise(value): Specifies the fallback value when no conditions match (returns a complete ColumnExpr).
• $df.coalesce(...exprs): Returns the first non-null value among the provided expressions or literals.

📂 Namespaces

To maintain a clean and uncluttered API namespace, specific data transforms are grouped under dedicated accessors.

🔤 String Operations (.str)

Available on any expression via .str:

$df.col("name").str.lower()
$df.col("code").str.starts_with("A")
$df.col("description").str.replace(/foo/i, "bar")

• Methods: lower(), upper(), to_titlecase(), len(), len_bytes(), len_chars(), trim(), trim_start(), trim_end(), starts_with(prefix), ends_with(suffix), contains(pattern), contains_any(patterns), count_matches(pattern), find(pattern), find_many(patterns), replace(pattern, repl), replace_all(pattern, repl), replace_many(patterns, replacements), slice(offset, length?), split(delimiter, options?), explode(), reverse(), lpad(w, f), rpad(w, f), zfill(w), strip_chars(chars?), strip_chars_start(chars?), strip_chars_end(chars?), strip_prefix(pfx), strip_suffix(sfx), escape_regex(), extract(pattern, groupIndex?), extract_all(pattern), extract_groups(pattern), extract_many(patterns), encode(encoding), decode(encoding, strict?), json_decode(options?), json_path_match(jsonPath), normalize(form?), join(separator), strptime(format, strict?), to_integer(), to_decimal(p, s), to_date(), to_datetime(), to_time().

📅 Temporal Operations (.dt)

Available on datetime or duration values via .dt:

$df.col("timestamp").dt.year()
$df.col("timestamp").dt.convert_time_zone("America/New_York")
$df.col("duration").dt.total_seconds()

• Datetime Methods: year(), month(), day(), hour(), minute(), second(), millisecond(), microsecond(), nanosecond(), weekday(), week(), quarter(), century(), millennium(), ordinal_day(), is_leap_year(), month_start(), month_end(), date(), time(), offset_day(n, options?), offset_business_day(n, options?), convert_time_zone(tz), cast_time_unit(unit), with_time_unit(unit), replace(options), truncate(every), utc_offset(timeZone?, options?), epoch(unit), timestamp(unit), strftime(format, locale?).
• Duration Methods: total_days(), total_hours(), total_minutes(), total_seconds(), total_milliseconds(), total_microseconds(), total_nanoseconds().

📊 Array/List Operations (.arr)

Available on any array or list column expression via .arr:

$df.col("tags").arr.contains("vip")
$df.col("matrix").arr.get(2)

// Element-wise manipulation inside arrays:
$df.col("numbers").arr.eval(element().mul(2)).alias("numbers_doubled")
$df.col("tags").arr.eval(element().str.to_uppercase()).alias("upper_tags")

• Methods: lengths(), len(), get(idx, null_on_oob?), first(null_on_oob?), last(null_on_oob?), gather(indices, null_on_oob?), gather_every(n, offset?), slice(offset, length?), contains(item), count_matches(item), join(separator), sort(descending?), reverse(), unique(), sum(), mean(), median(), mode(), min(), max(), arg_min(), arg_max(), agg(expr), eval(expr).

🗃️ Struct/Object Operations (.struct)

Available on any struct or nested object column expression via .struct. You can access fields dynamically via properties or explicit methods:

// Sibling fields access via Proxy
$df.col("address").struct.city.alias("city")

// Explicit struct field access or struct creation
$df.struct({ city: $df.col("city"), state: $df.col("state") })

• Methods:
  • field(name): Accesses a field within the struct.
  • rename_fields(mapping): Renames fields in the struct based on a { oldKey: newKey } mapping.
  • with_fields(fields): Adds or overrides fields in the struct. Accepts an array of aliased expressions or an object.
  • unnest(): Expands the fields of the struct into individual top-level columns in a select projection.

    // Flattens the address struct into "city", "state", etc. at the top-level
    df.select($df.col("address").struct.unnest())

🪟 Window & Rolling Expressions

DFScript provides full support for analytic partition window operations using .over() and rolling filters.

// Calculate partition cumulative sums and row numbers
df.select(
  $df.col("department"),
  $df.col("sales"),
  $df.col("sales").sum().over("department").alias("dept_total_sales"),
  $df.col("sales").cum_sum().over("department").alias("dept_running_sales"),
  $df.all().row_number().over("department").alias("dept_rank")
);

1. Cumulative Windows

• .cum_sum(reverse?)
• .cum_prod(reverse?)
• .cum_min(reverse?)
• .cum_max(reverse?)
• .cum_count(reverse?)

2. Rolling Metrics (Moving Window)

Apply moving calculations over a fixed window size:

• .rolling_sum(size)
• .rolling_mean(size)
• .rolling_median(size)
• .rolling_min(size)
• .rolling_max(size)
• .rolling_std(size)
• .rolling_rank(size)
• .rolling_quantile(quantile, size)

3. Positional & Rank Windows

• .lead(offset, defaultVal?)
• .lag(offset, defaultVal?)
• .rank()
• .dense_rank()
• .row_number()

🛡️ Typing and Schema Registry

You can optionally declare schemas to enforce precise data types and automatic type coercion during construction.

import { $df } from "df-script";

const schema = {
  id: $df.DataType.Int32,
  price: $df.DataType.Decimal(10, 2),
  active: $df.DataType.Boolean,
  created_at: $df.DataType.Datetime
};

const df = $df.data(rawData, schema);

🧠 TypeScript Type Inference & IDE Safety

When you pass a schema to $df.data(), DFScript's types automatically infer the target types of the fields, turning dynamic dataframes into compile-safe records. Your IDE will auto-complete column names and validate that operations match the underlying types.

import { $df } from "df-script";

const schema = {
  name: $df.DataType.Utf8,
  age: $df.DataType.Int32,
  is_active: $df.DataType.Boolean
};

// Inferred DataFrame type is DataFrame<{ name: string; age: number; is_active: boolean }>
const df = $df.data(rawData, schema);

// Full IDE autocomplete, type validation, and compiler safety!
const activeUsers = df.filter($df.col("is_active").eq(true));

⚡ Post-Operation Schema Type Deduction

DFScript features an intelligent post-operation schema inference engine that automatically determines the correct resulting DataType across complex expression trees without requiring manual .cast() calls:

• Temporal & Duration Arithmetic:
  • Datetime - Datetime => Duration
  • Datetime ± Duration => Datetime
  • Time - Time => Duration
  • Time ± Duration => Time
  • Duration * / Numeric => Duration
• Integer Promotion & Signedness Hierarchy: Automatically preserves or promotes integer widths:
  • Preserves exact unsigned/signed types (UInt8 + UInt8 => UInt8, Int16 + Int16 => Int16).
  • Correctly promotes across sizes and signedness (UInt16 + Int8 => Int16, UInt32 + Int32 => Int32, Int32 + Int64 => Int64).
• Floating-Point & Decimal Resolution:
  • Int + Float => Float64
  • Float32 + Float32 => Float32
  • Decimal + Int => Decimal
  • Evaluates non-integer operation results (e.g. division 10 / 3) and promotes to Float64.
• Conditional Branch Widening: Multi-branch expressions (when().then().otherwise()) progressively widen branch types to their common denominator.
• Statistical Aggregations: .mean() and .std() promote integer columns to Float64, while non-numeric .count() operations resolve to Int32.

Supported Data Types

• Integers: Int8, Int16, Int32, Int64, UInt8, UInt16, UInt32, UInt64
• Floats & Decimals: Float32, Float64, Decimal(precision?, scale?)
• General: Boolean, Utf8 (Strings), Binary, Null, Object
• Temporal: Date, Datetime, Time, Duration
• Nested Structures: List (Arrays), Struct (Objects)

🧑‍💻 Contributing & Development

We welcome contributions! Please make sure to review our Developer Guidelines when writing code.

Running Project Tests

DFScript has a comprehensive suite of unit tests. Run them using:

npx tsx _tests/run_all_project_tests.ts

📄 License

DFScript is open-source software licensed under the MIT License.