@srtv/toondash

v1.0.1

Published

a month ago

Toondash. High-performance data manipulation, parsing, serializing, streaming, and 40-70% LLM token reduction.

Downloads

366

0High
0Medium
0Low

ayush.srtv

toon toondash lodash llm tokens token-reduction prompt-optimization data-manipulation parser serializer streaming tree-shakable zero-dependencies

ToonDash ⚡

ToonDash is a high-performance, zero-dependency utility library for Token-Oriented Object Notation (TOON) format. It enables powerful data manipulation, filtering, and aggregation on token-compressed data structures used in LLM prompts, structured outputs, telemetry pipelines, and agent memory.

🚀 Why TOON & ToonDash?

JSON is the standard interchange format for the web, but its verbose repetition of keys and syntax characters (", {, }, [, ], ,) wastes 30% to 70% of LLM context window tokens.

TOON (Token-Oriented Object Notation) formats uniform collections into compact tabular arrays with single header declarations, eliminating repetitive keys while preserving types and nested hierarchies:

# Standard JSON: ~840 tokens
# TOON Equivalent: ~310 tokens (63% token reduction)
employees[3, id, name, role, dept, salary, active]:
  1, Sarah Chen, Staff Engineer, Engineering, 165000, true
  2, Marcus Vance, Product Lead, Product, 142000, true
  3, Elena Rostova, Security Architect, Engineering, 158000, true
tags[4]: high-growth, remote-first, tech, enterprise

ToonDash lets you query, filter, aggregate, reshape, and serialize TOON data in JavaScript/TypeScript with a rich, composable API.

✨ Features

50+ Collection & Object Utilities: map, filter, reduce, find, groupBy, keyBy, sortBy, orderBy, chunk, compact, uniqBy, merge, pick, omit, get, set, sumBy, meanBy, maxBy, minBy, and more.
Fluent Method Chaining: Seamless td(input).filter(...).sortBy(...).map(...).toToon() chaining.
Bidirectional Lossless Conversion: parseToon(str) and stringifyToon(obj) with strict type casting (booleans, integers, floats, quoted strings).
Ultra-Fast Streaming Scanner: iterate(toonString, (row, index) => ...) for processing million-row TOON streams without full AST memory allocations.
Zero Runtime Dependencies: Pure TypeScript implementation; lightweight, dual ESM/CJS build, and fully tree-shakeable.
Built-in Token Savings Benchmark: Live interactive token analysis across Gemini 2.0/1.5, Claude 3.5, GPT-4o, and Llama 3 models.
Comprehensive Unit Tests: Full test suite verifying correct collection operations, predicate handling, and deep object mutations.

🔄 Automated CI/CD Pipelines

This repository includes enterprise-grade GitHub Actions workflows to ensure zero-defect deployments and automated packaging.

1. 🛡️ Continuous Integration & Test Suite (`.github/workflows/ci.yml`)

Trigger: Every push or pull request to main and master.
Quality Gates:
- Matrix testing across Node.js 18.x, 20.x, and 22.x.
- Static type-checking and linting (npm run lint).
- Full automated collection operation & TOON parser unit tests (npm run test).
- Production build verification (npm run build).

2. 🌐 Tested Static GitHub Pages Deployment (`.github/workflows/deploy.yml`)

Pre-deployment Verification: Enforces that type-checking, linter, and all 44+ unit tests pass before deploying.
Free Hosting: Builds and deploys the interactive playground directly to GitHub Pages without server costs.

3. 📦 Publish to NPM Pipeline (`.github/workflows/publish-npm.yml`)

Trigger: Published GitHub releases, semantic version tags (e.g., v1.0.0), or manual trigger via Actions → Run workflow (supports dry-run mode).
Automated Steps:
1. Runs full validation and test suite.
2. Compiles dual ESM (dist-lib/index.mjs), CommonJS (dist-lib/index.cjs), and TypeScript definitions (.d.ts).
3. Publishes with provenance to the NPM registry using secret NPM_TOKEN.

🌐 Free Static Hosting on GitHub Pages Setup

Push your repository to GitHub:

git init
git add .
git commit -m "feat: initial commit of ToonDash"
git branch -M main
git remote add origin https://github.com/<your-username>/<your-repo-name>.git
git push -u origin main

Enable GitHub Pages in your Repository Settings:
- Go to your repository on GitHub.
- Navigate to Settings → Pages (under the "Code and automation" sidebar).
- Under Build and deployment > Source, select GitHub Actions.
Your site is live:
- The .github/workflows/deploy.yml workflow will automatically test and publish your app.
- Live URL: https://<your-username>.github.io/<your-repo-name>/

🚀 NPM Publishing Setup

To publish new releases to the NPM registry:

Generate an NPM Access Token:
- Log in to npmjs.com → Profile → Access Tokens.
- Generate a token with Automation or Publish permissions.
Add the Token to GitHub Secrets:
- Go to your GitHub repository → Settings → Secrets and variables → Actions.
- Click New repository secret.
- Name: NPM_TOKEN
- Value: Paste your NPM access token.
Publishing a Release:
- Option A (Release / Git Tag):
```
git tag v1.0.1
git push origin v1.0.1
```
  Creating a release or pushing a v*.*.* tag automatically tests, builds, and publishes to NPM.
- Option B (Manual via GitHub UI):
  - Navigate to the Actions tab on GitHub → Select Publish Package to NPM → Click Run workflow (optionally toggle dry-run or specify a distribution tag like beta or latest).

📦 Installation & Usage

npm install @srtv/toondash
# or using pnpm / yarn / bun
pnpm add @srtv/toondash

🌲 100% Zero Dependencies & Tree-Shakeable (sideEffects: false)

ToonDash has zero runtime dependencies. Installing @srtv/toondash downloads only the pure TypeScript/JavaScript distribution (~21KB minified for the full bundle).

Thanks to "sideEffects": false and modular architecture, modern bundlers (Webpack, Vite, Rollup, Next.js, Turbopack, esbuild) eliminate any uncalled functions during build.

// 1. Standard Named Imports (Fully Tree-Shakeable: ~770 bytes for a single utility)
import { map, filter, groupBy, sumBy } from '@srtv/toondash';

// 2. Or Direct Submodule Imports for micro-footprints
import { parseToon } from '@srtv/toondash/parser';        // ~4.1 KB minified
import { stringifyToon } from '@srtv/toondash/serializer'; // ~2.6 KB minified
import { td, filter, map } from '@srtv/toondash/core';    // Collection utilities

💡 Example Workflow

import td from '@srtv/toondash';

const toonData = `
employees[4, id, name, dept, salary, active]:
  1, Sarah Chen, Engineering, 165000, true
  2, Marcus Vance, Product, 142000, true
  3, Elena Rostova, Engineering, 158000, true
  4, David Kim, Growth, 88000, false
`;

// 1. Fluent Chaining with TOON input & TOON output
const topEngineers = td(toonData)
  .get('employees')
  .filter({ active: true, dept: 'Engineering' })
  .orderBy('salary', 'desc')
  .map((emp) => ({
    name: emp.name,
    annualComp: emp.salary,
    bonus: Math.round(emp.salary * 0.15),
  }))
  .toToon();

console.log(topEngineers);
/* Output:
[2, name, annualComp, bonus]:
  Sarah Chen, 165000, 24750
  Elena Rostova, 158000, 23700
*/

// 2. Direct Function Invocation
const parsed = parseToon(toonData);
const engStaff = filter(parsed.employees, { dept: 'Engineering' });
const totalPayroll = sumBy(engStaff, 'salary'); // 323000

🎯 Explicit Format Parsing

ToonDash offers explicit factory methods for clarity when working with known formats:

import td, { fromJSON, fromToon, from } from '@srtv/toondash';

// Parse JSON explicitly
const jsonUsers = fromJSON('{"users": [{"id": 1, "name": "Alice"}]}')
  .get('users')
  .filter(u => u.id > 0)
  .map('name')
  .value();

// Parse TOON explicitly
const toonUsers = fromToon(`users[2, id, name]:
  1, Alice
  2, Bob`)
  .filter(u => u.id === 1)
  .toJson();

// Wrap native JavaScript objects
const nativeArray = from([1, 2, 3, 4, 5])
  .filter(x => x > 2)
  .map(x => x * 2)
  .value(); // [6, 8, 10]

// Auto-detect JSON or TOON (convenient but explicit methods above are clearer)
const autoDetect = td('{"data": [1, 2, 3]}')
  .get('data')
  .value();

📤 Output in Any Format (JSON, TOON, or Raw Objects)

ToonDash lets you output results in any format:

import { from } from '@srtv/toondash';

const users = [
  { id: 1, name: 'Alice', dept: 'Engineering', score: 95 },
  { id: 2, name: 'Bob', dept: 'Product', score: 87 },
  { id: 3, name: 'Charlie', dept: 'Engineering', score: 92 }
];

const pipeline = from(users)
  .filter(u => u.dept === 'Engineering')
  .sortBy('score', 'desc');

// 1. Output as JSON string (formatted)
const json = pipeline.toJson({ pretty: true });
// {
//   "id": 1,
//   "name": "Alice",
//   "dept": "Engineering",
//   "score": 95
// }

// 2. Output as TOON (40-70% fewer tokens!)
const toon = pipeline.toToon();
// [2, id, name, dept, score]:
//   1, Alice, Engineering, 95
//   3, Charlie, Engineering, 92

// 3. Output as raw JavaScript object/array
const raw = pipeline.value();
// [
//   { id: 1, name: 'Alice', dept: 'Engineering', score: 95 },
//   { id: 3, name: 'Charlie', dept: 'Engineering', score: 92 }
// ]

🛠️ API Reference Summary

Core Functions

Collection & Array Operations

Transformations: map, flatMap, flatten, flattenDeep, chunk, compact, uniq, uniqBy, zip, unzip
Filtering & Slicing: filter, partition, take, drop, slice, head, first, last, initial, tail
Searching & Testing: find, findLast, findIndex, findLastIndex, every, some, includes, size
Grouping & Sorting: groupBy, keyBy, countBy, sortBy, orderBy
Set Operations: difference, intersection, union
Aggregations: sum, sumBy, mean, meanBy, min, minBy, max, maxBy
Randomization: sample, sampleSize, shuffle

Object & Utility Operations

Property Access: get(obj, path, defaultVal), set(obj, path, val), has(obj, path)
Object Manipulation: pick, omit, merge, mapKeys, mapValues, invert, clone, cloneDeep
Predicates & Math: isEqual, isEmpty, isNil, defaultTo, clamp, inRange

🧪 Local Verification Commands

# Type check and lint
npm run lint

# Run all 44+ automated unit & collection operation tests
npm run test

# Build production app and library bundles
npm run build

# Preview static app locally on port 3000
npm run preview

📊 Token Savings Benchmark