TokenWise

TokenWise is a lightweight utility for preparing JSON before sending it to AI models. It helps reduce payload noise, shrink token usage, and turn structured data into formats that are easier for LLMs to consume.

🧠 What is tokenwise?

tokenwise is a lightweight utility that optimizes your data before sending it to AI models.

Raw JSON is:

• ❌ verbose
• ❌ token-heavy
• ❌ expensive for LLMs

tokenwise helps you:

• 📉 reduce token usage
• ⚡ improve response speed
• 💰 lower API costs

Installation

npm install @sohqureshi/tokenwise

Available on npm.

Quick Usage

import ai, { compact, flatten, toNatural, toTOON } from "@sohqureshi/tokenwise";

const product = {
  product: {
    name: "Wireless Headphones",
    price: 79.99
  }
};

console.log(compact(product));
// {"product":{"name":"Wireless Headphones","price":79.99}}

console.log(flatten(product));
// {
//   "product.name": "Wireless Headphones",
//   "product.price": 79.99
// }

console.log(ai(product).compact().value());
// {"product":{"name":"Wireless Headphones","price":79.99}}

Core Functions

prune()

Removes fields you do not want to send to the model. By default it also removes null, undefined, and empty objects.

import { prune } from "@sohqureshi/tokenwise";

const input = {
  user: { name: "John", age: 28 },
  debug: true,
  internal: { apiKey: "secret" }
};

console.log(prune(input, ["debug", "internal"]));
// { user: { name: "John", age: 28 } }

compact()

Prunes empty/noisy values, then returns minified JSON.

compact({
  product: {
    name: "Wireless Headphones",
    price: 79.99
  }
});
// {"product":{"name":"Wireless Headphones","price":79.99}}

flatten()

Converts nested objects and arrays into one object with dot-notation keys.

flatten({
  policy: {
    claims: [
      { status: "approved", amount: 1200 },
      { status: "pending", amount: 500 }
    ]
  }
});
// {
//   "policy.claims.0.status": "approved",
//   "policy.claims.0.amount": 1200,
//   "policy.claims.1.status": "pending",
//   "policy.claims.1.amount": 500
// }

toNatural()

Converts JSON into readable sentences or numbered pointers.

toNatural([
  {
    user: {
      name: "Alice Johnson",
      email: "[email protected]",
      skills: ["Python", "JavaScript"]
    }
  }
]);
// 1. User Alice Johnson (email: [email protected], Having Python and JavaScript).

Medical and insurance-style data also becomes readable:

toNatural({
  policy: {
    holderName: "Carlos Rivera",
    policyNumber: "HLT-2048",
    claim: {
      status: "under review",
      requestedAmount: 64000
    }
  }
});
// Carlos Rivera has policy HLT-2048. The claim is under review for 64,000.

toNatural() uses semantic templates when the field relationships are unambiguous (for example, a policy holder, policy number, and claim). Unknown JSON shapes use a conservative key-value fallback so the formatter does not invent relationships or facts. For dynamic entity-shaped objects, it can recognize common subject fields such as name, title, label, displayName, entityName, fullName, userName, customerName, ownerName, accountName, companyName, organizationName, teamName, projectName, productName, serviceName, resourceName, deviceName, patientName, taskName, and eventName. These can be paired with state fields such as status, state, condition, stage, role, type, category, priority, phase, mode, availability, outcome, result, health, progress, visibility, access, membership, sentiment, and severity. Matching is case-insensitive and also supports keys that contain these concepts, such as patientName, orderStatus, currentPhase, or incidentSeverity.

toTOON()

Converts JSON into a compact TOON-like text format. Arrays of objects become table-style rows.

toTOON({
  users: [
    { id: 1, name: "Ali" },
    { id: 2, name: "John" }
  ]
});
// users:
//   [2]{id,name}:
//     1,Ali
//     2,John

If later rows contain extra keys, the schema includes them:

toTOON({
  claims: [
    { id: "C-1", status: "approved" },
    { id: "C-2", amount: 500 }
  ]
});
// claims:
//   [2]{id,status,amount}:
//     C-1,approved,
//     C-2,,500

Tested Use Cases

TokenWise currently has coverage for:

• Product JSON minification
• Dot-notation flattening
• Arrays and arrays of objects
• Medical patient and appointment data
• Insurance policy and claim data
• Natural-language user pointers
• TOON table formatting
• Null, undefined, and empty values

Why It Helps

LLMs charge and reason over tokens. Sending raw JSON often includes repeated keys, unnecessary metadata, and formatting whitespace. TokenWise gives you multiple ways to reshape the same data depending on your prompt:

• Use compact() when you need valid JSON with minimal whitespace.
• Use flatten() when retrieval, search, or simple key-value context is better.
• Use toNatural() when the model should read the data like human-friendly notes.
• Use toTOON() when arrays of objects should be shorter than repeated JSON.

CLI

node demo.js

Use --analyze to compare serialized input and output with the model's tiktoken encoding. estimateTokens() and analyze() use exact model-aware counts by default; pass exact: false to opt into the four-characters-per-token heuristic. Use model to select the tokenizer used by tiktoken.

Release Notes

v1.0.11 — 2026-09-13

• Preserve sibling entities when formatting wrapped JSON objects.
• Replace the demo's noisy sample with a meaningful dynamic workspace, incident, and order example.
• Update the browser demo to use the v1.0.11 CDN package.

v1.0.10 — 2026-09-13

• Add conservative contains-based semantic matching for dynamic subject and state keys, including fields such as patientName, orderStatus, and incidentSeverity.
• Expand toNatural() regression coverage for dynamic entities, nested objects, arrays, metadata filtering, and alternate subject fields.
• Update the browser demo to use the v1.0.10 CDN package.

v1.0.8 — 2026-09-06

• Expose exact tokenizer metadata when available (model + encoding), and fall back to a clear, model-aware estimator in browser demos.
• Demo updated to show selected model and expected encoding when the exact tokenizer (tiktoken) is not available in-browser.
• Updated analyze() to surface tokenizer encoding in analysis output so the demo shows "Estimator: model=gpt-4 expected_encoding=cl100k_base" even when using the heuristic fallback.
• Misc: build artifacts updated.

v1.0.9 — 2026-09-13

• Use tiktoken model-aware token counts by default in estimateTokens() and analyze().
• Keep the four-characters-per-token estimate available with exact: false.
• Correctly identify o200k_base for modern OpenAI models such as gpt-4o, gpt-4.1, o1, and o3.
• Update the browser demo to use the v1.0.9 CDN package.
• Make demo analysis actionable by showing signed token and character changes, the selected transformation, and an explanation of whether the result is cheaper or larger.
• Rename the demo's input-only Analyze view to Inspect, showing input size and structure without applying a transformation.

🚀 Roadmap

• [x] CLI support
• [x] NPM Support
• [ ] Streaming support (GB+ data)
• [ ] Schema-aware optimization
• [ ] SaaS API

🛠 Comparison to Existing Tools (Future Section)

Highlight where tokenwise stands out, offering better compacting and token estimation features compared to other libraries.

🤝 Contributing

PRs are welcome,Feel free to open issues or submit PRs, Ideas are welcome for:

• Better token compression strategies
• Multi-model optimization
• Streaming CLI support

note: Explore the CONTRIBUTING.md for more details for the contributons.

📄 License

MIT License

❤️ Support This Project

TokenWise is built to help developers reduce LLM cost and improve efficiency.

If it helps you, consider supporting its development:

💡 Vision

Make AI cheaper and faster by optimizing data before it reaches the model.