mongo-tson 🚀

High-Performance MongoDB to Token-Structured Object Notation (TSON) Serializer for LLMs & RAG Pipelines.

Reduce your LLM token consumption by 30% to 60% when feeding MongoDB and Mongoose query results directly to AI models like Gemini, OpenAI GPT-4o, Claude, and Llama.

💡 Why mongo-tson?

When building Retrieval-Augmented Generation (RAG) or AI Agents powered by MongoDB databases, raw JSON.stringify() outputs create massive token bloat:

• ❌ Repeated Keys: Standard JSON duplicates key names across every item in an array ("name", "email", "created_at").
• ❌ BSON Type Overhead: ObjectId("64f..."), ISODate("..."), Decimal128, and Mongoose __v version flags waste context window space.
• ❌ High API Bills & Latency: Extra tokens increase inference cost and slow down generation speed.

mongo-tson solves this by converting MongoDB documents into TSON (Token-Structured Object Notation) — a lossless columnar format optimized specifically for tokenizers.

📊 Token Savings Benchmark

| Record Count | Standard JSON (Tokens) | TSON Format (Tokens) | Token Savings | Savings % | | :--- | :--- | :--- | :--- | :--- | | 10 Documents | ~1,250 tokens | ~580 tokens | 670 tokens | 53.6% ⚡ | | 50 Documents | ~6,200 tokens | ~2,410 tokens | 3,790 tokens | 61.1% ⚡ | | 200 Documents | ~25,000 tokens | ~9,600 tokens | 15,400 tokens | 61.6% ⚡ |

📦 Installation

npm install mongo-tson
# or
yarn add mongo-tson
# or
pnpm add mongo-tson

⚡ Quick Start

1. Basic Usage with MongoDB / Mongoose

import { mongoToTson, analyzeTokenSavings } from 'mongo-tson';

// Fetch records from MongoDB/Mongoose
const users = await User.find({ active: true }).lean();

// Convert to TSON (and automatically track token savings)
const tsonPromptData = mongoToTson(users, { trackSavings: true });

console.log(tsonPromptData);
/*
Output:
_id: 64f123456789, 64f12345678a
name: Alice, Bob
role: admin, user
active: true, true
createdAt: 2026-08-27T15:45:00.000Z, 2026-08-27T15:46:12.000Z
*/

// Check exact token savings vs JSON
const savings = analyzeTokenSavings(JSON.stringify(users, null, 2), tsonPromptData);
console.log(`Saved ${savings.savingsPercentage}% tokens! (${savings.savedTokens} tokens saved)`);

2. Live Terminal Test & Demo

Run the interactive test suite to verify conversion and token savings:

npm run demo

3. Persistent Token Savings Tracker (SAVINGS_TRACKER.json)

mongo-tson includes a built-in persistent tracker that automatically logs cumulative token savings to SAVINGS_TRACKER.json every time conversion runs with trackSavings: true:

import { getGlobalSavingsTracker } from 'mongo-tson';

const tracker = getGlobalSavingsTracker();
console.log(tracker.getStats());
/*
{
  totalConversions: 42,
  totalJsonTokens: 15400,
  totalTsonTokens: 6200,
  totalSavedTokens: 9200,
  overallSavingsPercentage: 59.74,
  lastUpdated: "2026-08-27T16:08:00.000Z"
}
*/

4. Integration with LLMs (Gemini / OpenAI)

import { mongoToTson } from 'mongo-tson';
import { GoogleGenerativeAI } from '@google/generative-ai';

const ai = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
const model = ai.getGenerativeModel({ model: 'gemini-1.5-pro' });

// Fetch database records
const orders = await Order.find({ status: 'completed' }).limit(50);

// Compress payload for prompt
const tsonContext = mongoToTson(orders, {
  omitKeys: ['__v', 'internalLogs'],
  formatDates: 'iso'
});

const response = await model.generateContent(`
You are an expert data analyst. Based on the following order history in TSON format:

${tsonContext}

Summarize the revenue trends and top purchasing accounts.
`);

console.log(response.response.text());

🛠️ API Reference

mongoToTson(data, options?)

Converts MongoDB documents, query arrays, Mongoose models, or plain objects into TSON.

Options (MongoTsonOptions)

| Option | Type | Default | Description | | :--- | :--- | :--- | :--- | | style | 'tabular' \| 'compact' | 'tabular' | 'tabular' for clean indented headers; 'compact' for pipe-delimited single lines. | | omitKeys | string[] | ['__v'] | Keys to automatically exclude (e.g. ['__v', 'password']). | | includeKeys | string[] | undefined | Exclusive keys to include. | | stripNulls | boolean | false | Strips keys containing null values. | | stripUndefined | boolean | true | Strips keys containing undefined values. | | formatDates | 'iso' \| 'timestamp' \| 'raw' | 'iso' | Date formatting style. | | formatObjectId | 'string' \| 'prefixed' | 'string' | BSON ObjectId format ('64f...' or 'oid:64f...'). | | indentSpaces | number | 2 | Indentation spaces for tabular mode. | | trackSavings | boolean | false | Automatically updates SAVINGS_TRACKER.json with cumulative metrics. |

👤 Author & Contact

Developed with ❤️ by Abhi Asok.

For business inquiries, collaboration, or support:

• 📧 Email: [email protected]
• 📞 Phone / WhatsApp: +91 9142125724
• 💼 LinkedIn: linkedin.com/in/abhi-asok-09439788
• 🐙 GitHub: github.com/abhiasok

📄 License

MIT © Abhi Asok