tokemon

A Node.js library for reading streamed JSON.

📍 Introduction

tokemon is an open source Node.js library written in TypeScript for extracting fields from streamed JSON.

When working with LLMs, a common use case is having them respond with JSON, which is then processed in a subsequent step. This is straightforward when the LLM returns its response all at once. However, streamed responses make things more challenging. Because streamed JSON arrives token-by-token, it remains incomplete and invalid until the stream finishes - so calling JSON.parse() isn’t an option. You may still need a way to extract certrain fields from the JSON stream, even before it is fully complete.

In scenarios where responde times are a driving factor, such as chatbots, you want to deliver answers to the client as quickly as possible. That’s where tokemon comes in. It lets you extract specific fields from tokenwise streamed JSON, enabling you to process data early and, for example, stream relevant values back to the client while the LLM is still generating its remaining output tokens.

Note: tokemon can only extract fields that exist at the top level of the streamed JSON. It cannot correctly extract fields nested inside child objects.

📦 Installation

Install via npm:

npm i --save tokemon

🔨 Usage

tokemon can be easily integrated with common LLM libraries such as Ollama, OpenAI, and Gemini. All you have to do is pass the async iterator returned by the LLM to an extractor's extract() method and provide a mapper function that converts the LLM's response to a string.

The following example shows how to use tokemon with Ollama.

import ollama, { type ChatResponse } from 'ollama';
import { StringExtractor } from 'tokemon';

const response = await ollama.chat({
  model: 'llama3.1',
  messages: [
    {
      role: 'system',
      content: `
      You are a question / answer bot.
      Your task is to answer the given question in max. 3 sentences and identify an appropriate topic.
      Respond in JSON format like this:
      {
        "answer": "<answer>",
        "topic": "<topic>"
      }
      `
    },
    { role: 'user', content: 'How does an LLM work?' }
  ],
  stream: true, // Enable streaming
  format: {
    type: 'object',
    properties: {
      answer: { type: 'string' },
      topic: { type: 'string' }
    }
  }
});

const asyncIter = response[Symbol.asyncIterator]();

// Extract 'answer' field
const extractor = new StringExtractor('answer');

// Mapper to convert ChatResponse to string
const mapper = (res: ChatResponse): string => res.message.content;

for await (const token of extractor.extract(asyncIter, mapper)) {
  process.stdout.write(token);
}

The extract method yields every new token of the target field returned by the LLM. So in this example the value of the field answer is written tokenwise to the console as soon as it is part of the LLM's response.

Of course, you're not limited to JSON generated by an LLM. All you need is an async iterator that yields JSON tokens.

You can find more examples here.

📄 API

The following provides an overview of tokemon's API. The core components are its extractors. All extractors extend the BaseExtractor class, which provides shared properties and methods.

You can find the TypeDoc documentation here.

BaseExtractor

Properties

| Name | Description | Type | Default | | ------ | ------------------------------------------ | -------- | ------- | | buffer | Property containing the streamed JSON code | string | '' |

Methods

extractor.extract(iter, mapper);

• Parameters:
  • iter:
    • Type AsyncIterator<T>
    • The async iterator of the LLM stream
  • mapper
    • Type (value: T, index: number) => string
    • The mapper function that converts the LLM message to string
• Returns an AsyncIterable<string> including the extracted tokens

Events

The BaseExtractor class, in turn, extends the EventEmitter class, allowing it to emit events that you can subscribe to.

| Name | Description | | ----------------- | ------------------------------------------ | | FIELD_COMPLETED | Emitted when the target field is completed |

StringExtractor

Properties

| Name | Description | Type | Default | | ----- | --------------------------------------------- | --------------------- | ----------- | | value | Property containing the value of target field | string \| undefined | undefined |

NumberExtractor

Properties

| Name | Description | Type | Default | | ----- | --------------------------------------------- | --------------------- | ----------- | | value | Property containing the value of target field | number \| undefined | undefined |

BooleanExtractor

Properties

| Name | Description | Type | Default | | ----- | --------------------------------------------- | ---------------------- | ----------- | | value | Property containing the value of target field | boolean \| undefined | undefined |

🧩 Contributing

Any contribution is appreciated! See CONTRIBUTING.md

🔑 License

tokemon is released under MIT license.