ConvoKit: Flexible Conversation Processing & Export Toolkit

ConvoKit is a TypeScript-based framework for ingesting, normalizing, filtering, sampling, formatting, and exporting chat or conversation data for LLM training and analysis. It provides:

• A provider registry to plug in new data sources (Discord, Slack, custom exports, etc.).
• A plugin registry for formatters, converters, and filters to transform and export data to ChatML, Gemini JSONL, custom context formats, and more.
• A fully configurable, extensible pipeline: ingest → normalize → filter → importance‑score → sample → format → export.

ConvoKit saves you time building data preprocessing pipelines and lets you focus on models and prompts.

Table of Contents

• Key Features
• What It Can & Cannot Do
• Who Should Use It
• Installation
• Quick Start
• Configuration
• CLI Usage
• Provider Registry
  • Built‑in Providers
  • Writing Your Own Provider
• Plugin Registry
  • Formatters
  • Converters
  • Filters
  • Writing Your Own Plugin
• Contributing
• License

Key Features

• Dynamic Provider Loading
  Automatically discover and load data providers from your project’s providers folder.

• Normalized Conversation Format
  All data converges to a ConvoKitConversation interface: metadata + message arrays.

• Context Formatting
  Generate a single, line-delimited training string (CKContext) with options for time‑gaps, new‑conversation markers, and importance scoring.

• Turn‑List Conversion
  Break context into turn lists (CKTurnListConversation) for sampling or LLM‑specific export.

• Weighted Sampling
  Sample by conversation importance to focus on high‑value exchanges.

• Export Plugins
  Export to ChatML JSONL, Gemini JSONL, or add your own converter for other LLM formats.

• Filter Plugins
  Drop unwanted messages (e.g. links‑only, emoji‑only, code‑only) via a simple plugin API.

What It Can & Cannot Do

Can:

• Ingest JSON exports from Discord (via DiscordChatExporter), or any custom source you add via the Provider Registry.
• Normalize and filter conversations by message content, length, or custom rules.
• Score message & conversation importance automatically based on time, length, and frequency.
• Sample highly‑important conversations for training budgets.
• Export to popular LLM chat formats (ChatML, Gemini), or easily extendable.

Cannot:

• Perform LLM inference or model training directly. - Yet ;)
• Resolve references across conversations (thread linking across channels).
• Guarantee perfect import schema for every data source—you may need to write a provider to handle custom formats.
• Handle binary or non‑JSON data without extending a provider to preprocess it.

Who Should Use It

• NLP / ML Engineers preparing chat‑based LLM fine‑tuning or analysis datasets.
• Bot / Chat Service Developers needing to transform raw chat logs into structured training data.
• Researchers studying conversation dynamics or designing importance‑based sampling strategies.
• Community Contributors eager to add support for new platforms or export formats.

Possibly upcomming features

• Personality Generate a deep and comprehensive personality prompt based off your output ck_context
• Fine-tuning Fine-tune models with exported training data (Currently mainly looking at Gemini) (Contributions welcome!)
• Model Testing Test your fine-tuned model via the terminal (Currently mainly looking at Gemini) (Contributions welcome!)
• Unit Tests Adding unit tests would help keep everything maintainable and stable (or so i've heard)

Installation

# Install globally (recommended for CLI use)
npm install -g convokit

# Or install locally in your project
npm install convokit

Quick Start (Using the Library)

import { ConvoKit, loadConfig, getConfig } from 'convokit';
import { config } from 'dotenv';

config();
await loadConfig();

async function run() {
  const ck = new ConvoKit();
  await ck.loadProviders(); // This will load all included providers, and the providers in the LocalProvidersDir if set (in config)
  // We also automatically load all included plugins & the plugins in LocalPluginsDir if set (in config)
  const convoData = await ck.processDataFromProviders();

  const context = await ck.parseToContext({ targetUsers: getConfig().targetUsers });
  await ck.convertToCKTurnList();
  await ck.getWeightedSample(getConfig().sampleSize);
  const chatml = await ck.exportToChatML(getConfig().systemPrompt);
  const gemini = await ck.exportToGemini(getConfig().systemPrompt);
  // Do whatever you want with the outputs
}
run();

Make sure you have set up providers and dir structure first

Configuration

By default, ConvoKit reads convokit.config.json or environment variables - Here is an example config file

{
  "inputDataDirName": "InputData",
  "outputDataDirName": "OutputData",
  "targetUsers": [
    { "providerId": "discord", "id": "YOUR_DISCORD_USER_ID" }
  ],
  "sampleSize": 5000,
  "systemPrompt": "You are a helpful assistant.",
  "minImportanceChat": 120,
  "minImportanceMessage": 100,
  "enableDebugging": false,
  "enablePerformanceStats": false,
  "shouldMergeConsecutiveMessages": true,
  "enableWarnings": true,
  "anonymizeProviderConversationIds": false,
  "localProvidersDir": "LocalProviders",
  "localPluginsDir": "LocalPlugins",
}

| Key | Description | |-----------------------------------------|--------------------------------------------------------------------------------------| | inputDataDirName | Directory containing raw chat exports (relative to project root). | | outputDataDirName | Directory to write formatted outputs. | | targetUsers | JSON array mapping each provider to a target user ID for context generation. | | sampleSize | Number of conversations to sample by importance. | | systemPrompt | System prompt used in ChatML/Gemini exports. | | minImportanceChat (optional) | Minimum average importance score for a conversation (default: 120). | | minImportanceMessage (optional) | Minimum importance score for a single message (default: 100). | | enableDebugging (optional) | Enable or disable debug-level logs. | | enablePerformanceStats (optional) | Enable or disable performance stats (timers). | | shouldMergeConsecutiveMessages (optional)| Merge consecutive messages when converting to CKTurnList. | | enableWarnings (optional) | Toggle the display of warning messages. | | anonymizeProviderConversationIds (optional)| Anonymize provider conversation IDs to protect sensitive data. | | localProviderDirectory (optional)| Directory name of where to load custom providers from. | | localPluginDirectory (optional)| Directory name of where to load custom plugins from. (Contains a folder for each plugin type (formatters, filters, converters)! ) |

Directory Structure

In your convokit.config.json file you set a inputDataDirName, in here you will need to have a directory for each provider. In there you should store the exported data.

Example for use with the Discord provider, with inputDataDirName set to InputData:

convokit/
├── index.ts
├── convokit.config.json
├── ... other files and folders
└── InputData
    └── discord
        └── Direct Messages - fishylunar [000000000000000].json

Note: the filenames of the exported data doesnt matter, but the extension does.

CLI Usage

ConvoKit provides a command-line interface (CLI) for running the processing pipeline without writing TypeScript code. Ensure you have a valid convokit.config.json file in your project root or have set the corresponding environment variables.

Running Commands:

# If installed globally
convokit <command> [options]

# If installed locally, using npx
npx convokit <command> [options]

# Or via package.json script
# "scripts": { "ck": "convokit" }
# npm run ck -- <command> [options]

Common Options:

• -p, --providers <ids>: Specify a comma-separated list of provider IDs (e.g., discord,telegram) to process data from. If omitted, ConvoKit will attempt to use data from all providers found in your inputDataDirName that are registered.
• -o, --output <file>: Specify an output file path to save the results of commands like context or export. If omitted, results are generated but not saved to a file (stats/logs will still be shown).

Commands:

• create-config (alias: cfg): Creates an example convokit.config.json file in the current directory. Run this first if you don't have a config file.

  convokit create-config

• providers: Lists all registered providers (built-in and local) found by ConvoKit, including their ID, name, version, and expected input directory/extension. Useful for verifying provider setup and getting IDs for the --providers option.

  convokit providers

• plugins: Lists all registered plugins (formatters, converters, filters), including built-in and local ones. Shows plugin ID, name, and version. Useful for finding the <converter_id> for the export command.

  convokit plugins

• context: Processes data from specified (or all) providers and generates the CKContext output based on your configuration (targetUsers, importance scores, etc.).

  # Generate context from all providers and save to context.txt
  convokit context -o context.txt
  
  # Generate context using only 'discord' provider data and save
  convokit context --providers discord -o discord_context.txt
  
  # Generate context from all providers and save to context.json including stats
  convokit context -o context.json --stats

• export <converter_id>: Runs the full pipeline: loads data, processes it, generates context, converts to turn list, performs weighted sampling (using sampleSize from config), and finally exports the data using the specified <converter_id>.

  # Export data using the 'chatml' converter, save to chatml_export.jsonl
  convokit export chatml -o chatml_export.jsonl
  
  # Export using 'gemini' converter from 'telegram' provider only, save output
  convokit export gemini --providers telegram -o telegram_gemini.jsonl

Example Workflow:

# 1. Create a config file if you don't have one
convokit create-config
# (Edit convokit.config.json with your settings: input dir, target users, etc.)

# 2. Check which providers are available
convokit providers
# Output might show: ID: discord, ID: telegram

# 3. Check available export formats (converters)
convokit plugins
# Output might show Converters: ID: chatml, ID: gemini

# 4. Run the full export pipeline for ChatML using all providers
convokit export chatml -o training_data.jsonl

# 5. (Alternative) Generate only the CKContext for analysis
convokit context -o analysis_context.json

Provider Registry

ConvoKit discovers providers from providers via ProviderRegistry. Each provider must:

1. Implement ConvoKitProvider with Test() and Convert().
2. Export a static ProviderInfo object.
3. Register itself via ProviderRegistry.register(id, ProviderClass, ProviderInfo).

Built‑in Providers

• Discord (providers/discord.ts): Reads JSON exports from DiscordChatExporter.
• Telegram (providers/telegram.ts): Reads JSON exports from the Telegram Desktop app.

Contributions are more than welcome! <3

Writing Your Own Provider

1. Create /providers/MyPlatform.ts.

To make a local provider, put the MyPlatform.ts file in the LocalProvidersDir you specified in your config. If you are contributing and making a provider to be included in ConvoKit, put it in /providers/MyPlatform.ts

2. Define your data schema, compatibility check, and conversion:

export const ProviderInfo = {
  name: "MyPlatform Exporter",
  description: "Imports MyPlatform chat JSON.",
  version: "1.0.0",
  author: "You",
  InputDataInfo: { directoryName: "MyPlatform", fileExtension: ".json" }
};

export class Provider implements ConvoKitProvider {
  constructor(private raw: any) {}
  Test(): boolean {
    // return true if raw matches your schema
  }
  Convert(): ConvoKitConversation {
    // transform raw → ConvoKitConversation
  }
}

// Self-register
ProviderRegistry.register("myplatform", Provider, ProviderInfo);

3. Place your exports in InputData/MyPlatform/*.json.
4. Run ck.loadProviders() and ck.processDataFromProviders() to include your data.

Plugin Registry

Plugins extend ConvoKit’s pipeline at three points:

1. Formatters (formatters)
2. Converters (converters)
3. Filters (filters)

They self‑register via PluginRegistry.registerFormatter/Converter/Filter().

Formatters

• Context Formatter (id: context): Builds the CKContext string with importance and markers.

Converters

• ChatML Converter (id: chatml): Exports LLM chatml JSONL.
• Gemini Converter (id: gemini): Exports Gemini‑style JSONL.

Filters

• LinkOnlyFilter (id: link-only): Excludes messages that are URLs only.

Writing Your Own Plugin

1. Formatters

   export class MyFormatter implements FormatterPluginClass {
     PluginInfo = { id: "myfmt", name: "...", type: "formatter", version: "1.0.0" };
     apply(data, options) { /* return CKContextResult */ }
   }
   PluginRegistry.registerFormatter(MyFormatter);

2. Converters

   export class MyConverter implements ConverterPluginClass {
     PluginInfo = { id: "myconv", name: "...", type: "converter", version: "1.0.0" };
     async apply(convs, prompt) { /* return string[] */ }
   }
   PluginRegistry.registerConverter(MyConverter);

3. Filters

   export class MyFilter implements FilterPluginClass {
     PluginInfo = { id: "myfilter", name: "...", type: "filter", version: "1.0.0" };
     filterType: 'MUST' | 'MUST_NOT' = 'MUST_NOT';
     apply(content) { /* return boolean */ }
   }
   PluginRegistry.registerFilter(MyFilter);

Contributing

Contributions are very welcome!

• Suggest a feature via GitHub Issues.
• Report bugs or raise PRs to fix them.
• Add new providers (Slack, Teams, custom exports).
• Write plugins for new formats or filters.

License

This project is licensed under the MIT License.
Feel free to use, modify, and distribute as you see fit!