🔡 BPE Merge Visualizer

A simple CLI tool to visualize step-by-step Byte Pair Encoding (BPE) merge operations using a mock rule set. Great for learning how BPE tokenization works (like in GPT models).

🧪 Usage

After running npm link, you can run the CLI tool with:

bpeviz

You'll be prompted to enter a word:

Enter a word to visualize BPE merging: indivisibility
Step 0: [i] [n] [d] [i] [v] [i] [s] [i] [b] [i] [l] [i] [t] [y]
Step 1: [i] [n] [d] [i] [v] [is] [i] [b] [i] [l] [i] [t] [y]
Step 2: ...

📦 Folder Structure

bpe-merge-visualizer/
├── bin/
│   └── cli.js          # CLI entry file
├── src/
│   └── visualizer.js   # Merge logic
├── package.json
└── README.md

🛠 Features

• Step-by-step visualization of BPE merge rules
• Designed to simulate GPT tokenization logic
• Fully extensible to support real cl100k_base.json or other vocab

🔮 Future Improvements

• Support real BPE vocab files
• Add Web/GUI version
• Export to HTML/SVG

📜 License

MIT

🤖 Inspired by

• OpenAI's tiktoken
• HuggingFace's tokenizers

🙌 Author

Made with ❤️ by MOHD RAZA