@tlibnx/tokenizer-deepseek_v4_flash

DeepSeek-V4-Flash tokenizer for NodeJS/Browser, powered by @lenml/tokenizers.

Features

• DeepSeek-V4-Flash tokenizer with 128,256 vocabulary size
• Support for 1M token context length
• MoE (Mixture of Experts) architecture compatible
• Support for Node.js and browser environments
• Built-in special tokens (BOS, EOS, PAD)
• Full Hugging Face tokenizer compatibility

Installation

npm install @tlibnx/tokenizer-deepseek_v4_flash

Usage

import { fromPreTrained } from "@tlibnx/tokenizer-deepseek_v4_flash";

// Initialize tokenizer
const tokenizer = await fromPreTrained();

// Encode text
const encoding = tokenizer.encode("Hello, world!");
console.log(encoding.ids);

// Decode tokens
const text = tokenizer.decode(encoding.ids);
console.log(text);

Model Details

• Model: DeepSeek-V4-Flash
• Vocabulary Size: 128,256
• Max Context Length: 1,048,576 (1M)
• Tokenizer Class: PreTrainedTokenizerFast

License

Apache-2.0