you know how every chunking library claims to be fast? yeah, we actually meant it.

memchunk splits text at semantic boundaries (periods, newlines, the usual suspects) and does it stupid fast. we're talking "chunk the entire english wikipedia in 120ms" fast.

want to know how? read the blog post where we nerd out about SIMD instructions and lookup tables.

📦 installation

npm install memchunk

looking for rust or python?

🚀 usage

import { init, chunk } from 'memchunk';

// initialize wasm (required once)
await init();

const text = "Hello world. How are you? I'm fine.\nThanks for asking.";

// with defaults (4KB chunks, split at \n . ?)
for (const slice of chunk(text)) {
    console.log(slice);
}

// with custom size
for (const slice of chunk(text, { size: 1024 })) {
    console.log(slice);
}

// with custom delimiters
for (const slice of chunk(text, { delimiters: ".?!\n" })) {
    console.log(slice);
}

// with multi-byte pattern (e.g., metaspace ▁ for SentencePiece tokenizers)
for (const slice of chunk(text, { pattern: "▁", prefix: true })) {
    console.log(slice);
}

// with consecutive pattern handling (split at START of runs, not middle)
for (const slice of chunk("word   next", { pattern: " ", consecutive: true })) {
    console.log(slice);
}

// with forward fallback (search forward if no pattern in backward window)
for (const slice of chunk(text, { pattern: " ", forwardFallback: true })) {
    console.log(slice);
}

// collect all chunks
const chunks = [...chunk(text)];

pass strings and get strings back. for zero-copy performance with binary data, pass Uint8Array and you'll get Uint8Array views back.

📝 citation

if you use memchunk in your research, please cite it as follows:

@software{memchunk2025,
  author = {Minhas, Bhavnick},
  title = {memchunk: The fastest text chunking library},
  year = {2025},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/chonkie-inc/memchunk}},
}

📄 license

licensed under either of Apache License, Version 2.0 or MIT license at your option.