gobbet

Boundary-aware text chunking with overlap for LLM/embedding pipelines — respects paragraphs and sentences, pluggable size metric.

The problem

Feeding documents to LLMs or embedding models requires splitting text into size-constrained chunks. Naive splitters that cut every N characters break sentences mid-word and lose meaning at chunk boundaries, reducing the quality of downstream processing.

Effective chunking needs to respect text boundaries (paragraphs → sentences → words) while maintaining context through overlap. Most solutions either drag in heavy framework dependencies or don't handle both boundaries and overlap well.

Install

npm install gobbet
# or
pnpm add gobbet
# or
yarn add gobbet

Use

Quick start with default character counting:

import gobbet from "gobbet";

const text = "Paragraph one. Paragraph two. Paragraph three.";
const chunks = gobbet(text, { size: 30, overlap: 5 });
// Result: chunks split on sentence boundaries, with 5-char overlap

Realistic example with overlap and custom token measure:

import gobbet from "gobbet";

const document = `Gobbet handles boundary-aware chunking for LLM pipelines.
It respects paragraph and sentence boundaries automatically.

With overlap, context is preserved across chunk boundaries.`;

// Chunk by approximate tokens (4 chars ≈ 1 token)
const tokenMeasure = (s: string) => Math.ceil(s.length / 4);
const chunks = gobbet(document, {
  size: 50,           // 50 tokens max per chunk
  overlap: 10,        // 10 tokens overlap between chunks
  measure: tokenMeasure,
  separators: ["\n\n", "\n", ". ", " ", ""]
});

// Each chunk satisfies: tokenMeasure(chunk.text) <= 50
// Chunks prefer paragraph breaks, then sentence breaks, then word breaks

API

gobbet(input, options)

Main function that splits text into boundary-aware chunks with optional overlap.

Parameters:

• input: string — The text to chunk. Empty string returns empty array.
• options: GobbetOptions — Configuration object

Returns: Chunk[] — Array of chunks sorted by position in original text.

Throws: RangeError — If size <= 0 or overlap >= size.

GobbetOptions

interface GobbetOptions {
  size: number;              // Maximum chunk size (must be > 0)
  overlap?: number;          // Overlap between chunks (default: 0, must be < size)
  measure?: (s: string) => number;  // Size metric (default: s.length)
  separators?: string[];     // Boundary preferences (default: ["\n\n", "\n", ". ", " ", ""])
}

Chunk

interface Chunk {
  text: string;   // Chunk content (includes overlap for all but first chunk)
  start: number;  // Character offset in original input (non-overlap content)
  end: number;    // Exclusive end offset (non-overlap content)
}

Non-goals

gobbet intentionally does NOT provide:

• Markdown/HTML awareness: Does not understand document structure. Use a markdown parser first if needed.
• Semantic chunking: Does not analyze content meaning. For embedding-based chunking, post-process with vector similarity.
• Real tokenization: Uses pluggable measure functions instead. Pair with a tokenizer for accurate token counts.
• Language detection: Treats all text uniformly. Handle language-specific needs in your measure function.

Composability example for markdown:

// First extract markdown sections, then chunk each
const sections = markdown.split(/^##\s+/m);
for (const section of sections) {
  const chunks = gobbet(section, { size: 100, overlap: 20 });
  // Process chunks...
}

TypeScript

gobbet is written in TypeScript with full type definitions. Import types for full type safety:

import gobbet, { type GobbetOptions, type Chunk } from "gobbet";

const options: GobbetOptions = { size: 100, overlap: 10 };
const chunks: Chunk[] = gobbet(input, options);

Related Packages

Caching & Concurrency:

• @azghr/filterkit — Framework-agnostic, type-safe filtering for TypeScript
• @azghr/singlet — Deduplicate concurrent async calls
• staleness — Stale-while-revalidate caching for async functions

Text Processing:

• @azghr/shorn — Truncate strings by byte budget without breaking graphemes
• seriatim — Sequential processing utilities

HTTP & Network:

• forbear — Read server rate-limit instructions from HTTP responses
• forestall — Delay execution until a condition is met
• obviate — Render operations unnecessary through caching

System & Process:

• quiesce — Ordered, timeboxed graceful shutdown for Node
• sortition — Deterministic percentage rollouts and A/B bucketing
• stanch — Stop flows or operations based on conditions

Utilities:

• expunge — Remove or exclude items from collections
• occlude — Hide or mask data and functionality
• placemark — Geographic location and mapping utilities
• specie — Currency and financial calculations

License

MIT