message-chunker

Safe preparation module for splitting long rich-text (Markdown) messages for delivery through transports with message length limits (Telegram, etc.).

Pipeline: markdown → parser → normalized IR → planner → renderer → typed chunks

Features

• Markdown parsing via markdown-it, normalization to a compact IR
• Two rendering modes: rich-html (safe subset: <b>, <i>, <code>, <pre>, <a>; code blocks preserve language info via class="language-*") and plain-text
• 5-level strategy escalation: preserve → split-blocks → split-blocks-soft → plain-text → forced-plain-text
• Greedy packing (maximal prefix per chunk)
• Budget is checked against the final rendered content (content.length); in rich-html this includes HTML escaping/markup overhead
• Unicode-safe splitting (never breaks surrogate pairs)
• replanTail() for replanning undelivered tail after transport reject
• Deterministic: same input + same transport profile = same plan
• No network requests, no transport SDK dependency

Installation

npm install message-chunker

Requires Node.js >= 18.

This package is ESM-only. Use import / export, not CommonJS require().

Quick start

import { planDelivery } from 'message-chunker';

const plan = planDelivery({
    markdown: '# Hello\n\nThis is a **long** message...',
    preferredMode: 'auto',       // 'auto' | 'rich-html' | 'plain-text'
    strategy: 'preserve',         // starting strategy
    transport: {
        maxTextLength: 4096,
        safeTextBudget: 3600,
        supportsPlainText: true,
        supportsMultipartPlainText: true,
        supportsRichHtml: true,
        countMethod: 'string-length',
    },
});

for (const chunk of plan.chunks) {
    console.log(`[${chunk.index + 1}/${chunk.total}] (${chunk.mode})`);
    console.log(chunk.content);
}

console.log('Strategy used:', plan.diagnostics.usedStrategy);
console.log('Mode used:', plan.diagnostics.usedMode);
console.log('Had forced split:', plan.diagnostics.hadForcedSplit);

Within a single DeliveryPlan, usedStrategy and usedMode apply to the whole plan. Mixed rich-html/plain-text delivery is possible only across separate plans, for example: original plan in rich-html + replanned tail in plain-text.

Replanning after reject

import { planDelivery, replanTail, nextStrategy } from 'message-chunker';

const markdown = '...';
const transport = { /* ... */ };

const plan = planDelivery({ markdown, preferredMode: 'auto', strategy: 'preserve', transport });

// Send chunks sequentially...
// If chunk i is rejected by the transport:
const tail = replanTail({
    markdown,
    previousPlan: plan,
    failedChunkIndex: 2,          // chunk 2 failed
    preferredMode: 'auto',
    nextStrategy: nextStrategy(plan.diagnostics.usedStrategy) || 'forced-plain-text',
    transport,
    rejectReason: 'too-long',     // 'too-long' | 'invalid-markup'
});

// tail.chunks has fresh indices 0..M-1
// Chunks 0..1 from the original plan are considered delivered

replanTail() returns a new, separate plan for the undelivered tail. It may use a different usedStrategy and usedMode from the original plan. This is how mixed-format delivery is supported when, for example, the original rich-html chunk is rejected as invalid-markup.

API

planDelivery(request): DeliveryPlan

Build a delivery plan for a Markdown message.

PlanRequest:

| Field | Type | Description | |---|---|---| | markdown | string | Source Markdown text | | preferredMode | 'auto' \| 'rich-html' \| 'plain-text' | Rendering mode preference | | strategy | SplitStrategy | Starting strategy | | transport | TransportProfile | Transport capabilities |

DeliveryPlan:

| Field | Type | Description | |---|---|---| | chunks | PlannedChunk[] | Ordered chunks ready for delivery | | diagnostics | PlanDiagnostics | Detailed diagnostic information |

PlanDiagnostics

| Field | Type | Description | |---|---|---| | sourceLength | number | Source markdown length | | plainTextLengthEstimate | number | Plain-text length estimate after normalization/rendering | | normalizedBlockCount | number | Number of top-level blocks after normalization | | chunkCount | number | Number of chunks in the plan | | requestedStrategy | SplitStrategy | Strategy requested by the caller | | usedStrategy | SplitStrategy | First strategy that produced a valid plan | | requestedMode | 'auto' \| 'rich-html' \| 'plain-text' | Mode preference requested by the caller | | usedMode | 'rich-html' \| 'plain-text' | Rendering mode actually used for this plan | | hadDegradation | boolean | true if strategy/mode had to degrade or unsupported markdown was simplified | | degradedToPlainText | boolean | true if planning ended up in plain-text after a non-plain-text preference | | hadForcedSplit | boolean | true if at least one actual chunk boundary in this plan used forced Unicode-safe split | | splitBlockTypes | string[] | Unique block types that actually had to be split |

replanTail(request): ReplannedTail

Replan the undelivered tail after a transport reject.

PlannedChunk

| Field | Type | Description | |---|---|---| | index | number | 0-based index in the plan | | total | number | Total number of chunks | | mode | 'rich-html' \| 'plain-text' | Rendering mode used | | content | string | Rendered chunk content | | estimatedLength | number | content.length | | sourceRange | SourceRange | Opaque reference into normalized IR |

TransportProfile

| Field | Type | Description | |---|---|---| | maxTextLength | number | Hard transport limit | | safeTextBudget | number | Safe budget (>= 200, must not exceed maxTextLength), checked against the final rendered chunk content | | supportsPlainText | boolean | Transport accepts plain text | | supportsMultipartPlainText | boolean | Transport accepts multiple plain-text messages | | supportsRichHtml | boolean | Transport accepts rich HTML | | countMethod | 'string-length' | Length counting method |

Helpers

• STRATEGY_LADDER — array of all strategies in escalation order
• nextStrategy(strategy) — returns the next more aggressive strategy, or null
• isAtLeastAsAggressive(a, b) — compares two strategies
• validateTransportProfile(tp) — throws on invalid profile

Strategy escalation

| Strategy | Description | |---|---| | preserve | Keep as single chunk if it fits | | split-blocks | Split at block boundaries (paragraphs, headings, etc.) | | split-blocks-soft | Split within blocks (sentences, punctuation) | | plain-text | Same as split-blocks-soft but in plain-text mode | | forced-plain-text | Last resort: split at \n\n → \n → whitespace → Unicode-safe forced cut |

Planning semantics

• Splitting is based on the maximal prefix that fits the budget, not on a balanced split.
• For rich-html, fit is checked after the final render/escape step, so HTML overhead can move the split point left compared with plain text.
• Within that fitting prefix, the planner prefers softer boundaries according to the current block rule.
• Forced Unicode-safe split is used only when no softer allowed boundary exists inside the fitting prefix.

Reject handling scenarios

The library provides replanning tools but does not hardcode the retry policy — that is the caller's responsibility.

too-long — chunk exceeded the transport limit

Typical caller reaction: lower the budget, raise the strategy, or both.

// Transport rejected chunk 1 as too long → lower budget and escalate strategy
const tail = replanTail({
    markdown,
    previousPlan: plan,
    failedChunkIndex: 1,
    preferredMode: 'auto',
    nextStrategy: nextStrategy(plan.diagnostics.usedStrategy) || 'forced-plain-text',
    transport: { ...transport, safeTextBudget: transport.safeTextBudget - 400 },
    rejectReason: 'too-long',
});

invalid-markup — transport rejected the markup

Typical caller reaction: switch to plain-text mode for the remaining tail.

// Transport rejected rich-html chunk → replan tail as plain-text
const tail = replanTail({
    markdown,
    previousPlan: plan,
    failedChunkIndex: 2,
    preferredMode: 'plain-text',
    nextStrategy: plan.diagnostics.usedStrategy,
    transport,
    rejectReason: 'invalid-markup',
});

This may produce a mixed final delivery:

• already delivered prefix stays rich-html;
• replanned tail goes as plain-text.

That mixed result is expected and supported.

Other transport errors (401, 429, 5xx, network failures)

These are not module-level reject reasons. The integration layer must decide whether to retry sending, abort, or map the error to too-long / invalid-markup before calling replanTail().

Limitations

• Underscore emphasis (_text_, __text__) is intentionally not supported — treated as literal text
• Tables are not supported — they fall through as text paragraphs
• Images become text: alt (src)
• Raw HTML is escaped in rich-html mode, kept literal in plain-text
• Unicode splitting is surrogate-pair safe but not grapheme-cluster safe (ZWJ sequences may be split)

Testing

npm test

Development

npm install
npm test
npm run pack:check

License

MIT