@mdream/js

Installation

These examples use v2. For 2.0.0-beta.1, read the version note in Migrating from v1 first.

# pnpm
pnpm add @mdream/js

# npm
npm install @mdream/js

# yarn
yarn add @mdream/js

Entry Points

| Import | Description | |---|---| | @mdream/js | Tree-shakable Markdown conversion and streaming | | @mdream/js/text | Tree-shakable plain text conversion and streaming | | @mdream/js/html | Tree-shakable safe HTML conversion and streaming | | @mdream/js/clean | clean() cleanup rules for the clean option | | @mdream/js/plugins | Optional plugin factories | | @mdream/js/preset/minimal | Explicit composition of the minimal plugin set | | @mdream/js/negotiate | HTTP content negotiation: shouldServeMarkdown, parseAcceptHeader | | @mdream/js/parse | Low-level HTML parser: parseHtml, parseHtmlStream | | @mdream/js/splitter | Single-pass markdown splitter: htmlToMarkdownSplitChunks, htmlToMarkdownSplitChunksStream | | @mdream/js/llms-txt | llms.txt artifact generation: generateLlmsTxtArtifacts, createLlmsTxtStream |

Migrating from v1

Using a coding agent? Give it the v1 to v2 migration prompt.

This guide covers v2. @mdream/[email protected] uses the declarative plugin object, hooks, and format option. For that release's examples, use the beta.1 README.

In v2, each output format and each plugin is a separate import. Your bundle contains only the code that you import. This guide applies to @mdream/js only. For the Rust engine, see the mdream migration guide. Each converter throws a TypeError for an option that it does not read, such as a typo. In v1, it ignored these options. For a removed option, such as format, hooks, or a plugins object, the error names the fix. The top-level mdream options minimal, frontmatter, isolateMain, tailwind, filter, and extraction also throw. cleanUrls throws too. Pass clean: clean({ urls: true }) with clean from @mdream/js/clean. An option set to undefined does not throw. The splitter also reads its chunk options, such as chunkSize. The blankLines cleanup rule is removed. It had no effect, so delete it.

Output formats

The format option is removed. Import the converter for the format.

| v1 | v2 | |---|---| | htmlToMarkdown(html, { format: 'text' }) | htmlToText(html) from @mdream/js/text | | htmlToMarkdown(html, { format: 'html' }) | htmlToSafeHtml(html) from @mdream/js/html | | streamHtmlToMarkdown(stream, { format: 'text' }) | streamHtmlToText(stream) from @mdream/js/text | | streamHtmlToMarkdown(stream, { format: 'html' }) | streamHtmlToSafeHtml(stream) from @mdream/js/html |

The text and HTML converters accept origin, plugins, and tagOverrides. htmlToText also accepts wrapWidth. Both apply the urls rule from clean. htmlToText also applies emptyImages. They skip the Markdown post-processing pass, as in v1. The CLI --format flag is unchanged. frontmatterPlugin now writes YAML frontmatter to Markdown output only. In v1, safe HTML output started with a YAML block. This also applies to --preset minimal --format html. To read the metadata in any format, use onExtract.

- import { htmlToMarkdown } from '@mdream/js'
- const text = htmlToMarkdown(html, { format: 'text' })
+ import { htmlToText } from '@mdream/js/text'
+ const text = htmlToText(html)

Plugins

The declarative plugins object and the hooks option are removed. plugins is now an array of plugin instances. Import each factory from @mdream/js/plugins.

| v1 | v2 | |---|---| | plugins: { frontmatter: true } | plugins: [frontmatterPlugin()] | | plugins: { frontmatter: fm => {} } | plugins: [frontmatterPlugin({ onExtract: fm => {} })] | | plugins: { frontmatter: { additionalFields, metaFields, onExtract } } | plugins: [frontmatterPlugin({ additionalFields, metaFields, onExtract })] | | plugins: { isolateMain: true } | plugins: [isolateMainPlugin()] | | plugins: { tailwind: true } | plugins: [tailwindPlugin()] | | plugins: { filter: { exclude, include, processChildren } } | plugins: [filterPlugin({ exclude, include, processChildren })] | | plugins: { extraction: { selector: fn } } | plugins: [extractionPlugin({ selector: fn })] (see below) | | plugins: { tagOverrides } | top-level tagOverrides | | hooks: [myPlugin] | plugins: [myPlugin] |

  import { htmlToMarkdown } from '@mdream/js'
+ import { filterPlugin, frontmatterPlugin } from '@mdream/js/plugins'

  htmlToMarkdown(html, {
-   plugins: {
-     frontmatter: true,
-     filter: { exclude: ['nav'] },
-     tagOverrides: { 'x-heading': 'h2' },
-   },
-   hooks: [myPlugin],
+   plugins: [frontmatterPlugin(), filterPlugin({ exclude: ['nav'] }), myPlugin],
+   tagOverrides: { 'x-heading': 'h2' },
  })

Plugins run in array order. In v1, the built-in plugins always ran first, in this order: frontmatter, isolateMain, tailwind, filter, extraction. hooks ran after them. To keep the v1 result, use the same order.

Callback changes

• frontmatterPlugin({ onExtract }) runs once after the conversion finishes, as in v1. It now also runs for streams, when the stream ends.
• onExtract now removes the YAML escape from backslashes. In v1, the title a \ b came back as a \\ b.
• extractionPlugin callbacks run when each matching element closes. In v1, the declarative extraction callbacks ran after the conversion finished.
• extractionPlugin callbacks receive the parsed element and the PluginState. The element does not have selector or tagName. Use element.name for the tag name. Use the object key or a separate callback to identify the selector.

- plugins: {
-   extraction: {
-     'img[alt]': el => images.push({ tag: el.tagName, src: el.attributes.src }),
-   },
- },
+ plugins: [
+   extractionPlugin({
+     'img[alt]': el => images.push({ tag: el.name, src: el.attributes.src }),
+   }),
+ ],

Splitter headings, filter selectors, and hook state

| v1 | v2 | |---|---| | import { TAG_H2, TAG_H3 } from '@mdream/js' | Remove the import. The root export has no TAG_H1 to TAG_H6. | | headersToSplitOn: [TAG_H2, TAG_H3] | headersToSplitOn: [2, 3] | | A tag id number in filterPlugin include or exclude | The tag name, such as filterPlugin({ exclude: ['form'] }) | | Hook state with the converter's internal fields | Hook state typed as PluginState |

headersToSplitOn takes heading levels from 1 to 6. If it gets another value, the splitter throws a TypeError. A stale TAG_H2 (8) fails this way. filterPlugin takes only strings in include and exclude. A tag name compiles to the same fast tag match that a tag id gave. If an entry is not a string, filterPlugin throws a TypeError.

Each hook gets a read-only PluginState with options, outputFormat, and depth. The hook type no longer shows the converter's internal fields, such as buffer and depthMap. To keep data for a node, use node.context. To keep data for one document, use a setup function.

ParseState from @mdream/js/parse no longer has the quote fields that the parser stopped using. These are inSingleQuote, inDoubleQuote, inBacktick, inRawTextQuoteAware, and lastCharWasBackslash. PluginContext no longer has the fields of the removed readability plugin.

Splitter output and timing

The splitter changes from #316 also affect earlier v2 betas. Old chunk IDs and line ranges may identify different text after this upgrade.

1. Rebuild stored chunks and embeddings from the original HTML.
2. Replace the stored chunk text and metadata.loc together.
3. Switch your application to the rebuilt index.

• Overlap starts at a word boundary. Chunk text and boundaries can change.
• A fenced code block stays in one chunk, even if it exceeds chunkSize.
• Chunks omit separators and leading or trailing blank lines, including with clean() or withMinimalPreset().
• With stripHeaders: true, metadata.loc covers the kept content lines, excluding the removed heading.
• htmlToMarkdownSplitChunksStream converts the whole document before yielding its first chunk.

The generator keeps the converted Markdown in memory. Stopping it early skips the remaining chunk work. HTML conversion has already finished. If you need output during conversion, use streamHtmlToMarkdown. It yields Markdown pieces without chunk metadata. The splitter accepts keepSeparator, but drops whitespace between chunks.

Cleanup

The clean option now takes the result of clean() from @mdream/js/clean. The cleanup code is only in your bundle when you import it.

| v1 | v2 | |---|---| | clean: true | clean: clean() | | clean: { urls: true, fragments: true } | clean: clean({ urls: true, fragments: true }) |

  import { htmlToMarkdown } from '@mdream/js'
+ import { clean } from '@mdream/js/clean'

- htmlToMarkdown(html, { clean: true })
+ htmlToMarkdown(html, { clean: clean() })

The rules now act on the links and images that the converter writes, as in the Rust engine. Escaped brackets and code keep their text, and streamHtmlToMarkdown applies the rules too. With fragments, a stream yields all output when the document ends, because a link can point to a later heading.

If a converter gets clean: true or a plain rules object, it throws a TypeError. withMinimalPreset still enables all cleanup by default. To turn it off, pass clean: false.

Plugin instances

You can create plugins once and reuse them for many conversions. Each conversion gets fresh state from the built-in plugins. If your own plugin keeps state for one document, use the setup form of createPlugin. See Plugins with per-document state.

Minimal preset

• Import withMinimalPreset from @mdream/js/preset/minimal. The root export is removed.
• The preset returns plugin instances. Plugins that you pass are added after the preset plugins.
• An appended filterPlugin() adds exclusions. It cannot restore elements excluded by the preset.
• You cannot disable or reconfigure one preset plugin through the preset. To do that, compose the plugins yourself:

import { htmlToMarkdown } from '@mdream/js'
import { clean } from '@mdream/js/clean'
import { filterPlugin, isolateMainPlugin, tailwindPlugin } from '@mdream/js/plugins'

// Minimal preset without frontmatter, with a custom filter
htmlToMarkdown(html, {
  clean: clean(),
  plugins: [isolateMainPlugin(), tailwindPlugin(), filterPlugin({ exclude: ['nav', 'footer'] })],
})

Output changes

<title> text no longer appears in the body, in any format. Use frontmatterPlugin() for Markdown metadata, or its onExtract callback to read the title separately.

Streams now join to the same output as one-shot conversion. Chunk boundaries can change. Join chunks before comparing a complete document. The parser also preserves emoji split between chunks and decodes byte chunks across UTF-8 boundaries. Review saved output for parser, whitespace, plugin, and cleanup fixes from #242, #243, and #241. The converter also removes trailing spaces from link text before empty elements. See #318. Regenerate snapshots from reviewed output; do not depend on individual stream boundaries.

Removed exports

| v1 | v2 | |---|---| | @mdream/js/core | @mdream/js. The root entry is now tree-shakable. | | withMinimalPreset from @mdream/js | withMinimalPreset from @mdream/js/preset/minimal | | extractionCollectorPlugin | extractionPlugin | | BuiltinPlugins type | Plugin[] | | FrontmatterConfig type | FrontmatterPluginOptions from @mdream/js/plugins | | CoreOptions type | MdreamOptions |

API Reference

Each output format has its own entry point. Import only the format and plugins that your application uses.

htmlToMarkdown(html, options?)

Converts an HTML string synchronously.

import { htmlToMarkdown } from '@mdream/js'

const md = htmlToMarkdown('<h1>Hello</h1><p>World</p>')
// # Hello\n\nWorld

htmlToText(html, options?)

Converts HTML to readable plain text.

import { htmlToText } from '@mdream/js/text'

const text = htmlToText('<h1>Hello</h1><p><strong>World</strong></p>')
// Hello\n\nWorld

htmlToSafeHtml(html, options?)

Converts HTML to allowlisted semantic HTML.

import { htmlToSafeHtml } from '@mdream/js/html'

const html = htmlToSafeHtml('<h1>Hello</h1><p><strong>World</strong></p>')
// <h1 id="hello">Hello</h1><p><strong>World</strong></p>

Parameters:

| Parameter | Type | Description | |---|---|---| | html | string | The HTML string to convert | | options | Partial<MdreamOptions> | Optional configuration (see MdreamOptions) |

Returns: string

streamHtmlToMarkdown(htmlStream, options?)

Converts an HTML stream incrementally. Useful for large documents or streaming HTTP responses.

import { streamHtmlToMarkdown } from '@mdream/js'

const stream = streamHtmlToMarkdown(response.body, {
  origin: 'https://example.com',
})

for await (const chunk of stream) {
  process.stdout.write(chunk)
}

Parameters:

| Parameter | Type | Description | |---|---|---| | htmlStream | ReadableStream<Uint8Array \| string> \| null | A web ReadableStream of HTML content | | options | Partial<MdreamOptions> | Optional configuration (see MdreamOptions) |

Returns: AsyncIterable<string>

streamHtmlToText from @mdream/js/text and streamHtmlToSafeHtml from @mdream/js/html take the same parameters.

Optional plugins

Import and compose each plugin explicitly.

import { htmlToMarkdown } from '@mdream/js'
import { filterPlugin } from '@mdream/js/plugins'

const markdown = htmlToMarkdown(html, {
  plugins: [filterPlugin({ exclude: ['nav', 'footer'] })],
})

Options

MdreamOptions

| Option | Type | Default | Description | |---|---|---|---| | origin | string | undefined | Origin URL for resolving relative image paths and internal links | | plugins | Plugin[] | undefined | Explicit conversion plugins, applied in array order | | tagOverrides | Record<string, TagOverride \| string> | undefined | Custom tag output or aliases | | clean | Cleaner | undefined | Cleanup rules from clean() in @mdream/js/clean (see CleanOptions). Streams apply them too. With fragments, a stream yields all output at the end. | | wrapWidth | number | undefined | Hard-wrap prose at this many characters on word boundaries |

Any other key throws a TypeError that names the key.

TagOverride

| Option | Type | Description | |---|---|---| | enter | string | Markdown string to emit when entering the tag | | exit | string | Markdown string to emit when exiting the tag | | spacing | [number, number] | Newlines to add [before, after] the tag | | isInline | boolean | Whether the tag is inline (no block-level spacing) | | isSelfClosing | boolean | Whether the tag is self-closing | | collapsesInnerWhiteSpace | boolean | Whether inner whitespace should be collapsed |

CleanOptions

Cleanup rules. Pass them to clean() from @mdream/js/clean:

import { htmlToMarkdown } from '@mdream/js'
import { clean } from '@mdream/js/clean'

htmlToMarkdown(html, { clean: clean() }) // all rules
htmlToMarkdown(html, { clean: clean({ urls: true, fragments: true }) })

| Option | Type | Default | Description | |---|---|---|---| | urls | boolean | false | Strip tracking query parameters (utm_*, fbclid, gclid, etc.) from URLs | | fragments | boolean | false | Strip fragment-only links that do not match any heading slug in the output | | emptyLinks | boolean | false | Strip links with meaningless hrefs (#, javascript:void(0), data:, vbscript:) and replace with plain text | | redundantLinks | boolean | false | Strip links where text equals URL, with or without a title: [https://x.com](https://x.com) becomes https://x.com | | selfLinkHeadings | boolean | false | Strip fragment links inside headings, such as permalink anchors: ## [Title](#title) becomes ## Title | | emptyImages | boolean | false | Drop images with missing, empty, or whitespace-only alt text | | emptyLinkText | boolean | false | Drop links that produce no visible text: [](url) is removed entirely |

clean() without rules enables all options.

Plugins

createPlugin(plugin)

Factory function for creating type-safe transform plugins. All hooks are optional. Each hook gets the read-only PluginState of the conversion.

import { createPlugin } from '@mdream/js/plugins'

const myPlugin = createPlugin({
  beforeNodeProcess(event, state) {
    // Return { skip: true } to skip this node entirely
  },
  onNodeEnter(element, state) {
    // Return a string to inject markdown at this position
  },
  onNodeExit(element, state) {
    // Return a string to inject markdown at this position
  },
  processAttributes(element, state) {
    // Inspect or modify element attributes
  },
  processTextNode(textNode, state) {
    // Return { content: string, skip: boolean } to transform text
    // Return undefined for no transformation
  },
})

Plugin Hook Reference

| Hook | Parameters | Return Type | Description | |---|---|---|---| | beforeNodeProcess | (event: NodeEvent, state: PluginState) | { skip: boolean } \| void | Called before any node processing. Return { skip: true } to skip the node. | | onNodeEnter | (element: ElementNode, state: PluginState) | string \| void | Called when entering an element. Return a string to inject markdown. | | onNodeExit | (element: ElementNode, state: PluginState) | string \| void | Called when exiting an element. Return a string to inject markdown. | | processAttributes | (element: ElementNode, state: PluginState) | void | Called to inspect or modify element attributes. | | processTextNode | (textNode: TextNode, state: PluginState) | { content: string, skip: boolean } \| void | Called for each text node. Return an object to transform text or skip it. | | onDocumentEnd | (state: PluginState) | void | Called once after the whole document is converted, including for streams. |

PluginState

| Field | Type | Description | |---|---|---| | options | Readonly<EngineOptions> | The options passed to the converter | | outputFormat | 'markdown' \| 'text' \| 'html' | The output that the running converter writes | | depth | number | The nesting depth of the current node. It is 0 before the first node. |

The state is read-only. To keep data for a node, set it on node.context.

Plugins with per-document state

If a plugin keeps state for one document, pass a setup function to createPlugin. Each conversion calls it once and uses the hooks that it returns. Then you can reuse one plugin array for many conversions, including concurrent streams.

import { createPlugin } from '@mdream/js/plugins'

const headingCounter = createPlugin(() => {
  let headings = 0
  return {
    onNodeEnter(element) {
      if (element.name === 'h2')
        headings++
    },
  }
})

The built-in frontmatterPlugin, isolateMainPlugin, and extractionPlugin use this form.

filterPlugin(options)

Filters elements by CSS selectors or tag names. A tag name, such as 'nav', compiles to a fast tag match when you create the plugin. If an entry is not a string, filterPlugin throws a TypeError.

import { filterPlugin } from '@mdream/js/plugins'

// Exclude navigation and footer
const plugin = filterPlugin({
  exclude: ['nav', 'footer', '.sidebar', '#ads'],
})

// Include only specific elements
const plugin2 = filterPlugin({
  include: ['article', 'main'],
  processChildren: true, // default: true
})

| Option | Type | Default | Description | |---|---|---|---| | include | string[] | [] | CSS selectors or tag names for elements to include (all others excluded) | | exclude | string[] | [] | CSS selectors or tag names for elements to exclude | | processChildren | boolean | true | Whether to also process children of matching elements |

frontmatterPlugin(options?)

Extracts metadata from HTML <head> into YAML frontmatter. Collects <title> and <meta> tags.

import { frontmatterPlugin } from '@mdream/js/plugins'

const plugin = frontmatterPlugin({
  additionalFields: { source: 'crawler' },
  metaFields: ['robots', 'viewport'],
  onExtract: (frontmatter) => {
    console.log(frontmatter.title)
  },
})

| Option | Type | Description | |---|---|---| | additionalFields | Record<string, string> | Extra fields to add to the frontmatter | | metaFields | string[] | Meta tag names to extract in addition to the defaults | | onExtract | (frontmatter: Record<string, string>) => void | Receives the structured frontmatter once the conversion finishes. Runs for all formats and for streams. |

Default meta fields extracted: description, keywords, author, date, og:title, og:description, twitter:title, twitter:description.

isolateMainPlugin()

Isolates the main content area of a page using a priority-based strategy:

1. If an explicit <main> element exists (within 5 depth levels), use its content exclusively.
2. Otherwise, find content between the first heading (h1-h6) that is not inside a <header> tag and the first <footer>.
3. The <head> section is always passed through for other plugins (e.g., frontmatter extraction).

import { isolateMainPlugin } from '@mdream/js/plugins'

const plugin = isolateMainPlugin()

tailwindPlugin()

Converts Tailwind utility classes to Markdown formatting. Supports mobile-first responsive breakpoints (sm:, md:, lg:, xl:, 2xl:).

| Tailwind Class | Markdown Output | |---|---| | font-bold, font-semibold, font-black, font-extrabold, font-medium | **text** | | italic, font-italic | *text* | | line-through | ~~text~~ | | hidden, invisible | Element skipped | | absolute, fixed, sticky | Element skipped |

import { tailwindPlugin } from '@mdream/js/plugins'

const plugin = tailwindPlugin()

extractionPlugin(selectors)

Extracts elements that match CSS selectors during conversion. Each callback runs when a matching element closes. It receives the parsed element with its trimmed textContent, and the PluginState. Use element.name for the tag name and element.attributes for the attributes.

import { extractionPlugin } from '@mdream/js/plugins'

const plugin = extractionPlugin({
  'h2': (element, state) => {
    console.log('Heading:', element.textContent)
  },
  'img[alt]': (element, state) => {
    console.log('Image:', element.attributes.src)
  },
})

Presets

withMinimalPreset(options?)

Returns frontmatter, isolate-main, Tailwind, and filter plugin instances. It enables all cleanup rules by default. To turn cleanup off, pass clean: false.

import { htmlToMarkdown } from '@mdream/js'
import { withMinimalPreset } from '@mdream/js/preset/minimal'

const md = htmlToMarkdown(html, withMinimalPreset({
  origin: 'https://example.com',
}))

The minimal preset excludes these elements by default: <form>, <fieldset>, <object>, <embed>, <footer>, <aside>, <iframe>, <input>, <textarea>, <select>, <button>, <nav>.

You can append custom plugins:

const md = htmlToMarkdown(html, withMinimalPreset({
  origin: 'https://example.com',
  clean: clean({ urls: true, fragments: true }),
  plugins: [myPlugin],
}))

Content Negotiation

shouldServeMarkdown(acceptHeader?, secFetchDest?)

Determines if a client prefers Markdown over HTML using HTTP content negotiation.

• Returns true when text/markdown or text/plain has higher quality than text/html in the Accept header.
• If qualities are equal, earlier position wins.
• Bare wildcards (*/*) do not trigger Markdown (prevents breaking OG crawlers).
• sec-fetch-dest: document always returns false (browser navigation).

import { shouldServeMarkdown } from '@mdream/js/negotiate'

export function markdownResponse(request: Request, markdown: string): Response | undefined {
  if (shouldServeMarkdown(
    request.headers.get('accept') ?? undefined,
    request.headers.get('sec-fetch-dest') ?? undefined,
  )) {
    return new Response(markdown, {
      headers: { 'Content-Type': 'text/markdown' },
    })
  }
}

Parameters:

| Parameter | Type | Description | |---|---|---| | acceptHeader | string \| undefined | The HTTP Accept header value | | secFetchDest | string \| undefined | The Sec-Fetch-Dest header value |

Returns: boolean

parseAcceptHeader(accept)

Parses an HTTP Accept header into an ordered list of media types with quality values.

import { parseAcceptHeader } from '@mdream/js/negotiate'

const entries = parseAcceptHeader('text/markdown;q=0.9, text/html')
// [{ type: 'text/markdown', q: 0.9, position: 0 }, { type: 'text/html', q: 1, position: 1 }]

Returns: Array<{ type: string, q: number, position: number }>

Markdown Splitter

htmlToMarkdownSplitChunks(html, options?)

Converts HTML to Markdown and returns an array of chunks. Compatible with LangChain's Document structure.

import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'

const chunks = htmlToMarkdownSplitChunks(html, {
  chunkSize: 1000,
  chunkOverlap: 200,
  origin: 'https://example.com',
})

for (const chunk of chunks) {
  console.log(chunk.content)
  console.log(chunk.metadata.headers) // e.g., { h2: 'Section Title' }
  console.log(chunk.metadata.code) // e.g., 'typescript'
  console.log(chunk.metadata.loc) // e.g., { lines: { from: 1, to: 10 } }
}

Returns: MarkdownChunk[]

htmlToMarkdownSplitChunksStream(html, options?)

Converts the whole document, then yields chunks one at a time. It keeps the converted Markdown in memory, but avoids collecting the chunk array. See Splitter output and timing before upgrading a stored index.

import { htmlToMarkdownSplitChunksStream } from '@mdream/js/splitter'

for (const chunk of htmlToMarkdownSplitChunksStream(html, options)) {
  process.stdout.write(chunk.content)
}

Returns: Generator<MarkdownChunk, void, undefined>

SplitterOptions

Extends EngineOptions with chunking-specific settings.

| Option | Type | Default | Description | |---|---|---|---| | headersToSplitOn | (1 \| 2 \| 3 \| 4 \| 5 \| 6)[] | [2, 3, 4, 5, 6] | Heading levels that start a new chunk, from 1 (<h1>) to 6 (<h6>). Another value throws a TypeError. | | returnEachLine | boolean | false | Return each non-empty line as an individual chunk | | stripHeaders | boolean | true | Strip header lines from chunk content | | chunkSize | number | 1000 | Target chunk size, measured by lengthFunction. Complete code fences can exceed it. | | chunkOverlap | number | 200 | Overlap between chunks for context preservation. Must be less than chunkSize. | | lengthFunction | (text: string) => number | (text) => text.length | Function to measure chunk length. Replace with a token counter for LLM applications. | | keepSeparator | boolean | false | The splitter drops whitespace between chunks for either value. |

MarkdownChunk

| Field | Type | Description | |---|---|---| | content | string | The markdown content of the chunk | | metadata.headers | Record<string, string> \| undefined | Header hierarchy at this chunk position (e.g., { h2: 'API', h3: 'Methods' }) | | metadata.code | string \| undefined | Code block language if chunk contains code | | metadata.loc | { lines: { from: number, to: number } } \| undefined | Inclusive, one-based line range in converted Markdown, excluding stripped headings. |

llms.txt Generation

generateLlmsTxtArtifacts(options)

Generates llms.txt content, optionally including llms-full.txt and individual markdown files.

import { generateLlmsTxtArtifacts } from '@mdream/js/llms-txt'

const result = await generateLlmsTxtArtifacts({
  files: processedPages,
  siteName: 'My Site',
  description: 'A description of my site',
  origin: 'https://example.com',
  generateFull: true,
  generateMarkdown: true,
  sections: [
    {
      title: 'Documentation',
      description: 'API reference and guides',
      links: [
        { title: 'Getting Started', href: '/docs/start', description: 'Quick start guide' },
      ],
    },
  ],
  notes: 'Generated by mdream',
})

// result.llmsTxt         -- index file with links to pages
// result.llmsFullTxt     -- single file with all page content (if generateFull: true)
// result.markdownFiles   -- array of { path, content } (if generateMarkdown: true)
// result.processedFiles  -- the input files passed through

LlmsTxtArtifactsOptions

| Option | Type | Default | Description | |---|---|---|---| | files | ProcessedFile[] | (required) | Array of processed page files | | siteName | string | 'Site' | Site name for the header | | description | string | undefined | Site description (rendered as blockquote) | | origin | string | '' | Origin URL to prepend to relative URLs | | generateFull | boolean | false | Generate llms-full.txt with complete page content | | generateMarkdown | boolean | false | Generate individual markdown files under md/ directory | | outputDir | string | undefined | Output directory (used to compute relative file paths) | | sections | LlmsTxtSection[] | undefined | Custom sections to write before the Pages section | | notes | string \| string[] | undefined | Notes to append at the end |

ProcessedFile

| Field | Type | Description | |---|---|---| | title | string | Page title | | content | string | Markdown content of the page | | url | string | URL path of the page | | filePath | string \| undefined | Optional file path on disk | | metadata | { title?, description?, keywords?, author? } | Optional metadata extracted from the page |

LlmsTxtSection

| Field | Type | Description | |---|---|---| | title | string | Section heading | | description | string \| string[] | Section description (can be multiple paragraphs) | | links | LlmsTxtLink[] | Links in the section |

LlmsTxtLink

| Field | Type | Description | |---|---|---| | title | string | Link title | | href | string | Link URL | | description | string \| undefined | Optional description shown after the link |

createLlmsTxtStream(options)

Creates a WritableStream<ProcessedFile> that generates llms.txt artifacts as pages arrive. It writes full page content to disk and keeps page metadata in memory. When the stream closes, it sorts the llms.txt listing by URL path hierarchy.

import { createLlmsTxtStream } from '@mdream/js/llms-txt'

const stream = createLlmsTxtStream({
  outputDir: './output',
  siteName: 'My Site',
  origin: 'https://example.com',
  generateFull: true,
})

const writer = stream.getWriter()
await writer.write({ title: 'Home', content: '# Home\nWelcome', url: '/' })
await writer.write({ title: 'About', content: '# About\nInfo', url: '/about' })
await writer.close()
// Writes llms.txt and llms-full.txt to ./output/

Returns: WritableStream<ProcessedFile>

Low-Level Parser

parseHtml(html, options?)

Parses HTML into a list of DOM events. Returns the events and any remaining unparsed HTML.

import { parseHtml } from '@mdream/js/parse'

const { events, remainingHtml } = parseHtml('<p>Hello</p>')

Returns: { events: NodeEvent[], remainingHtml: string }

parseHtmlStream(html, state, onEvent)

Streaming parser that calls onEvent for each DOM event. Returns any remaining unparsed HTML (useful for processing partial chunks).

import type { NodeEvent } from '@mdream/js'
import { finalizeParse, parseHtmlStream } from '@mdream/js/parse'

const state = { depthMap: new Uint16Array(1024), depth: 0 }
const onEvent = (event: NodeEvent) => console.log(event)
const chunks = ['<p>Hello ', 'world</p>']
let remaining = ''

for (const chunk of chunks) {
  remaining = parseHtmlStream(remaining + chunk, state, onEvent)
}
// Flush trailing text and close open elements at the end of the input.
finalizeParse(remaining, state, onEvent)

Returns: string (remaining unparsed HTML)

CLI

Reads HTML from stdin and writes the selected format to stdout.

# Basic conversion
curl -s https://example.com | npx @mdream/js

# With origin URL for resolving relative paths
curl -s https://example.com | npx @mdream/js --origin https://example.com

# With minimal preset
curl -s https://example.com | npx @mdream/js --origin https://example.com --preset minimal

# Plain text output
curl -s https://example.com | npx @mdream/js --format text

# HTML output
curl -s https://example.com | npx @mdream/js --format html

CLI Options

| Flag | Description | |---|---| | --origin <url> | Origin URL for resolving relative image paths and links | | --preset <preset> | Conversion preset. Currently supports: minimal | | --wrap-width <n> | Hard-wrap prose at n characters | | --format <format> | Output format: markdown, text, html | | --text | Alias for --format text | | -v, --version | Show version number | | -h, --help | Show help |

If you pass an unknown flag, a flag without a value, or an unknown preset, the CLI writes one line to stderr and exits with code 1.

Usage Examples

Custom Tag Overrides

Map custom HTML elements to standard Markdown behavior:

import { htmlToMarkdown } from '@mdream/js'

const md = htmlToMarkdown('<x-heading>Title</x-heading>', {
  tagOverrides: {
    // String alias: make <x-heading> behave like <h2>
    'x-heading': 'h2',

    // Object override: custom enter/exit strings
    'callout': {
      enter: '> **Note:** ',
      exit: '\n',
      spacing: [2, 2],
    },
  },
})

Extraction

Extract data from elements during conversion:

import { htmlToMarkdown } from '@mdream/js'
import { extractionPlugin } from '@mdream/js/plugins'

const images: { src: string, alt: string }[] = []

const md = htmlToMarkdown(html, {
  plugins: [
    extractionPlugin({
      'img[alt]': (element) => {
        images.push({
          src: element.attributes.src,
          alt: element.attributes.alt,
        })
      },
    }),
  ],
})

console.log('Found images:', images)

Frontmatter with Callback

Receive structured frontmatter data for further processing:

import { htmlToMarkdown } from '@mdream/js'
import { frontmatterPlugin } from '@mdream/js/plugins'

let metadata: Record<string, string> = {}

const md = htmlToMarkdown(html, {
  plugins: [
    frontmatterPlugin({
      additionalFields: { source: 'crawler' },
      metaFields: ['robots'],
      onExtract: (fm) => {
        metadata = fm
      },
    }),
  ],
})

console.log('Title:', metadata.title)
console.log('Description:', metadata.description)

Composable Plugin for Content Filtering

import { htmlToMarkdown } from '@mdream/js'
import { createPlugin } from '@mdream/js/plugins'

const md = htmlToMarkdown(html, {
  plugins: [
    createPlugin({
      beforeNodeProcess(event) {
        const { node } = event
        if (node.type === 1 /* ELEMENT_NODE */) {
          const el = node as any
          // Skip ads and promotional content
          if (el.attributes?.class?.includes('ad')
            || el.attributes?.id?.includes('promo')) {
            return { skip: true }
          }
        }
      },
      processTextNode(textNode) {
        // Uppercase all TODO markers
        if (textNode.value.includes('TODO')) {
          return { content: textNode.value.toUpperCase(), skip: false }
        }
      },
    }),
  ],
})

Token-Based Chunking for LLMs

import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
import { encoding_for_model } from 'tiktoken'

const enc = encoding_for_model('gpt-4')

const chunks = htmlToMarkdownSplitChunks(html, {
  chunkSize: 512,
  chunkOverlap: 64,
  lengthFunction: text => enc.encode(text).length,
  origin: 'https://example.com',
})

Exported Types

import type {
  CleanOptions,
  ElementNode,
  EngineOptions,
  ExtractedElement,
  MarkdownChunk,
  MdreamOptions,
  Node,
  NodeEvent,
  OutputFormat,
  Plugin,
  PluginContext,
  PluginState,
  SplitterOptions,
  TagOverride,
  TextNode,
  TransformPlugin,
} from '@mdream/js'

Exported Constants

import {
  ELEMENT_NODE, // Node type for HTML elements (1)
  NodeEventEnter, // Event type for entering a node (0)
  NodeEventExit, // Event type for exiting a node (1)
  TEXT_NODE, // Node type for text content (3)
} from '@mdream/js'