@silyze/kb-scanner-html

HTML implementation of DocumentScanner<T> for @silyze/kb, using jsdom to extract visible text content and TextScanner for token-based chunking.

Features

• Parses HTML strings and extracts visible text via innerText.
• Automatically includes nested element content in reading order.
• Uses TextScanner to split extracted text into token-based chunks compatible with OpenAI’s tiktoken.
• Fully async via AsyncReadStream.

Installation

npm install @silyze/kb-scanner-html

Usage

import HtmlScanner from "@silyze/kb-scanner-html";

const scanner = new HtmlScanner();

const html = `<h1>Hello World!<div>2</div></h1>`;

async function run() {
  const chunks = await scanner.scan(html).transform().toArray();
  console.log(chunks);
}

run().then();

Output:

["Hello World!2"];

Configuration

HtmlScanner accepts all TextScanner configuration options:

type HtmlScannerConfig = TextScannerConfig;

Examples:

• tokensPerPage – tokens per chunk (default: 512)
• overlap – overlap ratio or count (default: 0.5)
• model – tokenizer model name (default: "text-embedding-3-small")
• encoding – text encoding (default: "utf-8")

How It Works

1. Parses the HTML string using jsdom.
2. Reads innerText from each document child to capture visible text in reading order, including nested elements.
3. Joins all extracted text with newlines.
4. Passes the combined text into TextScanner for token-based chunking.

Example

For:

<h1>
  Hello World!
  <div>2</div>
</h1>
<p>This is an HTML example.</p>

You might get:

["Hello World!2\nThis is an HTML example."];

Longer HTML documents will be chunked according to your configured token size and overlap.