boilerstrip
v0.4.1
Published
Learn site boilerplate selectors from a page set and convert HTML to Markdown
Maintainers
jongleberryReadme
boilerstrip
Learn site boilerplate selectors from a set of HTML pages and convert HTML to clean Markdown with the boilerplate stripped.
Given multiple HTML pages from the same website, learn discovers which CSS selectors and HTML snippets are boilerplate (navigation, footers, cookie banners, legal disclaimers, etc.) by finding elements whose text content is stable across pages. The resulting Removals can then be fed into convert or convertMany, which strips the boilerplate and converts the remaining content to Markdown.
Install
npm install boilerstripMigration: v0.1 → v0.2
Breaking changes:
- All HTML inputs are now
Buffer. The previous API acceptedBuffer | string;stringinputs are no longer accepted and will cause a TypeScript type error. PassBuffer.from(html, 'utf8')or read files withreadFileSync(which returns Buffer by default). convertnow processes a singleBufferand returns aConvertResultobject (withcontent,title,lang,meta,link,canonicalUrlfields), not a raw Markdown string.convertManyis a new batched API for converting multiple pages in one call — more efficient than callingconvertin a loop.- Non-UTF-8
Bufferinputs (e.g. windows-1252, shift_jis) now return an explicit error rather than silently replacing invalid bytes.
Usage
import { learn, convert, convertMany } from 'boilerstrip'
// Learn boilerplate from multiple pages of the same site
const removals = await learn([page1Buffer, page2Buffer, page3Buffer])
// Convert a single page
const result = await convert(htmlBuffer, { removals })
console.log(result.content) // Markdown string
console.log(result.title) // <title> text
// Convert many pages in one batched call (more efficient than N convert() calls)
const results = await convertMany([html1, html2, html3], { removals })
results.forEach(r => console.log(r.content))API
learn(pages, options?): Promise<Removals>
Analyzes an array of HTML pages from the same site and returns the discovered boilerplate selectors and snippets. Requires at least 2 pages.
pages—Array<Buffer>— at least 2 HTML pages to analyzeoptions— optionalLearnOptions
convert(html, options?): Promise<ConvertResult>
Strips boilerplate and converts a single HTML page to Markdown.
html—Buffer— HTML page to convertoptions— optionalConvertOptions
convertMany(htmls, options?): Promise<ConvertResult[]>
Batched version of convert. Converts multiple HTML pages in a single N-API call. More efficient than calling convert in a loop when processing many pages.
htmls—Array<Buffer>— HTML pages to convertoptions— optionalConvertOptionsapplied to all pages
ConvertResult
interface ConvertResult {
content: string // cleaned Markdown
title?: string // <title> text
lang?: string // <html lang="...">
canonicalUrl?: string // <link rel="canonical" href="...">
meta: Record<string, string> // <meta name/property> map
link: Record<string, string> // <link rel> map
}ConvertOptions
interface ConvertOptions {
removals?: Removals // from learn()
cssSelectorsToRemove?: string[] // additional CSS selectors to strip
contentSelectors?: string[] // CSS selectors for the main content root
linkTextContentToRemove?: string[] // remove <a>/<button> by visible text
linkHrefsToRemove?: string[] // remove <a> by href prefix (e.g. "javascript:")
linkRelTokensToRemove?: string[] // exclude <link rel="..."> from link map
useTextDensityFilter?: boolean // use text-density scoring to find main content
}Removals
interface Removals {
cssSelectorsToRemove: string[]
htmlToRemove: string[]
}The serializable result of learn. Can be stored and reused across runs.
LearnOptions
interface LearnOptions {
boilerplatePatterns?: string[] // override built-in patterns; [] disables snippet matching
maxSelectorMatchesPerPage?: number // default 20
minSelectorAverageStableRatio?: number // default 0.6
minSelectorPerPageStableRatio?: number // default 0.35
minSnippetTextLength?: number // default 40
maxSnippetTextLength?: number // default 240
}License
MIT