@bureau-of-weights-and-measures/content-distillation

Deterministic HTML → Markdown conversion for AI agents.

Extracts the primary content signal from web pages and converts it into clean Markdown optimized for LLM consumption.

Part of the Bureau of Weights and Measures ecosystem.

Install

Run:

npm install @bureau-of-weights-and-measures/content-distillation

Usage

import { distill } from "@bureau-of-weights-and-measures/content-distillation"

const result = await distill(document)

Overview

This package extracts the primary content signal from web pages and converts it into structured Markdown suitable for AI agents, crawlers, and LLM systems.

Instead of parsing noisy HTML, agents can request Markdown directly.

Features:

• deterministic HTML → Markdown conversion
• WASM-powered pipeline (Rust → wasm32-unknown-unknown)
• browser, worker, and edge compatible
• optional ONNX intent classification — local inference, no external services
• fully inspectable

Agent access via Accept header

Drop in one script tag.

Any HTTP client that sends:

Accept: text/markdown

will receive clean Markdown instead of HTML.

Human visitors in a browser see nothing different.

<script type="module" src="./distill.js" data-wasm-url="./dist/wasm/distill_core_bg.wasm" data-onnx-url="./models/intent.onnx" data-workers="2"> </script>

Example:

curl https://yoursite.com/page
 -H "Accept: text/markdown"

Response headers:

Content-Type: text/markdown
X-Markdown-Tokens: 465
X-Conversion-Ms: 3
X-Actions: 1

The response body contains the distilled Markdown representation of the page.

Example output

Article Title

This is the primary content of the page...

Section

More text...

Bureau ecosystem

This package implements the content signal detection layer within the Bureau of Weights and Measures measurement architecture.

It is the first primitive in a growing set of standards and instruments for agentic web measurement, including:

• agent discovery
• bot detection
• content permission signals
• content freshness
• interaction traces
• automation scoring

Notes

• Designed for AI agents, crawlers, and LLM systems.
• Fully deterministic — repeated distillation of the same page produces identical Markdown.
• Works in browser, Service Worker, and edge environments.
• Optional ONNX intent classifier enables local semantic understanding of content.