@sdxc/distill

Distill the article out of a web page: fetch under bounds, score the candidates, sanitize what is left.

A page is not a document. It is an article wrapped in navigation, a share rail, a comment thread and three related-post blocks, and deciding which subtree is the article is a scoring heuristic rather than a question the markup answers. This package does that scoring, and it treats the page it scored as what it is — untrusted markup from an origin nobody vetted — so what comes back out is sanitized before any caller can render it.

Installation

npm add @sdxc/distill

Every answer is a Result, so install @sdxc/result alongside this package to read one.

Usage

import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

let article = await distill(post.url, {
	userAgent: "MyApp/1.0 (+https://myapp.example/about)",
});

if (isFailure(article)) return excerptAnd(post.url, article.error.outcome);

render(article.data.html, article.data.title, article.data.byline);

Honouring the origin's robots.txt, with the outcome retrieved (and cached for its lifetimeMs) through @sdxc/robots:

import { distill } from "@sdxc/distill";
import { fetchRobots } from "@sdxc/robots/fetch";

let userAgent = "MyApp/1.0 (+https://myapp.example/about)";
let robots = await fetchRobots(post.url, { userAgent });
let article = await distill(post.url, { userAgent, robots });

Already holding the markup — from a fixture, or from a response somebody else retrieved:

import { distillFrom } from "@sdxc/distill";

let article = distillFrom(source, "https://example.com/post");

API

distill(url, options)

Retrieves a page and reads the article out of it, answering Result<Distill.Retrieved, DistillError>.

options.userAgent is required: a publisher who wants to refuse should be able to tell who is asking, and only the caller knows what to call itself. options.robots takes the RobotsFetch.Outcome that @sdxc/robots/fetch's fetchRobots answered for the origin, and a path it disallows is refused before any request goes out: a parsed file is evaluated for the agent, an unavailable one (a 4xx) permits everything, and an unreachable one (a 5xx, a 429 or a network failure) refuses everything. Omitting it consults nothing. maxBytes, maxRedirects, timeoutMs and signal move the four bounds below.

distillFrom(source, url)

The same scoring, sanitization and metadata over markup in hand, answering Result<Distill.Article, DistillEmptyError>. url is what every relative URL in the markup resolves against, so it is the address the page was actually served from.

addressable(url)

Whether an address is somewhere this package is willing to go, before any request is made, answering Result<URL, DistillRefusedError>. It applies the same host rule as every request and redirect hop distill makes, so a caller can refuse a link before queueing it.

Outcomes

Three errors, each carrying an outcome a caller renders copy from.

| Error | outcome | What happened | | --------------------- | --------- | ------------------------------------------------------------- | | DistillRefusedError | refused | The site said no, or the address is one this will not ask for | | DistillLimitError | timeout | Time, bytes or hops ran out | | DistillEmptyError | empty | The page arrived carrying no article |

Distill.Retrieved adds bytes, and mayCache — false for a response whose X-Robots-Tag carries noarchive for every agent or for the one asking, which is the exact name for asking not to be kept.

The four bounds

Exported as MAX_BYTES, MAX_REDIRECTS and TIMEOUT_MS, and moved per call through options.

• HTTP(S) only, checked before the request and again on every redirect hop, refusing a URL that carries credentials, any literal IP address, a single-label name, and a name under a reserved suffix such as localhost, local, internal or test.
• Five redirects, walked manually with redirect: "manual", which is what gives the chain a length it can exceed and makes the final URL a fact this package tracked.
• Two megabytes, counted off the stream and abandoned mid-body, so a response lying about its length is refused by the same bytes as one that is honest.
• Eight seconds for the whole chain and the body read after it, because somebody is waiting on this one. A signal the caller passes runs alongside the deadline.

The request carries no cookies, no credentials and no header naming whoever asked for it.

Sanitization

distill and distillFrom both sanitize before they answer, so no consumer can forget to. An allow-list decides what survives: prose, lists, tables, figures, links and images, with href and src restricted to http:, https: and mailto:, every relative URL resolved against the article's own address, and every surviving image carrying referrerpolicy="no-referrer" and loading="lazy". Everything else — on* handlers, style, class, id, scripts, iframes and forms — is gone.

Sanitization alone leaves the publisher able to see an address when an image loads. Closing that needs an image proxy, which belongs to whoever is rendering rather than here.

Pattern: Caching What Comes Back

Nothing about a distilled article is about who asked for it, so the URL is the whole key and one distillation serves everybody who opens the same link. Keep a failure too, for far less time than a success: a blocked site asked once an hour is politer than one asked on every open, and mayCache is a page asking not to be kept at all.

import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

const DAY_MS = 86_400_000;

/** Whatever store you already have, keyed by the article's URL. */
interface ArticleStore {
	read(url: string): Promise<{ html: string | null } | null>;
	write(url: string, entry: { html: string | null }, options: { ttl: number }): Promise<void>;
}

async function articleFor(url: string, cache: ArticleStore): Promise<string | null> {
	let hit = await cache.read(url);
	if (hit !== null) return hit.html;

	let article = await distill(url, { userAgent: "MyApp/1.0 (+https://myapp.example/about)" });

	if (isFailure(article)) {
		// An hour, so a site that refuses today is asked again tomorrow rather than hourly.
		await cache.write(url, { html: null }, { ttl: DAY_MS / 24 });
		return null;
	}

	// A page carrying `X-Robots-Tag: noarchive` is answered, never stored.
	if (article.data.mayCache) {
		await cache.write(url, { html: article.data.html }, { ttl: DAY_MS * 7 });
	}

	return article.data.html;
}

Pattern: Deciding The Article Was Worth It

A teaser, a consent interstitial and an error page are all pages that parse, score and sanitize perfectly well while carrying nothing the reader did not already have. Comparing chars against the excerpt already in hand is the one predicate that covers all three.

import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

async function betterThanExcerpt(url: string, excerpt: string): Promise<string | null> {
	let article = await distill(url, { userAgent: "MyApp/1.0 (+https://myapp.example/about)" });
	if (isFailure(article)) return null;
	return article.data.chars > excerpt.length ? article.data.html : null;
}

Versioning

Releases are dated rather than semantic. A version is the UTC date it was published, written YYYY.M.D, so 2026.9.4 is the release from 4 September 2026. At most one release goes out per day.

Those numbers say when, not what: a later date means a later release and carries no compatibility promise. Any release may change or remove an export.

Depend on one exact date, and move it when you are ready to take the change:

{
	"dependencies": {
		"@sdxc/distill": "2026.9.4"
	}
}

A caret or tilde range reads the date as major, minor and patch, so it accepts every later release in the same year. An exact version keeps the upgrade yours to schedule.

License

MIT

Author

Sergio Xalambrí