hazo_scrape

Generic source-agnostic web scraping engine with a network-free parse core.

hazo_scrape/parse maps HTML/IR table columns to logical fields by matching header text, not column position — so it survives a source reordering or adding columns. It has no network dependency (no fetch, no crawler) and no domain knowledge (no currency, no franking rules): you hand it HTML you already fetched plus a keyword map describing the fields you want, and it tells you which header(s) matched each field, flags anything ambiguous, and parses each matched cell's raw text into a number, an ISO date, or trimmed text. Because it's pure and network-free, it's fully unit-testable against static HTML fixtures.

Installation

npm install hazo_scrape

Quick start

import { extractTable, mapColumns, mapRows, type ColumnKeywords } from 'hazo_scrape/parse';

const html = `
  <table>
    <tr><th>Ex-Dividend Date</th><th>Amount</th><th>Franking</th></tr>
    <tr><td>5 Mar 2026</td><td>$0.45</td><td>100%</td></tr>
  </table>
`;

// 1. Pull the best-guess data table out of the page (or pass { select } to
//    target a specific element).
const table = extractTable(html); // { headers, rows, warning? }

// 2. Declare the fields you care about and how to recognise their header.
const keywords: ColumnKeywords = {
  exDate: { keywords: ['ex-dividend', 'ex date'], type: 'date' },
  amount: { keywords: ['amount', 'dividend'], type: 'number' },
  franking: { keywords: ['franking'], type: 'text' },
};

// 3. Map headers -> candidate columns. dateGuard keeps a "date"-looking
//    header (e.g. "Ex-Dividend Date") from also matching a non-date key
//    (e.g. "amount", even though the header contains "dividend").
const map = mapColumns(table.headers, keywords, { dateGuard: true, required: ['amount'] });

if (!map.ok) {
  throw new Error(`Missing required column(s): ${map.unmatched.join(', ')}`);
}
if (map.ambiguous.length > 0) {
  console.warn('Ambiguous columns, caller must disambiguate:', map.ambiguous);
}

// 4. Join the map back onto the row data and parse each matched cell.
const rows = mapRows(table, map, keywords);
console.log(rows[0].amount[0].value); // 0.45
console.log(rows[0].exDate[0].value); // "2026-03-05"

API

extractTable(html, opts?): { headers: string[]; rows: string[][]; warning?: string }

extractTable(html: string, opts?: {
  select?: string;
  grid?: { container?: string; row: string; cell: string; headerRow?: string };
  combine?: { headerSelector: string; dropLabelRows?: boolean };
  cellIgnoreSelectors?: string[];
}): ExtractTableResult

Parses network-free HTML and returns the best-guess data table as raw header and row strings. With no options a scoring heuristic picks the most data-table-like <table> on the page. A warning is returned (never thrown) when no table-like element is found, or when a selector matches nothing.

Precedence: grid > combine > select.

• select — a CSS selector targeting a specific <table> instead of letting the scoring heuristic choose.

• grid — read a "table" built from styled <div>s (CSS grid/flex) rather than a real <table>; common on IR pages whose real table is JS-rendered. row/cell are required; container scopes the search (when it matches several grids, the one with the most data rows wins); headerRow identifies the header among the row matches (rows matching it are never data). Omit headerRow — or give one that matches nothing — and the first row is used as the header and excluded from the data.

• combine — reconstruct one logical table from a header-only <table> followed by one <table> per section (e.g. per year), a common Computershare-style layout. Headers come from headerSelector's first row; data rows are gathered from that table plus each immediately-following sibling, stopping at the first sibling that is not a <table> or whose column count differs — so an unrelated later section can't leak in. Set dropLabelRows: true to drop bare section-divider rows (only the first cell non-empty, e.g. a lone 2025).

• cellIgnoreSelectors — CSS selectors whose matching descendants are removed from every cell and header before its text is read, e.g. ['sup'] so 100<sup>4</sup> reads as 100, not 1004. Off by default (cells are read verbatim); applies to all three modes; the removal happens on a per-cell clone and never mutates the document.

Row-header columns. Data cells are read as th, td, so an accessible table whose first column is a <th scope="row"> label stays column-aligned. When the header row omits the matching corner cell, the leading row-header cell(s) are dropped so the row still lines up with the headers.

mapColumns(headers: string[], keywords: ColumnKeywords, opts?: { dateGuard?: boolean; required?: string[] }): ColumnMap

Matches each header against every key's keywords (case-insensitive substring match) and returns all matching headers per key — never just the first. opts.dateGuard drops date-looking headers from any key whose type !== 'date'. opts.required lists keys that must have at least one candidate for map.ok to be true.

mapRows(table, map, keywords, opts?): MappedRow[]

mapRows(
  table: { headers: string[]; rows: string[][] },
  map: ColumnMap,
  keywords: ColumnKeywords,
  opts?: { dateFormats?: string[]; dateExtractLeading?: boolean; dateYearPivot?: number },
): MappedRow[]

Joins a ColumnMap back onto table.rows, producing one MappedRow per data row. Each matched key holds an array of Cells (one per candidate header), each parsed according to that key's declared type. opts.dateFormats, opts.dateExtractLeading and opts.dateYearPivot are forwarded verbatim to parseDate's opts.formats, opts.extractLeading and opts.yearPivot for every date-typed cell; all three default off, so omitting opts preserves the exact prior behaviour.

parseNumber(raw: string): number | null

Strips currency symbols, 3-letter currency codes, thousands-separator commas, and %, then parses what remains as a number. Returns null when nothing parseable is left. Does not convert cents to dollars — domain scaling is the caller's job.

parseDate(raw: string, opts?: { formats?: string[]; extractLeading?: boolean; yearPivot?: number }): string | null

Parses a handful of common date text formats (ISO, D Mon YYYY, Mon D, YYYY, and D/M/Y slash dates) into an ISO yyyy-mm-dd string. Returns null — never a guess — for anything unrecognised or genuinely ambiguous (e.g. 05/03/2026 with no opts.formats hint and both components <= 12).

opts.extractLeading (default off) accepts a date at the start of the cell even when trailing text follows it — parseDate('25/06/2024 - special dividend', { formats: ['DD/MM/YYYY'], extractLeading: true }) → '2024-06-25'. Only the leading date token is read; the trailing text is ignored, never parsed. Leading junk is still rejected (the date must be the first token) and the date itself is still matched exactly and disambiguated by the same rules — so the "never guess" contract holds. For real IR pages that staple a label onto a date cell.

opts.yearPivot (default off) enables 2-digit years, which are otherwise rejected (null) rather than guessed. It is the base of a sliding 100-year window: parseDate('10 Mar 26', { yearPivot: 2000 }) → '2026-03-10', while yearPivot: 1950 reads 26→2026 and 99→1999. Applies to the named-month formats only (D Mon YY, Mon D, YY); slash dates stay 4-digit-only, since a 2-digit slash year compounds day/month and century ambiguity.

Types

type ColumnSpec = { keywords: string[]; type: 'number' | 'date' | 'text' };
type ColumnKeywords = Record<string, ColumnSpec>;
type Candidate = { header: string; index: number };
type ColumnMap = {
  ok: boolean;                          // false when any required key has zero candidates
  matched: Record<string, Candidate[]>; // every match, in header order, one entry per key
  unmatched: string[];                  // headers that matched no key at all
  ambiguous: string[];                  // keys with 2+ candidates — reported, never resolved
  headers: string[];
};
type Cell = { header: string; index: number; raw: string; value: number | string | null };
type MappedRow = Record<string, Cell[]>;

Design

• One-to-many, never resolved. Every header matching a key is returned as a candidate; if a table has both "Dividend (USD)" and "Dividend (AUD)", both show up under amount and the key is reported in ambiguous. The engine will not guess which one is "right" — the caller (e.g. an /aud/i rule for a specific stock) makes that call.
• dateGuard is generic, not hardcoded. Rather than special-casing header text like "exDate", it uses each key's declared type: any header that reads as a date is only eligible for keys typed 'date'.
• Required, not scored. mapColumns doesn't guess at "good enough" — the caller declares which keys are required, and ok reflects exactly that.
• Domain interpretation is the caller's job. Cents-to-dollars scaling, "Unfranked" -> 0, currency selection, and similar business rules are deliberately outside this engine. parseNumber and parseDate return the literal parsed value (or null) and nothing else.

Fetch layer (hazo_scrape/fetch)

Everything above is hazo_scrape/parse — network-free and safe to bundle for the client. The fetch layer is the opposite: it's server-only. Import it from hazo_scrape/fetch (or the root hazo_scrape entry, which re-exports it) inside server code only — it pulls in undici, hazo_secure, node:fs, and node:crypto, and will break a client bundle if imported there.

import { scrapeTable, type ColumnKeywords } from 'hazo_scrape/fetch';

const keywords: ColumnKeywords = {
  amount: { keywords: ['amount', 'dividend'], type: 'number' },
  exDate: { keywords: ['ex date', 'ex-dividend'], type: 'date' },
};

const result = await scrapeTable('https://example.com/dividends', keywords, {
  dateGuard: true,
  required: ['amount'],
});

if (result.partial) console.warn('missing required column(s):', result.unmatched);
console.log(result.rows[0]?.amount?.[0]?.value, result.source.finalUrl);

fetchDocument(url, opts?)

Polite, resilient HTTP built on hazo_secure's safeFetch (SSRF guard, undici, timeout, correlation-id). Resolves an optional INI config (opts.configPath) and overlays per-call opts on top before each request, then: checks robots.txt (fail-open on a robots fetch error), checks the on-disk cache, waits out the per-host rate limit, and runs the retry loop (exponential backoff + jitter, honoring Retry-After) with a rotating UA on each attempt. Returns a FetchResult, never throws for a terminal HTTP status (4xx/2xx/3xx all resolve normally) — it only throws for robots disallow, exhausted retries, or a non-retryable safeFetch error (bad URL, disallowed protocol/host, private-IP block).

interface FetchResult {
  url: string;              // requested URL
  finalUrl: string;         // URL after following redirects
  status: number;
  statusText: string;
  ok: boolean;
  headers: Record<string, string>;
  body: string;
  contentType?: string;
  fromCache: boolean;
  attempts: number;         // 0 when served from cache
  timingMs: number;
  usedProxy?: string;       // set when a configured proxy served this attempt
}

Notable opts (all optional):

• retry: { maxAttempts, backoffMs, backoffFactor, jitter, retryOn } — attempt count, exponential backoff base/factor, jitter toggle, and which HTTP statuses are retried (default [429, 500, 502, 503, 504]).
• rateLimit: { minIntervalMs, perHost } — minimum spacing between requests; the actual wait is max(minIntervalMs, robots Crawl-delay).
• robots: { respect } — set false to skip the robots.txt gate entirely.
• cache: { enabled, dir, ttlSeconds } — content-addressed on-disk response cache, off by default.
• proxy: { urls, username, password, rotation, cooldownMs } — round-robin ProxyAgent pool with failure cooldown; proxied requests bypass safeFetch's SSRF connect-IP guard (accepted trade-off for operator-configured proxies).
• userAgent / uaPool — pin a single UA (also disables rotation for that call) or supply/override the rotation pool.
• fetchImpl — test seam, passed straight through to safeFetch's deps.fetchImpl.

scrapeTable(url, keywords, opts?)

One-call bridge from the fetch layer to the network-free parse layer: fetchDocument → extractTable → mapColumns → mapRows. Takes every FetchOptions field plus select (CSS selector passed to extractTable), dateGuard, and required (both passed straight through to mapColumns).

interface TableResult {
  headers: string[];
  columnMap: ColumnMap;
  unmatched: string[];      // mirrors columnMap.unmatched
  rows: MappedRow[];
  partial: boolean;         // = !columnMap.ok
  warning?: string;         // surfaced from extractTable, e.g. no table found
  source: { url: string; finalUrl: string; fromCache: boolean; usedLlm: false };
}

Politeness & configuration

robots.txt is respected by default and fails open (a robots.txt fetch error is treated as "allowed", never as "blocked"); it honors Crawl-delay when present. Requests are rate-limited per host by default. Each attempt rotates through a browser User-Agent pool (with matching sec-ch-ua) unless you pass userAgent for a single identifiable UA or set ua_pool_enabled = false. The on-disk response cache is off by default. All of this is configurable via an INI file — see config/hazo_scrape_config.ini.sample for the full set of [http], [retry], [ratelimit], [robots], [cache], and [proxy] keys.

Tailwind v4 (@source required)

If this package renders UI, add the following to your app's CSS entry:

@source "../node_modules/hazo_scrape/dist";

License

MIT