@mailwoman/corpus

BIO-labeled training-corpus pipeline for the Mailwoman address parser.

Generates sequence-labeling training data from reference sources (OpenAddresses, libpostal dictionaries, synthetic training-data subsets) and assembles them into the TSV format consumed by the Modal training pipeline. This package produces the data that trains @mailwoman/neural-weights-*.

// The corpus pipeline is primarily build-time CLI tooling.
// Key entry points:
import { expandGolden } from "@mailwoman/corpus" // Expand reference addresses
import { synthesizeSlice } from "@mailwoman/corpus" // Generate synthetic training rows
import { alignRow } from "@mailwoman/corpus" // Align raw address → BIO tokens
import { validateCorpus } from "@mailwoman/corpus" // Validate corpus integrity

What it produces

The corpus pipeline assembles training data from multiple sources:

| Source | Description | | ------------------ | ------------------------------------------------------------------------ | | OpenAddresses | Real government address point data (US, FR, DE, …) | | NAD | National Address Database (US-specific) | | libpostal | Multilingual street/place name dictionaries | | Synthetic rows | Generated address variations (boundary stress, order variants, all-caps) | | Overture Maps | Address theme ingestion (alpha) | | OpenStreetMap | Pakistan, Bangladesh, Vietnam — ODbL, dropped by --exclude-share-alike |

Output format: TSV rows with raw<TAB>BIO_labels consumed by the Python training pipeline (corpus-python/).

Key modules

| Module | Purpose | | ----------------------- | ---------------------------------------------------------------- | | expand-golden.ts | Expand reference addresses into training rows with alignment | | align.ts | Tokenize raw address → BIO label sequence | | validate.ts | Validate corpus integrity, label coverage, subset balance | | synthesizers/*.ts | Synthetic row generators (boundary stress, order variants, etc.) | | ingest/ | Overture Maps + NAD ingestion | | slice-registry.ts | Subset metadata and composition | | stats.ts | Per-subset and per-tag statistics |

Layout

The top-level ownership boundary is the address system, using ISO 3166-1 alpha-2 country directories. A directory owns its recipes, source adapters, acquisition tools, and tests together: fr/{recipes,adapters,tools}, kr/{adapters,tools}, and us/{adapters,tools} are representative. Regional and global work lives under south-asia/ and international/; cross-locale primitives remain at the recipes/ and synthesizers/ roots.

Sources that deliberately span countries (OpenAddresses, GeoNames, Overture, OSM, and WOF) remain under adapters/, with their common fetching and processing utilities under tools/. Source kind is therefore a nested concern inside a country directory, not the package's organizing principle. Tests live beside the module they cover: fr/adapters/ban/adapter.test.ts belongs with fr/adapters/ban/adapter.ts. Production compilation excludes *.test.ts; the test project includes the same pattern. Public subpaths use this same locale-first layout.

Build-time tooling

The corpus is assembled via scripts in scripts/:

# Validate the corpus
node scripts/validate-corpus.mjs

# Rebuild slices
node scripts/build-boundary-stress-slice.mjs

# Corpus statistics
node scripts/corpus-stats.mjs

Design

• BIO (Begin/Inside/Outside) labeling over SentencePiece tokens.
• Character-offset aligned — labels track the raw string, not the normalized form, so the model learns real input distributions.
• Source-homogeneous subsets — each training-data subset comes from one source, ordered by type, so eval splits are clean (no bleed between train and held-out).

Related

• @mailwoman/neural — the runtime that loads and runs the trained model
• @mailwoman/neural-weights-en-us — the trained model itself
• Corpus Construction concepts
• Training Pipeline concepts
• CONTRIBUTING_MODEL_WORK

License

AGPL-3.0-only