@bandf/babel

Document-to-markdown converter for turning common document formats into clean Markdown.

@bandf/babel converts PDF, raster images, EPUB, Office/OpenDocument files, structured data, HTML, plain text, Markdown, and http(s) URLs into normalized Markdown plus source metadata. One function in, clean Markdown out.

Requires Node.js 24 or newer.

import { convert } from '@bandf/babel';

const result = await convert({ filePath: './paper.pdf' });
const markdown = result.markdown;
// → '# Title\n\nbody prose...\n\n## Section\n\n...'

Library API

• convert({ filePath, transliterateHomoglyphs, langs, ocr, pdf, images, url }) — the main export. Document file → conversion result with markdown, source fields, MIME type, and conversion metadata. Used to bring foreign formats into Markdown. For PDFs, pdf tunes the shaping and images additionally extracts the document's images into a directory and links them.
• validate({ markdown }) — takes a markdown string and returns cleaned, validated GFM with unsafe raw markup and unwanted text artifacts removed. convert() runs every result through it.
• createBabel(config) — builds a configured converter instance. Use this when you want instance-level defaults for deterministic conversion options.

import { convert, validate, createBabel } from '@bandf/babel';

const clean = await validate({ markdown: someUntrustedMarkdown });

Markdown Contract

babel's output contract is clean GitHub Flavored Markdown with useful document structure preserved. Every conversion produces:

• ATX headings (# , ## …) — stable section boundaries for downstream consumers.
• Pipe-format tables — kept together as one block, cells preserved.
• GFM structure — links, images, blockquotes, fenced code, and tables survive when the source format exposes them.
• Normalized prose — homoglyphs transliterated, smart punctuation and invisible characters stripped, no raw HTML tag residue.

A convert() call either returns a result whose markdown field is valid markdown or throws. validate() returns valid markdown or throws.

API

Functions

| Function | Returns | Description | |---|---|---| | convert(options) (@bandf/babel) | Promise<BabelConversionResult> | Zero-config document → { markdown, sourceUri, sourceType, mimeType, extension, meta }, plus assets when PDF images were requested. | | createBabel(config) (@bandf/babel) | BabelInstance | Builds a configured instance with convert, validate, and supported-format lists. | | validate({ markdown }) (@bandf/babel) | Promise<string> | Markdown string → cleaned + validated markdown. | | releaseScribe() (@bandf/babel) | Promise<void> | Ends any OCR worker processes still running. Safe to call at any time; call it at the end of a script so the process can exit promptly. |

Constants

These are named JavaScript exports from @bandf/babel, not environment variables or configuration settings:

| Exported constant | Type | Description | |---|---|---| | supportedExtensions | string[] | All recognized local-file extensions. | | supportedImageExtensions | string[] | OCR-capable image extensions: .png, .gif, .jpg, .jpeg, .webp. |

import {
    supportedExtensions,
    supportedImageExtensions
} from '@bandf/babel';

SUPPORTED_EXTENSIONS and SUPPORTED_IMAGE_EXTENSIONS are aliases of the same values.

convert(options)

filePath is required. Every other option is optional and may be supplied to the top-level convert() function or to babel.convert() on a configured instance.

| Option | Type | Default | Description | |---|---|---|---| | filePath | string | — | Local file path or an http:// / https:// URL. | | transliterateHomoglyphs | boolean | true | Maps common Unicode homoglyphs to stable ASCII equivalents during cleanup. | | langs | string \| string[] | ocr.langs, then ['eng'] | OCR language shorthand for PDF and image conversion. Strings may use + or ,, such as 'eng+fra'. Takes precedence over ocr.langs. | | ocr.langs | string \| string[] | ['eng'] | Language codes used for PDF and image OCR. | | ocr.tessdataPath | string | Bundled data directory | Optional directory override for OCR language files. Leave unset for English; eng.traineddata ships with the package and is used automatically. Set this only when supplying other <lang>.traineddata files. | | pdf.stripRepeatedLines | boolean | true | Removes running headers and footers from PDFs: lines repeated in the page margins on most pages (mastheads, page numbers, issue tags). Set false to keep them in the output. | | pdf.tables | boolean | true | Renders tables in PDFs as GFM pipe tables: cells drawn with borders, and tables set with horizontal rules only (top, header and bottom rules, as papers set them). Set false to leave the cell text in the paragraph flow. | | pdf.figures | boolean | true | Reads drawn diagrams, charts and illustrations in PDFs as figures: the text set inside them leaves the paragraph flow and, with images, each figure is rendered to one image file placed with its caption. Set false to leave the drawing's text in the flow. | | pdf.textOnly | boolean | false | Reads only the text a PDF already carries and never runs OCR: its native text, an invisible OCR layer added before, and short text pages that would otherwise be recognized. A PDF with no text layer at all, such as a scan whose pages are only pictures, fails with an error whose code is ERR_NO_TEXT_LAYER. Images are always recognized. | | images.dir | string | — | PDFs only. Extracts the document's content images into this directory (created if missing) and links them from the markdown at their page positions. Without images, no files are written and the markdown carries no image links. | | images.linkPrefix | string | Directory's own name | Path prefix used in the markdown image links, e.g. assets/report → ![Page 3 image 1](assets/report/p03-01.png). Choose it relative to where the markdown will be saved. | | url.timeoutMs | number | 30000 | Positive time in milliseconds allowed to connect and receive the response headers. | | url.bodyTimeoutMs | number | 30000 | Positive time in milliseconds allowed without data while receiving the response body. A large download that keeps making progress is never cut off. | | url.maxBytes | number | 26214400 (25 MiB) | Positive maximum response-body size for URL conversion, checked against the declared size and counted as the body arrives. | | url.maxRedirects | number | 2 | Positive maximum number of redirects followed during URL conversion. | | url.lookup | BabelUrlLookup | System DNS lookup | Advanced async DNS resolver override, used both to check remote addresses and to connect. |

The ocr options apply only to PDFs and raster images, the pdf and images options only to PDFs. The url options apply only when filePath is an HTTP(S) URL. Per-call options override defaults from createBabel().

The UI's --max-upload-size flag does not apply to this API. Local paths have no general configurable file-size limit, although available memory, underlying runtime constraints, and format-specific safety checks still apply. URL response size is controlled separately with url.maxBytes.

Result

| Field | Type | Description | |---|---|---| | markdown | string | The validated GFM output. | | sourceUri | string | The file path or URL converted. | | sourceType | 'file' \| 'url' | Where the document came from. | | mimeType | string | The source document's media type. | | extension | string | The source file's extension, for local files. | | assets | BabelAsset[] | PDFs with images: the image files written, as { path, link, page, width, height, bytes }. | | meta.markdownLength | number | Length of markdown. | | meta.byteLength | number | Size of the source file. | | meta.pageCount | number | PDFs: number of pages. | | meta.tableCount | number | PDFs: number of tables rendered as pipe tables. | | meta.figureCount | number | PDFs: number of drawn figures found. | | meta.imageCount | number | PDFs with images: number of image files written. | | meta.textStrategy | string | PDFs: confidence-filtered when the shaped text was used, scribe-text when the plain extracted text was kept instead. |

Other meta fields (OCR word counts and confidence, languages) are informational.

const result = await convert({
    filePath: 'https://example.com/article',
    url: {
        timeoutMs: 15_000,
        maxBytes: 50 * 1024 * 1024,
        maxRedirects: 2
    }
});

PDF images

const { markdown, assets } = await convert({
    filePath: './report.pdf',
    images: { dir: './out/assets/report', linkPrefix: 'assets/report' }
});
// markdown contains ![Page 3 image 1](assets/report/p03-01.png) between the
// paragraphs the image sat among; assets lists what was written:
// [{ path, link, page, width, height, bytes }, …]

Images are taken from the PDF's own image objects at native resolution and written as PNG (transparency preserved) or as JPEG when that is much smaller. Decoration is left out: images repeated on more than two pages (page backgrounds, logos), flat colour blocks and drop shadows, and images under 40 px or 30 pt. Files are named pNN-MM.ext by page and reading order. On scanned pages the links are appended after the text instead of positioned within it.

validate(options)

| Option | Type | Default | Description | |---|---|---|---| | markdown | string | — | Required Markdown string to clean and validate. | | sourceUri | string | '(string)' | Source label included in validation error messages. | | transliterateHomoglyphs | boolean | true | Enables or disables homoglyph transliteration. |

Supported formats

| Format | Conversion behavior | |---|---| | .txt / .text | UTF-8 text cleanup, then markdown-normalized + validated | | .md / .markdown | read as markdown, then markdown-normalized + validated | | .html / .htm | Main article content, structure, and links → GFM markdown | | .docx | Headings, paragraphs, lists, links, and tables → GFM markdown | | .csv / .tsv | Rows and columns → GFM table | | .json | JSON parse/validate → fenced json block | | .yaml / .yml | YAML parse/validate → fenced yaml block | | .xml | XML parse/validate → fenced xml block | | .xlsx | Visible cell values → one GFM table per worksheet | | .ods | Visible cell values → one GFM table per sheet | | .pptx | Slide text, speaker notes, and image alt text → markdown | | .odt | Headings, paragraphs, lists, and tables → GFM markdown | | .rtf | Text and basic document structure → markdown | | .pdf | Embedded text extraction with OCR when needed (never with pdf.textOnly); text-native pages are shaped by their typography (see below); images extracted on request | | .png / .gif / .jpg / .jpeg / .webp | Image OCR → markdown | | .epub | EPUB 2/3 chapters in reading order → GFM markdown | | http(s) URL | Safety-checked conversion of HTML, Markdown, plain text, JSON, or PDF. Markdown is requested first, then the server's usual representation; a PDF is downloaded to a temporary file and converted like a local PDF, with the pdf, ocr and images options |

PDF shaping

Text-native PDF pages are shaped from the document's own typography and annotations; scanned pages go through OCR and the same shaping where confidence allows.

• Headings — the largest heading on the cover becomes the title, display type becomes section headings, bold body-size lines become sub-headings. Numbered section headings (1 Introduction, 3.2 Evaluation Task, B.1 …) take their level from the number's depth, and lines wrapping under the number join them; short headings set a little larger than the body (Abstract, References) are headings too. A run-in heading that opens its paragraph stays in it.
• Inline structure — bold and italic runs become **strong** and _emphasis_; hyperlink annotations become [text](url); bullet glyphs and runs of dash-led lines become list items; superscript markers with their small-type notes, set at the page foot or in the margin, become GFM footnotes ([^1] / [^1]: …); indented italic passages become blockquotes (verse keeps its line breaks).
• Flow — paragraphs are rejoined across page breaks, column wraps and drop caps; paragraphs set without space between them are told apart by their first-line indents, and hanging-indent lists (references) item by item; running headers and footers are removed (pdf.stripRepeatedLines); page rules (---) are emitted only for page-oriented documents such as slides. Characters that would read as markdown (*, _, [, a leading + or #) are escaped, so page text never becomes markup.
• Images — with images, a caption directly below a picture (all its lines, set in one type) or a short line of its own directly above it is placed with the image and used as the alt text. A paragraph interrupted by a picture, a table or a caption is joined back together.
• Figures — a drawn diagram, chart or illustration (curves, boxes and pictures with text set inside them, under a centred title) is one figure (pdf.figures, default on): its text, ruling lines and part pictures leave the flow, and with images the whole is rendered to one image file (p03-f01.png, about 2000 pixels across, denser when a picture inside it is) placed with its caption. A preprint identifier stamp in the margin of the first page is dropped.

PDF tables

Tables become GFM pipe tables (pdf.tables, default on). Three layouts are recognised:

• Ruled tables — cells drawn with borders. The first row is the header; group labels spanning sub-headers fold into them (2008 Yes); merged cells keep their text in the first position; a bold line directly above the table is kept as its caption; a table continued on the next page under a repeated header is joined.
• Partly ruled tables — ruled only between some columns. The header row's cells complete the columns (they split where the header's cells start), and a header sitting on the top rule joins the table.
• Tables set with horizontal rules only — a rule above the header, one below it and one at the foot, as papers set them. The header row's cells name the columns (header cells that wrap onto several lines are read as one); body words fall into the nearest column; group labels over underlines span their columns; standard errors on the row below an estimate stay on their own row; cells that wrap onto further lines are joined; a section label between two rules heads the label column; prose between rules is left alone.

In every layout numeric columns are right-aligned, a bare - in a cell (no data) is kept, and single-letter marks such as X stay as they are. Tables with no ruling lines at all are not detected.

Spreadsheet conversion extracts visible values only. It does not evaluate formulas, render charts, or OCR images embedded inside Office/OpenDocument containers. PPTX conversion extracts slide text, speaker notes, and image alt text; embedded media stays out of scope unless it is already exposed as text.

URL conversion refuses anything but a routable public address: private, loopback, link-local, reserved and multicast ranges, and the IPv6 forms that carry an IPv4 address, written in any notation, as well as localhost, *.localhost and *.local names. It refuses URLs that name an executable or script file (.sh, .exe, .ps1, …) and responses that declare executable content. Resolved addresses are checked before each request and again for the connection itself, on every redirect, so DNS cannot switch to a private address in between. Redirect, response-size and timeout limits apply throughout. Hosted multi-tenant deployments should still isolate network access when accepting user-provided URLs.

All listed formats work after a normal npm install; no format-specific packages need to be installed separately. English OCR requires no configuration: eng.traineddata ships with the package and is used automatically. For other OCR languages, set ocr.tessdataPath to a directory containing each requested <lang>.traineddata file:

const { markdown, meta } = await convert({
    filePath: './scan.pdf',
    ocr: {
        langs: ['fra'],
        tessdataPath: './ocr-data'
    }
});

Configured instances

Use createBabel(config) when you want an instance with shared conversion defaults:

| Config option | Type | Default | Description | |---|---|---|---| | transliterateHomoglyphs | boolean | true | Default for calls to babel.convert(). | | ocr | BabelOcrConfig | { langs: ['eng'] } | Shared langs and tessdataPath defaults for PDF/image OCR. | | pdf | BabelPdfConfig | { stripRepeatedLines: true, tables: true, figures: true, textOnly: false } | Shared PDF defaults, see pdf.stripRepeatedLines, pdf.tables, pdf.figures and pdf.textOnly above. | | url | BabelUrlConfig | See convert() defaults | Shared timeoutMs, bodyTimeoutMs, maxBytes, maxRedirects, and lookup defaults for URL conversion. |

import { createBabel } from '@bandf/babel';

const babel = createBabel({
    transliterateHomoglyphs: false,
    ocr: {
        langs: ['eng'],
        tessdataPath: './tessdata'
    },
    url: {
        timeoutMs: 15_000,
        maxBytes: 50 * 1024 * 1024,
        maxRedirects: 2
    }
});

const result = await babel.convert({ filePath: './paper.pdf' });
console.log(result.markdown, result.meta);

Options passed to babel.convert() override these instance defaults. images is per call only, since its directory belongs to one document. babel.validate() takes the standalone validate() options shown above and does not inherit conversion defaults from createBabel().

Browser UI

Launch the local document-testing UI directly from the npm package:

npx @bandf/babel [options]

The server opens the UI in your default browser automatically. Upload a document and inspect the raw Markdown next to the rendered preview (GFM tables, footnotes with linked references and back-links, extracted images; front matter is left out of the preview and shown by the Front matter button instead). The raw pane is editable: the preview follows your edits as you type, Copy takes the edited text, and Revert restores the converted output. Download is a zip of the markdown as shown, with the document's images beside it under assets/ so the links work where you unpack it. The two panes scroll together, block for block, whichever one you scroll. The Raw / Split / Rendered switch shows either pane at full width or both side by side, and the choice is remembered. Six toggles re-run the conversion on the current document: homoglyph transliteration, and for PDFs the removal of running headers and footers, table detection, figure detection, image extraction and text-only reading (pdf.stripRepeatedLines, pdf.tables, pdf.figures, images and pdf.textOnly); hover a toggle for what it does. A new conversion replaces any edits after asking. Extracted images live in a temporary directory that is removed when the server stops. Stop the server with Ctrl+C.

| Flag | Default | Description | |---|---|---| | --port <number> | 0 | Listening port from 0 to 65535. Port 0 asks the operating system for an available port. | | --max-upload-size <size> | 1.5gb | Maximum HTTP upload body. Accepts bytes or kb, kib, mb, mib, gb, and gib, including decimal values such as 1.5gb. | | --max-upload-bytes <size> | 1.5gb | Alias for --max-upload-size; raw byte counts are accepted. | | --no-open | — | Start the server without opening a browser. | | --open | Enabled | Explicitly enable automatic browser opening. |

Value flags accept either --flag=value or --flag value. Options may be combined:

npx @bandf/babel --port=3187 --max-upload-size=2gb --no-open

The server always binds to 127.0.0.1 and prints the resulting URL. It is only intended for local use. The upload-size flags control the UI's HTTP request body; they do not limit files passed directly to the public convert() API.

License

AGPL-3.0-or-later. See LICENSE.