@gmb/bitmark-parser

v7.2.0

Published

a day ago

A parser for bitmark text, powered by WebAssembly.

Downloads

2,077

0High
0Medium
0Low

bitmark parser wasm webassembly

A high-performance parser and generator for bitmark. Provides a programmatic API (powered by WebAssembly) and a CLI (prebuilt native binary per platform) for converting between bitmark markup and JSON.

Installation

npm install @gmb/bitmark-parser

Programmatic API (JS / TS)

import {
  convert,
  semanticTokens,
  breakscapeText,
  unbreakscapeText,
  info,
  version,
} from "@gmb/bitmark-parser";

// Convert bitmark to JSON — set inputFormat explicitly when known (skips
// auto-detection); "auto" also works but costs an extra scan
const json = convert("[.article]\nHello **bold**", {
  inputFormat: "bitmark",
  warnings: true,
});
const parsedJson = JSON.parse(json);
console.log(parsedJson[0].bit.type);
console.log(parsedJson[0].parser.warnings);
console.log(parsedJson[0].bitmark);

// Generate bitmark from JSON
const bitmark = convert(json, { inputFormat: "json", outputFormat: "bitmark" });

// Parser-derived highlighting in the LSP semantic-tokens shape
const highlights = semanticTokens("[.article]\nHello **bold**", {
  tokensLayout: "absolute",
});

// Dump the lexer's token stream (debug tooling; bitmark input only)
const tokens = JSON.parse(
  convert("[.article]\nHello **bold**", {
    inputFormat: "bitmark",
    outputFormat: "lex",
  }),
);

// Breakscape / unbreakscape text
const escaped = breakscapeText("[!example]", {
  format: "bitmark++",
  location: "body",
});
const unescaped = unbreakscapeText(escaped, {
  format: "bitmark++",
  location: "body",
});

// Query supported bit types
const bitInfo = info({ infoType: "list", format: "text" });

// Get library version
console.log(version());

WASM variants (`full` / `browser-full` / `bitmark-json`)

The package ships three wasm builds of the same engine and selects one at runtime — the API surface never changes:

| Feature | Contents | Size (approx.) | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- | | full (Node default) | everything: rich info metadata + built-in translations + the semantic diff | ~1025 KB (~395 KB gzip) | | browser-full (browser default) | the same conversions and diff, without either | ~818 KB (~333 KB gzip) | | bitmark-json | bitmark ↔ JSON only: convert/canonicalize/transform (formats auto/bitmark/json, plus the text, semantic-tokens and diagnostics outputs), the editor services, info (JSON only), breakscape, text fragments — no diff | ~599 KB (~239 KB gzip) |

The rule. Every build includes every feature, except the two browser-targeted variants, which each carry an explicit exclusion list. That is the whole story — a new capability is in every build unless it is named below:

| build | features | | -------------------------------------------------- | ----------------------------------------------------------------------------------- | | native CLI, daemon, docgen, schemagen, wasm full | all | | wasm browser-full | all except info-meta (bit + tag prose) and translations (the baked table) | | wasm bitmark-json | all except the above and markup, mapping-report, diff, lex, info-text |

So, not in bitmark-json: the markup formats (html, xml, xml-niso-iec, …) as input or output (they throw an unsupported error), the lex output format and the mappingReport option (same error), diff (throws UnsupportedFeatureError), and the human-readable rendering of info — info({ format: "text" }) throws UnsupportedFeatureError there, so pass format: "json" (the JSON views are the contractual ones and are in every variant).

full and browser-full convert identically — they differ only in what info can report: the META fields (bit and tag descriptions, group provenance and raw mapping patterns) and the built-in translations table. Bit titles and the bit-group / resource-group catalogs are in every variant, in English; register supplies translations to the lean ones (see Display names and languages).

Why the defaults differ: only the browser pays a download-latency cost for those strings, and a Node backend loading wasm from disk has the same requirements as the native CLI. But which channel an artifact ends up in is not knowable from the artifact — browser code is routinely package-managed and webpack-built — so both variants are available on both platforms and the defaults are conveniences only. If you bundle the Node entry for the browser, select browser-full explicitly.

Migrating from ≤ 6.10: full used to mean what browser-full means now. Browser code that explicitly selected "full" should move to "browser-full" to keep its download size close to what it was; leave it on "full" to gain the metadata and built-in translations for ~62 KB gzip. Nothing else changes — the capabilities that were in full are in both.

import { init, convert, variant } from "@gmb/bitmark-parser";

await init({ feature: "bitmark-json" }); // load the minimal wasm
convert("[.article]\nHello"); // works
variant(); // "bitmark-json"

// Upgrade in place: the interface stays valid while the full wasm loads —
// calls keep being served by the active module, then swap atomically.
await init({ feature: "full" });
convert("[.article]\nHi", { outputFormat: "html" }); // now works

Notes:

Node: with no explicit init, the first call lazily loads full synchronously — existing zero-init usage is unchanged. initSync({ feature }) selects a variant synchronously.
Browser: init() is required as before, and loads browser-full; only the selected variant's .wasm is fetched (all three small JS glue modules are in the bundle). Overlapping init calls: same-feature calls coalesce; a different-feature call supersedes an unfinished one (last call wins); every promise resolves once the finally-active module is live. init({}) is the same request as init(). In the browser, initSync(bytes, { feature }) instantiates from bytes you supply, and takes part in the same last-call-wins order: an init still loading when initSync runs never swaps in afterwards.
Supplying your own .wasm: glue and module are a pair, so name the feature the bytes belong to — init({ feature, module_or_path }). The back-compat positional form init(module_or_path) stays on full, since those bytes are bitmark_wasm_bg.wasm.
Workers: transformParallel workers load the active variant.
Legacy API: works on bitmark-json (it follows the active module) except its HTML-dependent members (e.g. convertHtmlTable), which throw.
A previously active module's reference is dropped on swap; wasm-bindgen's glue retains the instance until that variant is re-initialised, so switching back is cheap but the memory of an abandoned variant is only reclaimed on page/process end or its re-init.

API

Errors

Errors are thrown, never returned. Every fallible WASM export raises a JavaScript Error whose message has one shape:

<kind> at <path>: <message>

A returned string is therefore always a result — there is no prefix or sentinel to test for, and every call site is a try/catch.

<kind> is a stable kebab-case identifier — the part to match on. <path> locates the failure (a JSON pointer-ish path, offset N for a JSON syntax error, or empty) and <message> is human-readable prose that may change in any release. The kinds:

| kind | meaning | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | invalid-json | the input string is not well-formed JSON | | compose | the reverse composer rejected the JSON (shape / field / type) | | unsupported | the requested conversion is not available (an output-only format used as input, or a format compiled out of this variant) | | patch-parse-error | a patch or patch document is malformed | | patch-index-out-of-range | a patch path addressed an array element that does not exist | | patch-type-mismatch | a patch value's JSON type is incompatible with its target | | patch-conflict | a patch document's entries contradict each other (a bit removed or moved twice, or an anchor that is itself removed or moved) |

Every TypeScript wrapper propagates that Error unchanged — convert, transform, canonicalize, bitmarkToObjects, countBits, diff, … — as does the parallel worker pool, which rejects with the first failing bit's error. register re-wraps it as a RegisterError; a capability the loaded variant compiled out raises UnsupportedFeatureError before any wasm call.

convert(input: string, options?: ConvertOptions): string

Convert between bitmark, JSON, and the mapped markup formats (html/xml/…).

A JSON input may be an array or a single value, each entry a { bit: {…} } envelope (the forward output) or a bare bit object. An entry that is not a bit produces no bitmark: it is skipped and reported as an unknown-bit-type issue naming its JSON path ($[2] is not a bit: no "type" — skipped) on the warnings channel of the CLI (--warnings) and the daemon; the bits beside it still convert, and a document in which nothing is a bit converts to empty output. This convert returns a bare string and so cannot show the skip. Only malformed JSON text throws (invalid-json).

Options:

inputFormat — "auto" (default), "bitmark", "json", or a mapping id ("html", "xml", …)
outputFormat — "auto" (default: the opposite direction), "text" (lossy plain-text extraction; output-only), "semantic-tokens" / "diagnostics" / "lex" (output-only, bitmark input only — see their sections), or as above
mode — "optimized" (default) or "full"
warnings — include validation warnings (default: false)
plainText — output text as plain text (default: false)
pretty — prettify the emitted JSON (default: false). For JSON output this is the whole document; for bitmark output it pretty-prints the embedded JSON bodies of json-format bits (cosmetic — they re-parse to the same value)
indent — indent size for pretty JSON (default: 2)
includeUnknownProperties — bitmark → JSON only (default: false): include unknown properties in the bit JSON. An unknown property ([@key:value] the bit's config cannot resolve, at the bit body/footer — never inside cards or consumed by a chain) is appended after all real keys as an array of strings (a valueless [@key] → true); a key colliding with a configured key of the bit is _-prefixed. A bare tag the bit does not declare ([!…], [?…], [#…], … on a bit whose config has no such tag) is an unknown too, under the key a declaring bit would use ([!x] → "instruction": ["x"]); a rejected resource attachment is not — it is still emitted, with a warning. Unknown properties are never converted back to bitmark, and each occurrence warns whether or not it is included. Also available on bitmarkToObjects and as the CLI's --include-unknown-properties
mappingReport — replace the converted output with a human-readable MAPPING REPORT of what each input construct mapped to, including the inputs that are NOT mapped (default: false). The report lists every input occurrence, grouped by bit (mapped → its output construct, carried → what consumed it, or NOT MAPPED), a per-name summary table, and a NOT-MAPPED inventory. Composite conversions (e.g. xml → json) report every leg, one section after the other. Most useful on foreign document import (inputFormat: "xml-niso-iec"), where NOT MAPPED means recorded loss. Also the CLI's --mapping-report

Performance: inputFormat: "auto" detects the format by scanning the input for a bitmark bit header — for non-bitmark input that is a full pass over the whole input before parsing starts, and the detection is a heuristic (markup formats are never auto-detected). If you know the input format, set inputFormat explicitly. The same applies to canonicalize and transform, whose inputFormat also defaults to "auto".

Input limits: JSON objects/arrays and markup elements may nest at most 256 levels deep. Deeper JSON is rejected like any other malformed JSON ("error: … nesting depth exceeds 256"); a deeper markup open tag is kept as literal text. The bound keeps a crafted document from exhausting the WASM stack — an instance that rejects a too-deep input stays usable for the next call.

convert is the single string-based conversion surface — parsing is { inputFormat: "bitmark", outputFormat: "json" }, generating is { inputFormat: "json", outputFormat: "bitmark" }. For typed objects use bitmarkToObjects / objectsToBitmark (below), which pin their formats.

For bitmark input, output is a JSON array of entries with:

bit — serialized bit payload
parser — parser metadata, errors / warnings, and infos (informational notices for recoveries that are not necessarily wrong: text kept as text because it only looks like an unclosed tag, unclosed-tag, or a formatting mark with no partner, unclosed-formatting — in bitmark malformed markup is text, not an error). Each issue carries a stable machine-readable code ("unknown-property", "missing-required-tag", …) beside its human-readable message, plus text and location. Branch on code — a shipped code never changes meaning, whereas the message text is not contractual and may be reworded in any release
bitmark — source bitmark text (a convenience echo; every bit is fully described by bit alone)

A region the parser could not read — an unknown bit type, or non-blank text that never opened a bit — comes back as a bit of type _error beside its parser.errors entry. It carries originalType (the header exactly as written between the level dots and the ], including a leading | for a commented-out bit and any :format / &resource suffix; absent when there was no header) and body (the raw text after it, as a plain string), so generating bitmark from it reproduces the original region.

canonicalize(input: string, options?: CanonicalizeOptions): string

Re-emit input in its own format in canonical form — the single same-format surface. mode: "optimized" (default, omit natural defaults) or "full" (all keys).

transform(input: string, options?: TransformOptions): string

Apply a patch document to a document, bit by bit, and re-emit it. transformParallel is the worker-thread variant returning a Promise<string>.

patch — a PatchDocument, or the shorthand: a bare array of entries applied to every bit (build entries with patchEntry). Either form may be a JSON string
preHook / postHook — per-bit hooks; the pre-hook may return patches, an output-format override or drop. Hooks see the original bits, in order, before the document reshapes anything; a hook's patches apply after the document's
inputFormat, outputFormat, mode, pretty, indent, spacesAroundValues — as for convert. outputFormat (default "json") takes every per-bit format: "json", "bitmark", "text" or a mapping id such as "html" (per-bit text outputs are joined with newlines). The whole-document outputs "lex", "semantic-tokens" and "diagnostics" are refused with an unsupported error in every driver — use convert. An explicit inputFormat is honoured as given, patch or hooks or not; it is never re-detected from the content

A patch document addresses bits by their position (or id) in the input before any entry is applied, so it reads like a diff:

{
  "version": 1,
  "entries": [
    {
      "select": { "index": 3, "id": "mc-014" },
      "patch": [{ "path": "title", "op": "set", "value": "New" }]
    },
    { "select": { "index": 9 }, "remove": true },
    {
      "insert": { "after": 4 },
      "bit": { "type": "article", "body": "Inserted" }
    },
    { "select": { "index": 7 }, "move": { "after": 12 } },
    { "select": "all", "patch": [{ "lang": "de" }] }
  ]
}

after names an anchor — an input bit that is neither removed nor moved, or -1 for the document start. A bit may be the target of at most one remove or move entry: moving a bit twice, removing it twice, or removing and moving it are all rejected, naming the second of the conflicting entries. diff produces such a document from two versions of a file (below).

diff(a: string, b: string, options?: DiffOptions): string

Semantic diff of two documents. Both sides are re-canonicalized and compared as bit JSON, so tag order, spacing, breakscaping and omitted defaults are not differences; every real change is rendered as the canonical bitmark an author would type. The inputs may be bitmark or JSON, and need not share a format.

@@ bit 1 -> 2 (id mc-014) [.multiple-choice] changed
 instruction[0]
-[!Pick one]
+[!Pick exactly one]
 quizzes[0]
 ====
 [-Bonn]
-[-Berlin]
+[+Berlin]
@@ bit 4 (id hero) [.image] removed
-[.image]
-[@id:hero]
-[&image:https://example.com/hero.png]

Bits are paired by id, then by sourceBB on the same sourceRL, then by exact content, then by text similarity; a bit whose position changed among the kept ones is reported moved. Empty output means the documents are semantically equal.

outputFormat — "bitmark" (default; the text above), "json" (one object per aligned bit, see diffBits), or "patch" (a patch document that turns a into b — feed it to transform)
inputFormat — applies to both sides (default: "auto", sniffed per side)
context — unchanged lines kept around each change (default: 3)
locate — add source line numbers to bit positions (bitmark inputs; default: false)
similarity — pairing threshold for bits with no id, box or exact match, 0..1 (default: 0.6; 1 disables)
bboxTolerance — pixels per coordinate for the sourceBB stage (default: 8)
spacesAroundValues — in the rendered bitmark lines (default: 0)

The text layout and the JSON shape are stable contracts. An unknown property is shown as a JSON line, since the generator cannot render it, and cannot be re-applied. Not available on the bitmark-json variant (UnsupportedFeatureError).

diffBits(a: string, b: string, options?: DiffOptions): BitDiff[]

The typed form of diff: one BitDiff per aligned bit, unchanged ones included so the alignment is complete — status (unchanged | changed | moved | removed | added), the a/b positions, ops (a patch turning the A bit into the B bit) and revert, the parser issue codes gained or lost, and the rendered hunks.

countBits(input: string): number / splitBits(input: string): BitSlice[]

Count or split the bits of a bitmark/JSON input without full parsing. A JSON input may be an array or a single value, each entry a { bit: {…} } envelope or a bare bit object (the four shapes convert accepts); splitJsonBits always returns envelopes. An entry that is not a bit is not counted and not returned — these calls have no warnings channel; convert reports the skip.

semanticTokens(input: string, options?: SemanticTokensOptions): SemanticTokens

Parser-derived highlighting of bitmark source in the LSP semantic-tokens shape — the typed form of convert(input, { inputFormat: "bitmark", outputFormat: "semantic-tokens" }). The parser runs (never the validator; diagnostics stay on their own channel) and the AST is walked loss-lessly, so every character of the source is covered exactly once and the highlighting agrees with the JSON the same input produces: an unpaired ** is text, an abandoned [!tag is text and the ==== / -- it would have swallowed are dividers, the body of a bit whose body format is not bitmark ([.code], ==== text ====) is plain text. Bitmark input only.

Options (SemanticTokensOptions, also ConvertOptions keys):

positionEncoding — "utf-16" (default; the LSP default, what browser editors index by) or "utf-8" (bytes). Echoed in the response.
tokensLayout — "lsp" (default): data, the LSP relative encoding, five integers per token (deltaLine, deltaStart, length, tokenType, tokenModifiers; the type is a legend index, the modifiers a bitset); or "absolute": tokens, one { line, start, length, type, modifiers } per token, 0-based, by name. Both carry legend and positionEncoding.

Lines break at \n, \r\n and \r; no token contains a line terminator. The legend orders are stable (a change is semver-major; new entries may be appended). Token types name sigil classes and syntax positions, never a bit or tag name; each has a recommended standard super type so an editor's default theme colours it — paste the table into a VS Code extension's semanticTokenTypes:

| type | covers | super type | | ---------------------------------------------------------------------------------- | -------------------------------------------------------- | ---------------------------------- | | frontmatter | the region before the first bit | comment | | bitSigil | [., level dots, \|, :, &, ] of a bit header | keyword | | bitType / bitFormat / bitResourceType | the header's three fields | type / modifier / type | | tagSigil | every tag's own syntax ([@, [!, [##, :, ], …) | keyword | | propertyKey / resourceType | a property tag's key / a resource tag's type | property / type | | tagText | plain text inside any tag value | string | | cardDivider / sideDivider / variantDivider / footerDivider / textDivider | ====, --, ++, ==== footer ====, ==== text ==== | operator | | paragraphBreak | a \| paragraph break | operator | | headingSigil / heading | # / the heading's plain text | keyword / string | | listMarker | • , •1 , •+ , the indent | keyword | | codeSigil / codeLanguage / codeBody | \|code: / the language / the body | keyword / type / string | | imageSigil / imageSrc | \|image: / the source | keyword / string | | markSigil | a mark's open and close markers | operator | | bold / italic / highlight / light / inline | the marked text (in body text and bitmark++ values; a string value is one literal tagText) | string | | attrSigil / attrKey / attrValue | an attr chain's \| and : / key / value | operator / property / string | | url | a bare URL the parser auto-linked | string | | text | plain body / card / footer text, and every top-level gap | (none) | | plainText | a plain region (after ==== text ====, or a raw body) | string |

Modifiers, in bit order: the tag classes property, resource, title, reference, anchor, item, instruction, hint, true, false, gap, mark, solution (every token of a tag of that class, sigils included — exactly one per tag); comment (every token of a commented-out bit); unclosed (tokens inside a parser unclosed-mark / unclosed-tag notice); bold, italic, highlight, light (the text of an inline mark ==…==\|…\| whose chain carries that bare key — style it as you style the bold / italic / highlight / light token types). The full contract is .zen/specs/API-SEM-semantic-tokens.tsp.

diagnostics(input: string, options?: DiagnosticsOptions): Diagnostics

Every issue the JSON parser envelope would carry for bitmark source, as LSP 3.17 Diagnostics with editor positions — the typed form of convert(input, { inputFormat: "bitmark", outputFormat: "diagnostics" }). Parse and validate, no serialization: cheap enough for every keystroke. The validator owns every issue, so this is exactly the envelope's set — the same codes, the same spans, the same order (bit by bit) — with the envelope's buckets as LSP severities: errors → 1 (DiagnosticSeverity.Error), warnings → 2, infos → 3. Bitmark input only.

import { diagnostics, DiagnosticSeverity } from "@gmb/bitmark-parser";

const { positionEncoding, diagnostics: list } = diagnostics(source);
// list[i] = {
//   range: { start: { line, character }, end: { line, character } },  // 0-based, UTF-16 units
//   severity: 2,                        // DiagnosticSeverity.Warning
//   code: "unknown-property",           // the contract — branch on this
//   source: "bitmark",
//   message: "[@nope] is an unknown property …",   // prose, may change
//   data: { bit: 0 },                   // the bit's index in the JSON array
// }

Option: positionEncoding — "utf-16" (default) or "utf-8", as for semanticTokens; echoed in the response. The types (Diagnostic, Range, Position, DiagnosticSeverity) are written field for field as the protocol defines them, so a Diagnostic assigns to Monaco's marker input and to vscode-languageserver-types without a cast, and the package keeps its zero runtime dependencies. In every wasm variant. The contract is .zen/specs/API-EDT-editor-services.tsp.

complete(input, position, options?) / resolve(input, position, item, options?) / hover(input, position, options?)

The editor position queries, in LSP 3.17 shapes (PLAN-196, PLAN-202). Positions are 0-based { line, character } in positionEncoding units — "utf-16" by default, as for semanticTokens.

complete returns an LSP CompletionList of everything valid at the cursor:

| where the cursor is | what you get | | --- | --- | | a bit header's type ([.art) | every bit type (kind: Class), with its title | | a header's :format / &resource | the body formats / the resource types this bit allows | | a tag, or body / card / footer text | the tags valid in THAT scope — the bit body, a card variant, a chain — minus those already at their maximum, the ones the scope still requires sorted first (the first preselected) | | a tag's value | an enum tag's vocabulary, a boolean tag's true / false | | after a tag's ] | that tag's chain children | | an inline attribute chain (==x==\|bo) | the attribute keys, and the closed value set of one that has it | | a line holding a prefix of one (empty, =, ==== f) | the structural lines the bit's card set allows there (====, --, ++), ==== footer ====, ==== text ==== — each documented | | bitmark text (body text, a bitmark++ tag's value), after none, = or == | the inline mark ==…==\|…\| as a snippet (insertTextFormat: 2); never in a string value or a plain region |

Items carry kind, detail, tags: [1] when deprecated, sortText, insertText, and — in LSP's own data slot — the info record behind them, so you can build UI from fields rather than prose. isIncomplete is always false: every candidate for the context comes back and filtering by the typed prefix is the editor's job. A half-typed [@wi, [.art or ==x==\|bo resolves as if it were closed. A header resource attachment ([.flashcard&image]) is offered at the bit's level until it is there; a tag whose label is only its sigil (%, !) leads its detail with its name (item, instruction). The result adds one field beyond LSP, context, saying where the query resolved.

Pass triggerCharacter in the options when a character opened the query (LSP CompletionContext.triggerCharacter; Monaco and VS Code hand it to the provider). A trigger that opens nothing where the cursor is — a . or a - typed in prose — answers an empty list; [ opens a tag anywhere, and = / - / + narrow the list to the structural lines they begin. An explicit invocation (Ctrl+Space) is never narrowed. Editors that open the list on every letter (Monaco's quickSuggestions) should turn that off for bitmark: it is prose with markup in it.

Items carry NO documentation (PLAN-202): a header's list is every bit type, and its prose would be most of the bytes of every list. resolve is LSP's completionItem/resolve — pass the same input and position you gave complete plus one item as it was listed, and get that item back with its Markdown documentation (a bit's title and description; a tag's format, count, default, JSON key, chain and description in this scope; an inline attribute's shape). Wire it to Monaco's resolveCompletionItem or an LSP server's resolve handler, so only the item about to be shown is rendered. When the list at that position no longer holds the item, the item comes back as given.

hover returns an LSP Hover — Markdown contents plus the record in data — for the bit type, a tag AS RESOLVED IN ITS SCOPE (the same record info({ infoType: "bit", full: true }) shows for it there), a tag value, a header resource type, or an inline attribute key or value. It is null on body text, dividers and whitespace.

import { complete, resolve, hover, CompletionItemKind } from "@gmb/bitmark-parser";

const list = complete("[.article]\n[@", { line: 1, character: 2 });
list.context; // { bit: "article", scope: "bit" }
list.items.filter((i) => i.kind === CompletionItemKind.Property).map((i) => i.label);

const id = list.items.find((i) => i.label === "@id");
resolve("[.article]\n[@", { line: 1, character: 2 }, id).documentation.value; // Markdown for @id here

const h = hover("[.article]\n[@id:1]", { line: 1, character: 2 });
h.contents.value; // Markdown: format, count, default, JSON key, description
h.data.tag; // the info record for @id in this bit

Both need a variant built with editor (all three are) — otherwise they throw UnsupportedFeatureError. All three read bitmark SOURCE, so a JSON document is refused; an empty buffer, or one that is still only frontmatter, is source being typed and is answered normally (an empty one's complete offers the bit types). The CLI has the same three services under one command: bitmark editor diagnostics|complete|resolve|hover. The contract is .zen/specs/API-EDT-editor-services.tsp.

The `lex` output format

convert(input, { inputFormat: "bitmark", outputFormat: "lex" }) returns the lexer's token stream as a JSON array: one { kind, span, text } per token (the LexToken type), span in bytes. Debug tooling — the token kinds are the engine's own names, not a stable vocabulary, and the parser is never run (pairing, abandoned-tag repair and raw-body handling all happen after lexing), so a highlighter should use semantic-tokens instead. Bitmark input only; a JSON or markup document has no source text to lex. The former lex() export and bitmark lex command were removed in 7.0 (see the migration guide).

breakscapeText(input: string, options?: BreakscapeOptions): string

Breakscape text (escape bitmark special characters).

format — "bitmark++" (default) or "plainText"
location — "body" (default) or "tag"

unbreakscapeText(input: string, options?: BreakscapeOptions): string

Unbreakscape text (unescape bitmark special characters).

format — "bitmark++" (default) or "plainText"
location — "body" (default) or "tag"

info(options?: InfoOptions): string

Query information about supported bit types. "bit"/"all" default to a compact view — one line per tag with its format, effective count and default, chains and card-set tags nested. With full: true they report the complete detail: body/footer/resource rules, default provenance (configured vs format-natural), nullability (absence as a distinct state — orthogonal to defaults), JSON keys, per-context overrides, and the card set structure. "deprecated" lists deprecated bits with their deprecation version and (separately) any migration target.

infoType — "list" (default), "bit", "all", "deprecated", "bit-groups", "resource-groups", or "languages"
format — "text" (default) or "json"
bit — filter to a specific bit type (when infoType is "bit")
pretty — prettify JSON output (default: false)
indent — indent size for pretty JSON (default: 2)
includeDeprecated — include deprecated bits in "list"/"all" (default: false)
onlyDeprecated — restrict "list"/"all" to deprecated bits (default: false)
full — complete detail for "bit"/"all" (default: false = compact view)
language — BCP-47 tag(s) for display names: one ("de"), several (["de", "fr"] or "de,fr"), or "all" — exported as ALL_LANGUAGES (default: English). More than one adds a titles map. See Display names and languages below

"bit-groups" / "resource-groups" return the search/filter catalogs — each group's key, translated title, description, optional aliases and subgroupOf, and its members. They are available in EVERY build: the group catalog and per-bit titles are "descriptive" data, which the lean wasm variants carry too. Deprecated members are excluded by default and MARKED when included with includeDeprecated: true; a search index matching already-published content needs them, because a migrating bit is re-emitted under its target's name.

To derive quiz categories, intersect a bit's bitGroups with the groups carrying subgroupOf: "quizzes" — subgroupOf is metadata and never implies membership, so every member of a subgroup also declares the parent.

Only the META fields — BIT and TAG descriptions, group-inheritance provenance and raw mapping patterns (the info-meta cargo feature) — depend on the build. They are reported by the native CLI and the wasm full variant; browser-full and bitmark-json omit them to stay small (the bit descriptions alone are 14 KB gzip of the download). Those are absent structurally: the key is missing, never null or empty. Everything else info returns — including bit titles and the group catalogs — is identical in every variant.

Display names and languages

title — on bits, bit groups and resource groups — is a DISPLAY name, meant to be shown. Pass language to get it in another language:

info({ infoType: "bit-groups", format: "json", language: "de" });

Resolution narrows one subtag at a time (de-CH → de), then falls back to the English title, then to the technical key. A partially translated language is therefore safe: untranslated entries stay English rather than blank. Technical identity is never translated — name, key, tag names and member lists are the same in every language, so you can key off them freely.

Several languages at once. Building a multilingual index should not mean walking every bit once per language, so language also takes a list, or "all" — exported as the ALL_LANGUAGES constant, so you can name it rather than retype the literal:

import { info, ALL_LANGUAGES } from "@gmb/bitmark-parser";

const groups = JSON.parse(
  info({ infoType: "bit-groups", format: "json", language: ALL_LANGUAGES }),
);
groups[0].titles; // { en: "Cloze", de: "Lückentext", … }

// equivalent, and the idiomatic JS spelling of a list
info({ infoType: "bit-groups", format: "json", language: ["de", "fr"] });

// what "all" expands to
JSON.parse(info({ infoType: "languages", format: "json" })); // ["de","en",…]

Getting every name in one call. bit-groups carries each member's display name, so one request answers "every bit type, what it is called, and which categories it is in":

const groups = JSON.parse(
  info({
    infoType: "bit-groups",
    format: "json",
    includeDeprecated: true,
    language: "all",
  }),
);
groups[0].bitTypes[0]; // { name: "assignment", title: "Assignment", titles: {…} }

That is ~132 KB. Do not reach for infoType: "all" to get names — it carries the full per-tag detail for every bit (~5 MB) and is the wrong tool for this.

More than one language adds a titles map, and nothing else. It carries en when an English title exists, and is otherwise sparse: a requested language with no translation is absent rather than filled with English, so a gap stays visible. It is keyed by the tag you ASKED for — request de-CH where only de exists and you read titles["de-CH"].

A single tag (or none) renders exactly as it always has, with no titles. The shape follows your REQUEST rather than which language you picked: "all" adds titles even on a build that resolves nothing but English.

title is the name in your PRIMARY language — the first one you named. Name none and it is English; ask for "all" and it is English too, because with everything already in titles there is no non-arbitrary "first". So title tracks how you asked: consistent for any consumer that asks the same way every time, which in practice is all of them — a UI names one language, an index asks for "all".

title stays optional, as its type says — it is the name in the language you asked for, and a few bits have one in some languages but not in English (the internal _comment and _error). Every other key is the same in every language.

Which languages resolve depends on the build. The native CLI and the wasm full variant bake the table in; browser-full and bitmark-json do not, because ~76 KB of names is not something a browser should download without asking. Those variants — and any build whose translations you want to override — take the file at runtime:

import { init, register, info } from "@gmb/bitmark-parser";
import translations from "@gmb/bitmark-parser/translations" with { type: "json" };

await init({ feature: "browser-full" });
register({ type: "translations", data: JSON.stringify(translations) });
info({ infoType: "bit", bit: "cloze", format: "json", language: "de" });

register REPLACES rather than merges: a second call supersedes the first entirely, and the built-in table (where there is one) stays underneath, so a key your file omits still resolves. It applies process-wide and survives an init variant swap. The language option stays per-call, because that is the axis that actually varies — one process may serve many.

The same file backs all three routes: @gmb/bitmark-parser/translations (an opt-in subpath, so nothing pays for it unasked), the CLI's --translations <file> flag, and what the full builds bake in. It holds non-English names only — English lives in the config title, which every build carries.

The JSON shape is a stable contract; a change to it is a semver-major release. Language changes VALUES only — the key set is identical for every language. Parse it and assert the matching exported type:

import { info, type DeprecatedInfo } from "@gmb/bitmark-parser";

const deprecated = JSON.parse(
  info({ infoType: "deprecated", format: "json" }),
) as DeprecatedInfo;

Result types: BitListInfo, BitInfo (CompactBitInfo | FullBitInfo), AllBitsInfo, DeprecatedInfo, BitGroupsInfo, ResourceGroupsInfo, and InfoErrorResult — returned in place of a result when infoType is "bit" and the name is unknown. info itself returns a string because it renders a document (format: "text", pretty and indent only mean anything for one).

The text output is NOT a contract — it is for humans and may change in any release. Never parse it; every field it shows exists in the JSON form. The normative definition of the JSON is .zen/specs/API-INF-info-output.tsp.

version(): string

Return the library version string.

Typed API (generated types)

TypeScript types for the parser's JSON are generated from bitmark.json (npm run schema:ts, automatic in the build): per-bit types (ArticleBit, ClozeBit, …) form the Bit discriminated union (narrow on bit.type), AnyBit is the loose merged view, BitEntry is the {bit, parser?, bitmark?} envelope, and resource/card content have named types (ImageResource, FlashcardBitCard, …).

import {
  bitmarkToObjects,
  objectsToBitmark,
  patchEntry,
  type ArticleBit,
} from "@gmb/bitmark-parser";

// bitmark → typed entries
const entries = bitmarkToObjects("[.article]\nHello **world**");
if (entries[0].bit.type === "article") console.log(entries[0].bit.body);

// author bits as typed literals (no builder needed) → bitmark
const article: ArticleBit = { type: "article", body: "Hello" };
const bitmark = objectsToBitmark([article]);

// typed round-trip editing
entries[0].bit.id = ["42"];
const edited = objectsToBitmark(entries);

// patch values typed from the literal path
const patch = patchEntry("id", "append", "1234");

See examples/ for runnable scripts covering the whole API (executed by npm test, so they stay correct). Generated API documentation (TypeDoc) is published at https://getmorebrain.github.io/bitmark-parser/docs/api/ with each release (the site root hosts the bitmark language docs), or build it locally with npm run docs (→ docs/api/) and preview it with npm run docs:serve (http://localhost:8080, --port to change).

JSON Schema

The package also ships a JSON Schema (Draft 2020-12) describing the parser's JSON output for every bit type, generated from the same bitmark.json as the TypeScript types (npm run schema:json, automatic in the build):

import schema from "@gmb/bitmark-parser/schema.json" with { type: "json" };
// or (CJS): const schema = require("@gmb/bitmark-parser/schema.json");

Use it to validate parser output, or to enumerate the keys a bit type can carry — e.g. when auditing a consumer for the keys it must materialise itself under optimized output (see the migration guide). The key's type is all a consumer needs: configured (non-natural) defaults are always materialised in the output, so an absent key always means the natural default of its type (false, 0, "", [], {}).

Note: deprecated bit types with a configured migration target (the *-collapsible family) are re-emitted under the target type with isCollapsible: true — consumers keyed on the old type strings see the base type instead. collapsible itself is unaffected.

Property number values

A tag whose configured format is number accepts what JavaScript's Number(value) accepts, minus the values that are not finite:

surrounding whitespace, a leading +, leading zeros (01), a trailing point (1.) and a leading point (.5);
an exponent in either case (1e2, 1E2);
hexadecimal, binary and octal literals (0x1F, 0b101, 0o701) — prefix case-insensitive, with no sign, fraction or exponent.

NaN, Infinity, an exponent beyond ±4000, and a radix literal past 64 bits are rejected: the tag is dropped from the output and a property-format-mismatch warning names it, exactly like any other value that does not fit its format.

Accepted values print in JavaScript's JSON.stringify form — 0.5, 1.5, 100, 1e+21, 1e-7 — so [@width: 1.50] emits 1.5 and [@width:0x10] emits 16. Values with more than 17 significant digits, including integers at or above 2^63, keep the digits you wrote rather than being rounded through a floating-point double.

Legacy API (bpg-compatible)

A compatibility facade for consumers migrating from @gmb/bitmark-parser-generator (bpg v5.31.x). Only the import changes:

// before: import { BitmarkParserGenerator } from "@gmb/bitmark-parser-generator";
import { BitmarkParserGenerator } from "@gmb/bitmark-parser/legacy";
// or:    import { legacy } from "@gmb/bitmark-parser";

const bpg = new BitmarkParserGenerator();
const json = bpg.convert("[.article]\nHello **world**", {
  outputFormat: "json",
});

Supported (bpg-compatible signatures and return shapes): convert, upgrade, info, version, breakscapeText / unbreakscapeText, convertText, textAstToPlainText, extractPlainText, convertHtmlTable, createAst, bitmarkTextParse, the Builder / ResourceBuilder classes, the Input / Output / InputFormat / BitType (config-derived) / TextMarkType / TextNodeType enums, and the option / JSON model types (vendored from bpg, exposed only via legacy). Node file input/output (Input.file, outputFile) works as in bpg; the browser build throws on file writes, as bpg does.

Differences (all loud, never silent):

bitmark v3 only — bitmarkVersion: 2, cardSetVersion: 1 and jsonOptions.textAsPlainText: true throw.
Strict option policy — options with no equivalent here throw a LegacyUnsupportedError naming the option (explicitTextFormat, noBreakscaping, debugGenerationInline). The option types are narrowed to the supported values too (e.g. bitmarkVersion?: 3, noBreakscaping?: false), so TypeScript consumers get a build-time error rather than only the runtime throw — JS callers still get the loud runtime error. Exception: bitmarkParserType is accepted and ignored — bpg itself ignores it. jsonOptions.enableWarnings is accepted (parser warnings are always carried in the output envelope). bitmarkOptions.prettifyJson is honoured (pretty-prints the embedded JSON bodies of json-format bits in bitmark output, as bpg did).
Unknown properties — included in JSON output by default and excludable via jsonOptions.excludeUnknownProperties, exactly as bpg (arrays, last keys, _-prefix on collision with a real key, never converted back to bitmark). One deliberate divergence: the per-occurrence warning text names the new API's inverse flag (includeUnknownProperties), not bpg's excludeUnknownProperties.
The AST is bit JSON — createAst / Output.ast / Builder.buildBit produce the bit JSON ("pseudo-AST"), not bpg's internal TS node model. It round-trips through convert, but bpg's Ast walker (and the generator/parser/writer internals) throw as unsupported. buildBit's resources input follows bpg's routing: on bits whose config maps a repeatable resource group onto a collection key (e.g. images / logos), the entries land under that key; on other bits the first entry becomes the single resource (as bpg).
convertHtmlTable converts HTML <table> markup ↔ table bits via the core's html mapping (the <bitmark-bit> envelope is added/stripped at the boundary). The markup produced inside <table> follows this parser's html mapping and may differ from bpg's in detail. keepUnknownTags / noBreakscaping throw.
version() reports this package's version (6.x), not a bpg version.

CLI

The CLI binary is bitmark.

# Convert bitmark to JSON (auto-detects format)
bitmark convert input.bitmark
bitmark convert input.bitmark -o output.json

# Convert JSON to bitmark
bitmark convert input.json -o output.bitmark

# Convert with options
bitmark convert input.bitmark --mode full --pretty --warnings

# Include unknown properties in the bit JSON (bitmark → JSON only)
bitmark convert input.bitmark --include-unknown-properties

# Parse a bitmark file to JSON
bitmark convert input.bitmark --input-format bitmark --output-format json
bitmark convert input.bitmark --input-format bitmark -o output.json
bitmark convert input.bitmark --input-format bitmark --output-format json --mode full

# Generate bitmark from JSON
bitmark convert input.json --input-format json --output-format bitmark -o output.bitmark

# Import a NISO-STS standards document (config-driven forward import)
bitmark convert standard.xml --input-format xml-niso-iec --output-format bitmark

# Re-emit in the same format at optimized / full mode (canonical form)
bitmark canonicalize input.bitmark
bitmark canonicalize input.json --mode full

# Apply a patch document (per-bit patches by index/id, remove, insert, move;
# a bare array of entries still patches every bit)
bitmark transform input.json --patch patches.json   # alias: patch

# Semantic diff of two documents, rendered as bitmark
bitmark diff old.bitmark new.bitmark                       # unified-style text
bitmark diff old.bitmark new.bitmark --stat --exit-code    # counts; exit 1 when they differ
bitmark diff old.bitmark new.json --output-format json     # per-bit ops, revert, hunks
bitmark diff old.bitmark new.bitmark --output-format patch > d.patch
bitmark transform old.bitmark --patch d.patch --output-format bitmark   # reproduces new
# --context N, --locate, --similarity F, --bbox-tolerance N, --color auto|always|never

# Parser-derived highlighting: LSP semantic tokens (bitmark input only)
bitmark convert input.bitmark --input-format bitmark --output-format semantic-tokens
bitmark convert input.bitmark --output-format semantic-tokens --tokens-layout absolute --position-encoding utf-8 --pretty

# Dump the lexer token stream as JSON (debug tooling; bitmark input only)
bitmark convert input.bitmark --input-format bitmark --output-format lex --pretty

# Breakscape / unbreakscape text
bitmark breakscape input.txt
bitmark breakscape input.txt --format plainText --location tag
bitmark unbreakscape input.txt

# Query bit type information
bitmark info
bitmark info all -f json --pretty
bitmark info --bit article          # compact: one line per tag (format, count, default)
bitmark info --bit article --full   # full detail: defaults, provenance, keys, card set
bitmark info deprecated             # deprecated bits + migration targets
bitmark info list --all             # include deprecated bits in the listing
bitmark info list --deprecated      # only deprecated bits

All commands support -o, --output <file> and -a, --append flags. Commands that accept input support file paths, literal strings, or stdin (when no arguments are given).

Browser Usage

The package includes pre-built browser bundles with the WASM module.

CDN (jsdelivr / unpkg)

<script type="module">
  import init, {
    convert,
  } from "https://cdn.jsdelivr.net/npm/@gmb/bitmark-parser@latest/dist/browser/bitmark-parser.min.js";

  await init();

  const json = convert("[.article]\nHello **bold**", {
    inputFormat: "bitmark",
  });
  console.log(json);
</script>

The CDN bundle carries all three variants' JS glue and fetches only the selected variant's .wasm (from dist/browser/wasm/). A bare init() loads browser-full; init({ feature: "bitmark-json" }) fetches the smallest build, and a later init({ feature: "full" }) upgrades in place (see WASM variants).

Bundler (webpack / vite)

import init, { convert } from "@gmb/bitmark-parser/browser";

await init();
const json = convert("[.article]\nHello", { inputFormat: "bitmark" });