intl-content-tree

Parse structured content (markdown, JSON) into hashable trees for incremental localization.

Given a source file and its derivatives (translations), determine exactly what changed and whether each change requires retranslation or can be propagated automatically.

Install

npm install intl-content-tree

How It Works

1. Parse a content file into a tree of nodes, each classified as translatable or inert
2. Serialize the tree as a manifest (compact JSON with content hashes)
3. Diff a new tree against a stored manifest to get a precise changeset
4. Route each change: translatable drift needs retranslation, inert drift can be copied

Source file  -->  [parse]  -->  Tree  -->  [serialize]  -->  Manifest
                                            |
New source   -->  [parse]  -->  Tree  -->  [diff]  -->  Changeset
                                                         |
                                          unchanged | inertDrift | structuralDrift
                                          translatableDrift | added | removed
                                          renamed | reordered

Quick Start

import { parseMarkdown, diff, serialize, deserialize } from "intl-content-tree"

// Parse the current source
const tree = parseMarkdown(sourceContent)

// Compare against a stored manifest
const oldTree = deserialize(storedManifest)
const changes = diff(oldTree, tree)

// Route changes
for (const entry of changes.inertDrift) {
  // URL or code changed -- copy from source to all translations
}

for (const entry of changes.translatableDrift) {
  // Prose changed -- send this section for retranslation
}

for (const entry of changes.added) {
  // New section -- translate from scratch
}

// Store the new manifest
const manifest = serialize(tree, "path/to/file.md")

Parsers

Markdown

Parses markdown/MDX into a tree using heading structure. Each heading creates a group node; content between headings becomes child elements.

import { parseMarkdown } from "intl-content-tree"

const tree = parseMarkdown(content, {
  depth: "group",                    // "group" (default) or "element"
  translatableAttributes: ["title", "alt", "description"],
}, {
  headingIdPattern: /\{#([^}]+)\}/,  // custom heading ID syntax (default)
  proseFenceTags: ["md", "text"],    // code fences treated as translatable
})

What gets classified:

| Content | Classification | Example | |---------|---------------|---------| | Prose paragraphs | translatable | "This function returns a promise" | | Heading labels | translatable (separate labelHash) | "Getting Started" | | Link display text | translatable | [click here](...) | | Image alt text | translatable | ![diagram of the flow](...) | | Code comments | translatable | // Initialize the connection | | Frontmatter fields | per config | title: translatable, lang: inert (see below) | | URLs and paths | inert | https://example.com | | Code bodies | inert | const x = 1 | | Component attributes | per config | title= translatable, href= inert | | ICU variables | inert | {username}, {count, plural, ...} |

Frontmatter

The block between the leading --- fences is parsed as YAML. Each top-level key becomes a root-level node:

| YAML | Node shape | |------|-----------| | Scalar (title: Foo, published: 2023-01-01, abstract: >-) | one element node, frontmatter:<key> | | Sequence (block or flow) | a section node frontmatter:<key> with one child per item, IDs 0, 1, ... | | Mapping | a section node frontmatter:<key> with one child per subkey |

---
title: "Agents: a field guide"   # quotes and trailing comments are not content
summaryPoints:                   # translatable per config -> items translatable
  - Agents hold their own keys
  - Agents pay for their own compute
tags: ["solidity", "vyper"]      # flow and block spellings hash identically
author:
  name: Ada Lovelace             # classified by its own subkey
  title: Protocol researcher
---

Sequence items inherit the field's translatability; mapping values are classified by their own subkey. Because items are real child nodes, adding, removing, rewriting, or reordering one moves the field hash and the root hash. Scalar values are kept as written -- quotes stripped, block scalars folded, dates, numbers, and booleans never re-serialized (1.10 stays 1.10).

If the block is not valid YAML, parsing falls back to a single-line key: value scan and the root node carries meta.frontmatterParseError = "true".

JSON

Parses JSON key-value files. Top-level keys become group IDs. Nested objects create sub-groups.

import { parseJson } from "intl-content-tree"

const tree = parseJson(content, {
  depth: "group",
  translatableAttributes: ["title", "alt"],
}, {
  markdownValueDetector: (key, value) => value.includes("\n") && /\[.+\]\(.+\)/.test(value),
})

String values containing HTML, markdown, or ICU MessageFormat variables ({name}, {count, plural, ...}) are detected and decomposed automatically. URLs and variable names are classified as inert; prose between them is translatable.

Diff Engine

Compares two trees and classifies every change:

| Classification | Meaning | Action | |---------------|---------|--------| | unchanged | No changes | Nothing (check labelHashChanged for heading-only edits) | | inertDrift | Only URLs, code, or paths changed | Copy from source -- no translation needed | | structuralDrift | Nodes added/removed but no prose text changed | Propagate structure -- no translation needed | | translatableDrift | Prose or translatable content changed | Retranslate this section | | added | New group not in old tree | Translate from scratch | | removed | Group no longer in source | Remove from translations | | renamed | Same content, different ID | Update ID in translations | | reordered | Same content, different position | Reorder in translations |

Three-Hash Model

Each group node carries three hashes:

• contentHash: hash of translatable body content (prose, links, images)
• anchorHash: hash of inert content (URLs, code, paths)
• labelHash: hash of the group label (heading text in markdown, key name in JSON)

This separation enables precise change detection:

const changes = diff(oldTree, newTree)

// Heading text changed, body unchanged
const headingOnly = changes.unchanged.filter(e => e.labelHashChanged)

// Body changed (needs retranslation)
const bodyChanged = changes.translatableDrift

// URL changed (scriptable copy, no retranslation)
const urlChanged = changes.inertDrift

Validation

Check how well a tree supports incremental tracking:

import { validate } from "intl-content-tree"

const result = validate(tree)
// {
//   totalSections: 15,
//   stableIds: 12,
//   autoSlugs: 3,
//   duplicateIds: [],
//   coverage: 80
// }

Frontmatter sequences and mappings are section-shaped nodes, but they are schema, not authored groups, so they are excluded from these counts (and from getContainingSection).

Stable Identifiers

The effectiveness of incremental tracking depends on stable, explicit identifiers for content groups.

For markdown, this means custom heading IDs:

## Getting Started {#getting-started}

Content here...

## Installation {#installation}

More content...

Without explicit {#id} anchors, IDs are auto-generated from heading text (e.g., _auto:getting-started). These break when headings are edited and cannot match across languages (translations produce different slugs).

| ID strategy | Rename detection | Cross-language matching | Heading edits | |------------|-----------------|----------------------|---------------| | Explicit {#id} | Yes | Yes | Stable | | Auto-slug from text | No | No | Breaks key | | Position-based | No | No | Breaks on insert/remove |

For JSON, keys naturally serve as stable identifiers.

Utilities

import { getInertValue, getNodeByPath, hasChanges, walk } from "intl-content-tree"

// Get an inert value by path (e.g., a URL for propagation)
const url = getInertValue(tree, "getting-started/link:0")

// Find a node by path
const node = getNodeByPath(tree, "installation/prose:0")

// Quick check: did anything change?
if (hasChanges(tree, storedManifest)) {
  // Run full diff...
}

// Iterate all nodes
for (const node of walk(tree)) {
  console.log(node.id, node.contentType)
}

Serialization

Trees serialize to compact manifests for storage:

import { serialize, deserialize } from "intl-content-tree"

const manifest = serialize(tree, "docs/getting-started.md")
// {
//   version: 2,
//   sourceFile: "docs/getting-started.md",
//   generatedAt: "2026-04-04T...",
//   rootHash: "a1b2c3d4e5f6",
//   tree: { contentHash, anchorHash, labelHash?, children, childrenOrder }
// }

const restored = deserialize(manifest)

Manifests use childrenOrder arrays to preserve sequence (object key order is not guaranteed in JSON).

Manifest version 2

MANIFEST_VERSION is 2 as of v0.4.0: frontmatter sequences and mappings are now trees of item nodes instead of a single opaque line, so their hashes moved. Stored v1 manifests must be re-derived from the source they were generated against -- comparing a v1 manifest against a v2 tree reports drift on every file whose frontmatter has a sequence, a mapping, a multi-line scalar, a quoted scalar, or a trailing comment. Files whose frontmatter is only plain single-line scalars keep their v1 hashes.

What This Package Does NOT Do

• Does not translate anything
• Does not call any AI/LLM APIs
• Does not handle file I/O (caller provides content as strings)
• Does not enforce heading IDs (supports explicit and auto-generated)
• Does not know about any specific project's components or schema

License

MPL-2.0