TypeScript transform backend, published as @pydantic/llm-babel

Transpiles the llm-transform DSL (a pure Python AST, llm_transform/dsl/nodes.py) to TypeScript and proves conformance against the shared golden corpus. This is the backend a Node gateway consumes.

The reference interpreter (llm_transform/dsl/interp.py) is the source of truth; every helper in src/runtime.ts mirrors its _* handler exactly, and the corpus comparison is byte-for-byte via canonicalJson.

Install and use

npm install @pydantic/llm-babel

import { translate, parseJsonTokens, canonicalJson, capabilitiesFor, reconcile } from '@pydantic/llm-babel';

const wire = parseJsonTokens(body);                                  // wire JSON in, numbers kept as exact tokens
const out = translate('openai-chat', 'anthropic-messages', wire);   // source wire -> IR -> target wire
const bytes = canonicalJson(out);                                    // canonical JSON out

Plain ESM with type declarations, no runtime dependencies. Node 21 or newer (or any evergreen engine): parseJsonTokens uses the ES2023 JSON.parse source-text reviver. The public surface is src/index.ts: the codecs (decode/encode/translate and their response, error, and stream forms), reconcile and betas, the capability and routing data, and the typed IR (ir builders, IRRequest and friends).

The package is built by npm run build (tsconfig.build.json emits dist/ from src/index.ts and what it imports; the corpus runner and parity scripts stay out) and published by the repository's release workflow on a vX.Y.Z tag, see docs/status.md. dev/pack_smoke.sh installs the packed tarball into a scratch project and imports it under plain Node and a strict NodeNext tsc, so a consumer-breaking change fails here first.

Layout

• src/index.ts, hand-written: the curated public surface (re-exports only).
• src/runtime.ts, hand-written: the Value type (JSON value | MISSING sentinel, a unique symbol, never JSON null), TransformError{code, detail}, canonicalJson, and one helper per AST node.
• src/generated.ts, generated, do not edit. One function per program, every format × direction in the manifest (e.g. openaiChatDecode, anthropicMessagesEncode, geminiStreamStep, reconcile).
• src/translate.ts, the format registry and translate(source, target, input) (always source → IR → target through the hub).
• src/corpus.ts, the corpus runner (decode / encode / roundtrip).

Runnable examples built on this backend (a cross-provider translation gateway, a typed-IR adapter) live in examples/typescript, kept out of the backend so this stays pure core.

The Python emitter lives at llm_transform/dsl/transpile_ts.py (TsEmitter).

Regenerate generated.ts

From the llm-transform package root (so the llm_transform package is importable):

python3 -m llm_transform.dsl.transpile_ts
# or, from this directory:
npm run generate

This walks every program in the manifest (dsl/transpile.py::program_specs(), all formats × directions) and writes src/generated.ts. Rerun after any change to the DSL programs or the emitter. (dev/regen.sh regenerates all backends at once.)

Run the corpus

npm install        # first time: installs tsx, typescript, @types/node
npm test           # === npm run corpus === tsx src/corpus.ts

Expected output (exit 0 on success):

[PASS] openai_to_anthropic__system_tool_roundtrip :: decode
[PASS] openai_to_anthropic__system_tool_roundtrip :: encode
[PASS] openai_to_anthropic__system_tool_roundtrip :: roundtrip
…
all checks passed

The runner loads ../../llm_transform/corpus/cases/*.json directly, so the TS and Python backends assert against the identical golden files.

Type-check with npm run typecheck (tsc --noEmit), and format with npm run format (npm run format:check is what CI runs). Prettier is pinned to an exact version because package-lock.json is not checked in, so a range would reformat the tree on a prettier release with no code change. The generated files are left to their generators (.prettierignore); bash dev/regen.sh is what formats those.

Semantics notes

• MISSING is a unique symbol, distinct from JSON null. Record omits MISSING fields; ListLit/MapList drop MISSING elements; Present is "not MISSING and not null"; Coalesce returns the first non-MISSING non-null.
• canonicalJson sorts object keys by Unicode code point, not by UTF-16 code unit. JavaScript's String.prototype.sort and JSON.stringify do not sort, and default string comparison diverges from Python for supplementary characters (code point > 0xFFFF), so key sorting uses an explicit code-point comparator (Array.from iteration). Compact separators; NaN/Infinity are rejected with a TransformError rather than emitted as null.
• TransformError carries a stable code (unmatched / unmapped for Match/Lookup with no default; the Fail node's own code) so the corpus can assert which failure occurred.
• Numbers are preserved tokens (NumberToken): wire JSON is parsed with the ES2023 JSON.parse source-text reviver (parseJsonTokens), which captures each number's exact source token even though the parsed value has lost precision/format. canonicalJson emits the token verbatim, so 1.0/1e-7/-0.0/large integers are byte-identical with the Python and Rust backends (the old JS 1.0→1 collapse is gone); equality (_eq) is numeric, so 1 === 1.0. Runtime floor: the source reviver needs Node ≥21 / an evergreen JS engine; an older runtime would need a custom number scanner in parseJsonTokens.

Scope

A complete transpilation target, same as the Rust backend: every program in the manifest (all 8 formats × decode/encode/decode_response/encode_response/decode_error/encode_error/stream_step/betas, plus reconcile) is transpiled to TypeScript, byte-identical to the Python reference, the corpus runs with 0 skips.

Unimplemented nodes: none. All 23 AST nodes are transpiled: Var, Lit, Get, Index, Record, ListLit, Present, Coalesce, If, Eq, Kind, Match, Lookup, Join, IntToStr, MapList, MapListIndexed, Filter, Flatten, GroupAdjacent, JsonParse, JsonSerialize, Fail.