yuku-parser

A high-performance, spec-compliant JavaScript/TypeScript parser written in Zig, powered by Yuku.

• Install
• Usage
• ESTree / TypeScript-ESTree
• AST Types
• Path helpers
• Tokens
• Walking the AST
• Semantic analysis
• Options
• Result
• Comments

Install

npm install yuku-parser

Usage

import { parse } from "yuku-parser";

const result = parse("const x = 1 + 2;");

console.log(result.program);     // ESTree / TypeScript-ESTree Program node
console.log(result.diagnostics); // errors and warnings

ESTree / TypeScript-ESTree

For JavaScript and JSX, the AST is fully conformant with the ESTree specification, identical to what Acorn produces.

For TypeScript, the AST conforms to the TypeScript-ESTree format used by @typescript-eslint.

Yuku produces exactly the AST that Oxc produces, for both JS and TS.

On top of the base specs, the AST also carries:

• Stage 3 decorators.
• Stage 3 import defer and import source. Dynamic forms (import.defer(...), import.source(...)) are represented as an ImportExpression with a phase field set to "defer" or "source", following the ESTree convention.
• A non-standard hashbang field on Program for #!/usr/bin/env node lines.

Any other deviation from Acorn's ESTree or @typescript-eslint's TypeScript-ESTree would be considered a bug.

AST Types

All AST node types are exported directly from this package:

import type { Node, Statement, Expression, Identifier } from "yuku-parser";

The Node union type covers every possible AST node. Individual types like Statement, Expression, Declaration, etc. are also available. See the full list in the type definitions.

Path helpers

Two small helpers are exported for resolving the lang and sourceType options from a file path:

import { langFromPath, sourceTypeFromPath } from "yuku-parser";

langFromPath("foo.tsx");           // "tsx"
langFromPath("types.d.ts");        // "dts"
sourceTypeFromPath("foo.cjs");     // "commonjs"
sourceTypeFromPath("foo.mjs");     // "module"

Tokens

tokens: true keeps every token the parser consumed. The result carries a TokenList, a view over the parser's token table. Nothing is decoded up front, a token is an index, and each accessor is one typed-array read.

import { parse, TokenKind } from "yuku-parser";

const { program, tokens } = parse(source, { tokens: true });

for (let i = 0; i < tokens.length; i++) {
  if (tokens.kind(i) === TokenKind.Arrow) console.log(tokens.start(i), tokens.text(i));
}

tokens.kind(i)            // one of the 160 kinds in TokenKind, see the list below
tokens.text(i)            // source text, a string literal keeps its quotes
tokens.start(i)           // UTF-16 offsets, like nodes
tokens.end(i)

tokens.isKeyword(i)       // reserved words and contextual keywords
tokens.isReserved(i)      // reserved unconditionally or in strict mode
tokens.isUnconditionallyReserved(i)   // can never be an identifier
tokens.isStrictModeReserved(i)        // let, static, implements, ...
tokens.isIdentifierLike(i)
tokens.isNumericLiteral(i)
tokens.isBinaryOperator(i)
tokens.isLogicalOperator(i)
tokens.isUnaryOperator(i)
tokens.isAssignmentOperator(i)
tokens.precedence(i)      // binary precedence, 0 when none

tokens.newlineBefore(i)   // a line terminator precedes it, what ASI looks at
tokens.escaped(i)         // \u0061sync is an async token with this set
tokens.invalidEscape(i)   // a template chunk whose cooked value is undefined
tokens.loneSurrogate(i)   // a string with an unpaired surrogate

The queries take a node and answer with an index, -1 when there is none. They are binary searches, so they replace a token store without building one.

tokens.range(node)       // [from, to) of the tokens inside the node, empty for a node inside one token
tokens.first(node)
tokens.last(node)
tokens.before(node)      // last token ending at or before it, also takes an offset
tokens.after(node)       // first token starting at or after it, also takes an offset
tokens.at(offset)        // the token containing an offset

Why an index and not an array of objects? On a 1 MB file, about 215,000 tokens, tokens: true adds 1 ms to the parse and scanning every kind(i) another 0.4 ms. Building an object per token would add 8 ms and 20 to 50 MB of heap, which is what tokens cost in espree, acorn, and Babel, and 70 ms in typescript-estree.

TokenKind has 160 kinds, one per punctuator, literal form, keyword, and identifier form. The full list is tokens.d.ts.

Tokens are as the parser resolved them: a regex is one RegexLiteral, and the >> closing a nested generic is two GreaterThan. Comments are not tokens.

Walking the AST

The AST is standard ESTree, and yuku-ast walks it with typed visitors, alias groups, in-place mutation, and syntactic utilities.:

import { parse } from "yuku-parser";
import { walk } from "yuku-ast";

const { program } = parse(`console.log("hello");`);

walk(program, {
  Identifier(node) {
    console.log(node.name);
  },
});

Semantic analysis

yuku-analyzer builds on this parser and adds full semantics: scopes, symbols, resolved references, closure analysis, and cross-file module linking, computed natively in the same pass. Its walk carries the semantic model in context (ctx.scope, ctx.symbol, ctx.reference). See the analyzer documentation.

Options

All options are optional.

const result = parse(source, {
  sourceType: "module",
  lang: "jsx",
  preserveParens: true,
  semanticErrors: false,
  attachComments: false,
});

| Option | Values | Default | Description | | ---------------------------- | ----------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------- | | sourceType | "module", "script", "commonjs" | "module" | Module mode enables import/export, import.meta, top-level await, and strict mode. CommonJS mode parses script code whose top level behaves like a function body, allowing top-level return, new.target, and using. | | lang | "js", "ts", "jsx", "tsx", "dts" | "js" | Language variant controls which syntax extensions are enabled. | | preserveParens | true, false | true | Keep ParenthesizedExpression nodes in the AST. When false, parentheses are stripped and only the inner expression is kept. | | semanticErrors | true, false | false | Run semantic analysis and report semantic errors alongside syntax errors. | | attachComments | true, false | false | Also attach each comment to its host AST node. The flat result.comments list is always present. See Comments. |

Result

parse returns a ParseResult:

interface ParseResult {
  program: Program;
  comments: Comment[]; // every comment in source order
  diagnostics: Diagnostic[];
}

The parser is error-tolerant: an AST is always produced even when diagnostics are present.

Diagnostics

Diagnostics cover both syntax errors found during parsing and, when semanticErrors is enabled, semantic errors that require scope and binding information (e.g. duplicate let declarations, break outside a loop, unresolved private fields).

Each diagnostic includes:

• severity: "error", "warning", "hint", or "info"
• message: description of the issue
• help: fix suggestion, or null
• start / end: byte offsets into the source
• labels: additional source spans with messages for context

Semantic Errors

By default, the parser only reports syntax errors. Semantic errors require resolving scopes and bindings, which is done in a separate AST pass. Enable this with the semanticErrors option:

const result = parse(`let x = 1; let x = 2;`, { semanticErrors: true });
// result.diagnostics will include "Identifier `x` has already been declared", etc.

This incurs a very small performance overhead. If your build pipeline already handles semantic validation (e.g. through a linter or type checker), you can leave this off for faster parsing.

Comments

Every comment is always in result.comments, a flat list in source order with each comment's source span:

const { comments } = parse(`// a line comment\nconst x = 1; /* a block comment */`);

for (const c of comments) {
  console.log(c.type, JSON.stringify(c.value), c.start, c.end);
}
// Line " a line comment" 0 17
// Block " a block comment " 31 52

Each entry is:

interface Comment {
  type: "Line" | "Block";
  value: string; // body without delimiters
  start: number; // byte offset, delimiter included
  end: number;   // byte offset, delimiter included
}

The span (start/end) covers the whole comment, delimiters included, so source.slice(c.start, c.end) returns the raw text.

Attaching comments to nodes

Set attachComments: true to also hang each comment on the AST node it sits next to, read off node.comments. This is what a codegen pass needs, since attached comments move with their node through transforms.

const { program } = parse(`// header\nfunction foo() {} // trailing`, { attachComments: true });

const fn = program.body[0];
for (const c of fn.comments ?? []) {
  console.log(c.position, c.type, c.value);
}
// before Line " header"
// after  Line " trailing"

Each attached comment is:

interface AttachedComment {
  type: "Line" | "Block";
  position: "before" | "after" | "inside";
  sameLine: boolean;
  value: string; // body without delimiters
}

position is where the comment sits relative to its host: "before" (leading), "after" (trailing), or "inside" (interior to an otherwise empty host like function f() { /* hi */ }). sameLine is true when the comment shares a source line with the host's adjacent edge.

License

MIT