@agentscript/parser-javascript

Hand-written TypeScript parser for the AgentScript language. Error-tolerant recursive descent parser with an indentation-aware lexer, Pratt expression parsing, and full CST (Concrete Syntax Tree) output.

Features

• Zero runtime dependencies
• Error-tolerant: never crashes, always produces a CST
• NEWLINE and DEDENT tokens act as unconditional recovery points
• Implements the SyntaxNode interface consumed by dialect, LSP, monaco, and agentforce packages
• Syntax highlighting via CST walk (no tree-sitter query engine needed)

Usage

import { parse, parseAndHighlight } from '@agentscript/parser-javascript';

// Parse source code into a CST
const { rootNode } = parse(source);

// Parse and get syntax highlighting captures in one call
const captures = parseAndHighlight(source);

API

| Export | Description | |---|---| | parse(source) | Parse source and return { rootNode: CSTNode } | | parseAndHighlight(source) | Parse and return HighlightCapture[] | | highlight(node) | Walk an existing CST and produce highlight captures | | CSTNode | CST node class implementing SyntaxNode | | TokenKind | Enum of all token types |

Scripts

pnpm build        # Compile TypeScript
pnpm test         # Run test suite (vitest)
pnpm test:watch   # Run tests in watch mode
pnpm bench        # Run vitest benchmarks
pnpm perf         # Run detailed performance analysis with timing output
pnpm perf:report  # Generate PERFORMANCE.md report

Testing

Tests are located in test/ and use vitest:

• corpus.test.ts — Parses tree-sitter corpus files and compares s-expression output
• parity.test.ts — Compares parser-javascript and tree-sitter on corpus inputs
• fuzz.test.ts — Random mutations of corpus inputs, checks parser-javascript invariants
• fuzz-parity.test.ts — Random mutations checked against both parsers simultaneously
• error-recovery.test.ts — CST coverage metrics for error recovery scenarios
• robustness.test.ts — 100+ edge cases verifying error recovery (unclosed delimiters, malformed syntax, garbage input)
• single-quote.test.ts — Verifies single quotes produce errors (parity with tree-sitter)

Parity invariant

When an input parses without errors in either parser, it must parse without errors in both parsers, and the resulting parse trees must be identical (normalized s-expressions).

Parse trees are allowed to deviate when the input is not valid AgentScript — both parsers will have errors, but their error recovery strategies differ (recursive descent vs GLR), so the resulting CSTs may differ. These deviations are tracked via snapshots.

Performance Benchmarks

The package includes a comprehensive benchmark suite that stress-tests the parser across multiple dimensions.

See PERFORMANCE.md for the latest benchmark results. Run pnpm perf:report to regenerate after changes.

Running Benchmarks

# Detailed timing output (recommended)
pnpm perf

# Generate PERFORMANCE.md report (commit alongside code changes)
pnpm perf:report

# Vitest bench format (for CI)
pnpm bench

Benchmark Dimensions

| Dimension | What it tests | |---|---| | File size scaling | Linear scaling: 100 to 100K lines of flat mappings | | Deep nesting | Indent stack and recursion depth: 50 to 1,000 levels | | Wide mappings | Sibling key accumulation: 1K to 50K keys | | Chained expressions | Pratt parser with long a + b + c... chains | | Nested parentheses | Recursive descent depth with (((...))) | | Mixed precedence | Precedence climbing with interleaved + * - / | | Large strings | Lexer string scanning: 1KB to 1MB strings | | Escape-heavy strings | Character-by-character escape processing | | Template interpolations | {! expr } handling at scale | | Error recovery | Alternating errors, garbage input, unclosed delimiters | | Large sequences | - item syntax at 1K to 50K items | | Procedure-heavy | if/run/set statement parsing | | Highlighting overhead | parse() vs parseAndHighlight() comparison | | Realistic workloads | Mixed agent files from 50 to 50K lines | | Lexer isolation | Lex-only runs to separate lexer vs parser cost |

Adding Generators

Input generators live in test/perf-generators.ts. Each function returns a string of synthetic AgentScript. To add a new stress dimension:

1. Add a generator function to perf-generators.ts
2. Add benchmark calls in both perf.bench.ts (vitest) and run-perf.ts (direct runner)

Architecture

src/
  index.ts          — Public API (parse, parseAndHighlight)
  lexer.ts          — Indentation-aware tokenizer (INDENT/DEDENT/NEWLINE)
  parser.ts         — Recursive descent parser + statement parsing
  expressions.ts    — Pratt expression parser with precedence climbing
  cst-node.ts       — CST node class with SyntaxNode interface
  highlighter.ts    — CST walk for syntax highlighting captures
  errors.ts         — Error recovery utilities (synchronize, makeErrorNode)
  token.ts          — Token and TokenKind definitions