lezer-macaulay2

A Lezer parser for the Macaulay2 language. Lezer is the incremental, error-tolerant parser system used by CodeMirror 6, so this package can drive Macaulay2 syntax highlighting and structural editing in a browser.

Installation

npm install lezer-macaulay2

Usage

import { parser } from "lezer-macaulay2"

const tree = parser.parse("for i from 1 to 10 do print i")
console.log(tree.toString())

The exported parser is configured with highlighting tags. To use it in a CodeMirror 6 language:

import { LRLanguage, LanguageSupport } from "@codemirror/language"
import { parser } from "lezer-macaulay2"

export const macaulay2 = new LanguageSupport(
  LRLanguage.define({ parser })
)

macaulay2Highlight (the styleTags props) is also exported if you want to configure the parser yourself.

What it parses

• Literals: integers (decimal/binary/octal/hex), floats (including p precision and e/E exponents), strings, ///-delimited raw strings, identifiers (with ' and $).
• Comments: -- line, -* … *- block, #! shebang.
• The full Macaulay2 operator table — binary, prefix, and postfix — with faithful per-group precedence and associativity. Operators appear in the tree so themes can style them, under nodes named after Macaulay2's own operator groups: ArithmeticOp, ComparisonOp, EqualityOp, AssignmentOp, AugmentedAssignmentOp, AccessOp, FunctionOp, FunctorOp, and a generic Operator for the groups M2 documents as miscellaneous. Word operators (and, or, xor, not — M2 calls these predicates) stay keyword nodes tagged t.operatorKeyword, as in @lezer/python.
• Function application by adjacency (f x, f(x), f[x], f<|x|>), with #/. member access binding tighter than application.
• Control constructs: if/then/else, while, for (from/to/in/ when/list/do), try/then/else/except, new/of/from, and the control/debug keywords (return, break, throw, …).
• Comma sequences, including omitted elements (map(R,,f), {a, b,}), and ;/newline statement separation. Bracketed forms use M2's class names: Sequence, Array, List, AngleBarList. Parentheses stay ParenExpr, since (1) is just ZZ in M2 rather than a one-element Sequence.
• symbol/global/local/threadLocal/threadVariable quoting an identifier, an operator (symbol ==) or a keyword (symbol and).

Control constructs bind loosest and their clause bodies extend as far right as possible, matching Macaulay2:

1 + if x then 2 else 3      parses as   1 + (if x then 2 else 3)
if x then y else z + w      parses as   if x then y else (z + w)
(if x then y else z) + w    keeps the parenthesized if as a left operand

A newline only ends a statement when the next line can begin one, so an expression continued with else, and, |, => and the like keeps going:

if p then (a)      is one expression, not two statements
else (b)

Caveat: clause combinations are not validated

Every control construct accepts the same repeated list of clauses, so the parser will happily accept combinations Macaulay2 rejects:

if a do b        parses    (M2: syntax error)
new T then U     parses    (M2: syntax error)
for i            parses    (M2: syntax error)

This is deliberate. A Lezer grammar drives syntax highlighting and structural editing, not validation — M2 itself rejects these — and spelling out each construct's own chain of optional clauses cost about nine times the build memory (see the design notes below). It also costs nothing while typing: the strict and permissive grammars flag exactly the same partial inputs, so an editor behaves identically either way.

If you need the distinction (say, to drive @codemirror/lint from error nodes), giving each construct its own clause set restores it for roughly 3x the build memory.

Development

The grammar is generated: src/macaulay2.grammar is produced from src/macaulay2.grammar.template + src/operator-info.json by scripts/gen-grammar.js (run automatically as the prebuild step).

npm run generate   # regenerate src/macaulay2.grammar
npm run build      # generate, then bundle to dist/ with @lezer/generator
npm test           # run the test corpus in test/cases/*.txt
npm run check:mem  # assert the grammar still builds under a heap cap
npm run check:corpus   # parse every package in a Macaulay2 checkout

check:corpus is the check that catches constructs real Macaulay2 uses and the grammar does not; test/cases only covers snippets written with this grammar in mind. Every package must parse with zero error nodes.

It finds a corpus by trying, in order: a path argument, $M2_PACKAGES, an M2 source checkout beside this repo (../M2/M2/Macaulay2/packages), and finally the packages of an installed M2, which it locates by asking the binary for its prefixDirectory. If none of those exist it prints a note and exits 0, so it is safe to run without an M2. CI runs it against both Macaulay2 development and the current stable release from ppa:macaulay2/macaulay2.

Notes on the grammar design

• operator-info.json is the single source of truth for the operator table.

• gen-grammar.js also emits src/operator-data.js, the operator facts the external tokenizer needs at runtime, so the scanner cannot drift from the grammar.

• Control constructs are alternatives of the one shared expr nonterminal, sit at the loosest pctl precedence level, and use precedence-named clause nodes (ThenClause, FromClause, …). Adjacency/application and ;/newline separation are handled by a stack-aware external tokenizer in src/tokens.js (appSpace/appIndex/sep), ported from tree-sitter-macaulay2's SPACE scanner.

• The LR table is memory-sensitive to build. @precedence must be emitted from highest to lowest binding power (the order Lezer expects), and operators must share the single expr nonterminal; deviating from either makes the generator OOM.

• Every control construct shares one repeated clause list rather than spelling out its own chain of optional !trailer clauses. That is what keeps the build cheap. Because a clause body is the shared expr, the generator has to decide "extend this expression, or stop and take trailing clause Y?" at every position inside a clause body and at every one of the ~24 operator precedence levels — and when the set of possible Y differed per construct, those contexts multiplied with the levels. Measured peaks:

  | clause formulation | build | peak RSS | | --- | --- | --- | | one shared clause list (current) | 2.4 s | ~365 MB | | one clause set per construct | 12.5 s | ~1269 MB | | per-construct optional chains | 50 s | ~3463 MB |

  For scale, @lezer/python peaks at ~406 MB and @lezer/javascript at ~561 MB measured the same way. npm run check:mem caps the heap at 768 MB, so reintroducing the multiplication fails loudly.

License

MIT