@codincod/codemirror-lang-python

This package implements Python language support for the CodeMirror code editor, using the Lezer grammar Marijn Haverbeke wrote for @lezer/python, with the parse fixes below on top of it. The editor half comes from @codemirror/lang-python; the two are merged here so that one package holds the grammar and the language support that has to move with it.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {python} from "@codincod/codemirror-lang-python"

const view = new EditorView({
  parent: document.body,
  doc: `from contextlib import contextmanager

@contextmanager
def timed(label):
    yield

with (open("in") as src, open("out", "w") as dst):
    dst.write(src.read())
`,
  extensions: [basicSetup, python()],
})

API Reference

• python() → LanguageSupport
• pythonLanguage: LRLanguage
• parser: LRParser
• globalCompletion, localCompletionSource: the completion sources from @codemirror/lang-python, unchanged

What changed

Seven parse fixes over @lezer/python 1.1.19, each with grammar tests. On the Python 3.14 standard library and the packages installed beside it, 3870 files and 47 MiB, they take the share of files parsing with no error node from 95.58% to 99.87%, and the error nodes from 3936 to 61.

A bare yield. yield { kw<"yield"> (…)? }. The value was required, so every @contextmanager in the corpus failed on the one line that makes it a context manager. This was the single biggest cause: 45 of the 171 failing files.

Floats ending in a point. 1., 100./3, f(t=2.). Python's lexer reads digitpart "." as a float; the grammar wanted a digit after the point, so 1. went down the member-expression path instead. Half the time that failed outright and half the time it quietly succeeded, reading 2.j as the attribute j of 2, which is worse.

Parenthesised context managers. with (open(a) as f, open(b) as g):, Python 3.10. A new WithItemGroup node, reached through an ambiguity marker because nothing at the ( says which of the two readings is coming. Parentheses with no as in them still parse as the expression they used to be, so with (a): and with (a, b): keep their old trees.

Targets after as. with f() as (write, encoding): and with lock() as self.handle:. What follows as is a target, not a name. ForStatement already spelled its targets expression; this now matches it.

Decorators. PEP 614 dropped the dotted-name restriction in Python 3.9, so @buttons[0].clicked.connect is ordinary now. Decorator takes any expression. This changes the tree under a decorator: @a.b was Decorator(At, VariableName, VariableName) and is now Decorator(At, MemberExpression(VariableName, PropertyName)), which is how the same text is read everywhere else in a file.

Empty pattern lists. case Section():, case []:, case {}:. All three match-statement bracket rules required at least one element.

Starred comprehension targets. [f(n) for n, *rest in pairs]. ForStatement allowed the star and compFor did not.

Two smaller things ride along. Template strings (PEP 750, Python 3.14) are read by adding t to the format-string prefixes rather than by giving them sixteen string tokens of their own: a t"..." is written and read exactly as an f"..." is, and both come back as FormatString. And WithItemGroup joins the indentation, folding and bracket-tracking tables, so a wrapped with indents and folds like the tuple it resembles.

What is still not read

Five files of the 3870 leave error nodes, and @lezer/python leaves them in exactly the same places. They are long files of blank-line-separated top-level statements, and the errors are zero-width: @lezer/lr cuts the parse stack at Rec.CutDepth and force-reduces, which costs two error nodes and no colour. It is a parser limit rather than anything the grammar says.

Testing

npm test runs the grammar specs in test/*.txt and the editor tests beside them. The specs are @lezer/python's own, plus fixes.txt for the seven changes; the editor tests are @codemirror/lang-python's indentation suite converted to node:test, plus folding and highlighting.

npm run corpus -- <dir> parses a directory of real Python and reports the share of files with no error node, grouped by directory. No corpus ships with the package; point it at your own.