@codincod/codemirror-lang-hy

[ CHANGELOG ]

This package implements Hy language support for the CodeMirror code editor, using a Lezer grammar written for this package.

Hy had no grammar for CodeMirror anywhere: nothing on npm under any spelling of the name, and the one tree-sitter grammar that exists says of itself that it is alpha and very incomplete. A .hy file was plain text, or it was handed to a Lisp grammar, which is worse in a way that is easy to miss.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {hy} from "@codincod/codemirror-lang-hy"

const view = new EditorView({
  parent: document.body,
  doc: `(defn greet [name]\n  (print f"hello, {name}"))`,
  extensions: [basicSetup, hy()]
})

Why not a Lisp grammar

Hy parses clean under one. Everything made of parentheses does, which is why the number is worth nothing here: over 14942 files of published Hy the Lisp grammar in this family reads 97.85% of them without a single error node, and colours barely half of what it read.

| Hy writes | A Lisp grammar sees | It is | | --- | --- | --- | | f"hello, {name}" | the symbol f, then a string | an f-string, and name inside it is code | | #(1 2) | a vector | a tuple | | #{1 2} | a vector | a set | | #* args | a symbol #* | Python's *args | | #^ int x | plain symbols | a type annotation on x | | #[delim[ ... ]delim] | a reader error | a raw string carrying its own delimiter | | 1_000, 1,000 | symbols | one million, twice | | .append, self.name | plain symbols | attribute access, the whole of Python interop | | True, False, None | plain symbols | Python's constants | | defn, setv, lfor | plain symbols | the forms the language is built from |

Measured over the same files, this grammar puts 13 colours on the page where the Lisp grammar puts 10, and leaves 31.12% of the coloured text as an uncoloured name where the Lisp grammar leaves 41.44%. The definitions are the sharpest of it: 3.69% of the text against 0.14%, because a Lisp grammar knows defn names something only by accident. Attribute access, Python's constants and everything inside an f-string it does not see at all.

It goes the other way too. Hy has no #| block comment, no #\ character literal, no dotted pair, no vector reader, and one spelling of quote where Common Lisp has a family of them.

What the reader decides

Every rule in the grammar comes from hy/reader/hy_reader.py in the compiler rather than from the manual, because the reader is where the surprises are.

An identifier ends at whitespace or at one of ()[]{};"'`` ``~, and at nothing else. So a,b, x:y, @deco, foo#barand1abc` are each one symbol: characters other languages punctuate with are ordinary letters here, and a name may begin with a digit.

That rule is what makes #* a prefix that needs a space after it. A # reads the whole identifier that follows and looks it up as a reader macro, so #* args unpacks and #*args is the undefined reader macro *args. The same goes for #**, #^ and #_. This grammar reads #*args as unpacking anyway, since that is what it meant in every version of Hy before 1.0 and what it still means to whoever wrote it.

A number is a name the reader changed its mind about. It reads the identifier first and then asks int, float and complex, in that order, whether it was a number; a token all of them refuse stays a name. 1_000 and 1,000 are both one million, Inf and NaN are floats while inf and nan are names, and 1abc is a symbol.

An f-string component holds exactly one form, and the reader takes that form before it looks for !, = or :. Since each is an identifier character, f"{x:>5}" is the single symbol x:>5 with no format spec at all, and the space in f"{x :>5}" is the whole difference. A component may also hold a string of its own: f"{(get d "k")}" is one f-string, and the quote in the middle of it closes nothing.

Measured on

17787 .hy files, every version of every one of them, out of all 532 repositories GitHub files under Hy. Hy's own reader was run over the same files as an oracle and turns 2845 of them down, and of the 14942 that are valid Hy this grammar reads 100.00% with no error node. The raw share, junk included, is 96.71%.

It is deliberately looser than the reader in the other direction, and reads 2260 of the files Hy rejects: #*args and #^int from before Hy 1.0, reader macros the module imported rather than defined, and half-typed forms, which is what an editor is looking at most of the time.

npm run measure runs the grammar over a directory of real Hy, and npm run gaps groups whatever it could not parse by shape. npm run harvest builds that directory out of every public repository GitHub files under Hy, taking every version of every .hy file and asking Hy's own reader which of them are syntactically Hy: tools/read.py is that question, and a file it turns down is not a file the grammar failed. It needs hy importable by the python on the path, and it reads without evaluating, so none of the code it is given is ever run.