@codincod/codemirror-lang-lisp

[ CHANGELOG ]

This package implements Lisp language support for the CodeMirror code editor, using a Lezer grammar written for this package.

One parser serves the family: Common Lisp, Scheme, Racket, Emacs Lisp, and the smaller dialects that share their reader. They differ in what the reader does before the lists are built, and agree about everything after.

CodeMirror ships Common Lisp and Scheme as CodeMirror 5 stream modes. A stream mode colours tokens and stops there. Without a tree there is no structural folding, no indentation that knows what a form is, and no node-aware selection.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {lisp} from "@codincod/codemirror-lang-lisp"

const view = new EditorView({
  parent: document.body,
  doc: `(defun greet (name)\n  (format t "hello, ~a~%" name))`,
  extensions: [basicSetup, lisp()]
})

Coverage

Which reader constructs the grammar carries was decided by counting them across 5618 snippets, so a few things that look like omissions are deliberate.

Brackets and braces are lists. Racket and R6RS write [ where the standard writes (, and no dialect gives it a different meaning; brackets appear in 22% of the corpus. #lang takes its whole line, because that line names the reader instead of being a form the reader produces, and it appears in 17.5%.

Also handled: the quote family (', `, ,, ,@, #', #`, #,, #,@), both keyword spellings (:name and #:name), characters, booleans, vectors and structs (#(, #s(, #3a(), dotted pairs, |symbols with spaces|, ratios and the radix prefixes #x #b #o #d #e #i, #; datum comments, #+/#- reader conditionals, and comments in both forms with #| |# nesting properly.

Strings may span lines. That sounds like a detail and is not one. Every Common Lisp definition may carry a docstring, most of them wrap, and a token that stopped at the line end closed the string early, read the prose as symbols, and then took the real closing quote for the start of the next string. Every form after it in the file shifted by one.

Special forms are lifted out of Symbol into their own node, so structure can be told from application at a glance. The list is a union across dialects. A name that is ordinary in the dialect on screen costs one miscoloured symbol; leaving it out costs every file that uses it.

The head of a list is a Callee, which is the thing a Lisp does not say with punctuation: (display x) and (x display) are the same shape and different programs. 46% of the symbols in the corpus are the head of something, and 69% of them in a Racket file, so a theme that paints a call differently from the names it is passed has most of the file to say it with.

Binding forms are known by name, because a parameter list is a list of names and not a call. (defun f (a b) ...) gives f a Definition node and leaves a and b as ordinary symbols, where a grammar that only knew about heads would read a as a function. Nineteen forms are recognised (lambda, let and its relatives, do, dolist, destructuring-bind and so on), which is the set whose shape is always "an optional name, then a list of names".

Accuracy

Measured against 5590 Rosetta Code snippets. A file counts as clean only when the parse leaves no error node anywhere.

| | Snippets | Clean | | --- | --: | --: | | All | 5590 | 98.35% | | Racket | 1191 | 99.58% | | Common Lisp | 1161 | 98.36% | | PicoLisp | 953 | 98.11% | | Scheme | 579 | 98.62% | | EchoLisp | 310 | 96.45% | | Emacs Lisp | 260 | 98.46% |

92 files are not clean. Most of those have unbalanced brackets: they are fragments, transcripts of a REPL session, or shell command lines filed as source. No parser reads them.

npm run corpus -- path/to/files runs that measurement over your own directory. No corpus ships here. Rosetta Code is CC BY-SA and cannot be redistributed under this licence.

Known limits

PicoLisp reads ] as a super-parenthesis closing every open bracket at once. That cannot be reconciled with Racket's [ matching ] without knowing which dialect is on screen. Racket is the larger share, so brackets match.

Special forms are recognised by name and never by scope, so a variable called for or match is coloured as a keyword.

Three binding forms keep a shape the grammar does not follow: defstruct, defgeneric and defmethod name their arguments after a qualifier or without a list around them, so the first name in them still reads as a call. That is 0.05% of the heads in the corpus.

API Reference

lisp() → LanguageSupport

Lisp language support, with completion for the special forms.

lispLanguage: LRLanguage

A language provider based on the Lezer Lisp parser in this package, extended with highlighting and indentation information.

lispCompletion: CompletionSource

Autocompletion for the special forms the grammar recognises.

parser: LRParser

The raw Lezer parser, for tooling that wants the tree without an editor.