@codincod/codemirror-lang-elixir

[ CHANGELOG ]

This package implements Elixir language support for the CodeMirror code editor, using a Lezer grammar forked from lezer-elixir.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an Apache 2.0 license, the licence it was forked under. NOTICE says what was changed.

Usage

import {EditorView, basicSetup} from "codemirror"
import {elixir} from "@codincod/codemirror-lang-elixir"

const view = new EditorView({
  parent: document.body,
  doc: `defmodule Greeter do
  @moduledoc "Says hello."

  def greet(name) when is_binary(name) do
    IO.puts("Hello, #{name}!")
  end
end
`,
  extensions: [basicSetup, elixir()]
})

Coverage

Elixir 1.x. Sigils with any delimiter and their modifiers, heredocs, charlists, interpolation, bitstrings, structs and the update syntax, keyword lists, captures, stab clauses with their guards, do blocks with after, catch, else and rescue, and the operators, which in Elixir are a language of their own.

Measured against the Elixir standard library, Phoenix, Ecto, Livebook, Absinthe, Nx, Oban and Plug, 97.76% of 2233 files parse with no error node, against upstream's 96.06%. Over CodinCod's own pool of Elixir snippets, 96.70% against 96.37%.

What was fixed

A sigil's modifiers have to touch it. ~r/ab/i ends in a modifier and ~w(a b) do does not, and the only thing between them is the space. The tokenizer is asked for a token at the first thing after the whitespace rather than at the whitespace, so without the check it read do as modifiers, and the block that do opened was never seen. Every for x <- ~w(a b) do in a file failed, and this alone is most of the difference in the numbers above.

A tokenizer that asks the stack a question has to say so. Three of them ask stack.canShift to decide what to return: which delimiter a quoted body is closing, whether a bracket is a call's or a literal's, and whether a newline ends the expression. Lezer caches a token per position unless the tokenizer is declared contextual, so when the parse forked, one stack's answer was handed to another. What that broke in practice is a case inside a case clause with another expression after it: the newline after the inner end was never made a terminator, and the expression under it had nothing to attach to.

@ can follow a dot. Kernel.@(behaviour(Exception)) is how a module attribute is written when it is written out in full, and it appears in the standard library and in anything doing macro work.

What the tree does not say

Lezer keeps at most twelve parses alive at once and drops the rest. Elixir's grammar is ambiguous about whether a name begins a call with no parentheses around its arguments, so a long pipe chain or a bitstring of many elements can fork past that limit and lose the parse that was right. About twenty files in the corpus above fail for this and nothing else; raising the limit fixes each of them, and the real fix is a grammar with less to be ambiguous about.

Development

npm install
npm test                                    # the parse specs and the editor tests
npm run corpus -- path/to/elixir/checkouts  # clean share over real code
npm run gaps -- path/to/elixir/checkouts    # what it could not parse, grouped

corpus and gaps need lezer-survey checked out beside this repository. No corpus ships here, because none of that code is ours to redistribute.