@codincod/codemirror-lang-dart

[ CHANGELOG ]

This package implements Dart language support for the CodeMirror code editor, using a Lezer grammar written for this package.

CodeMirror 6 has never shipped Dart, and no Lezer grammar for it existed before this one. What gets used instead is the C-family stream mode ported from CodeMirror 5, which colours a list of keywords and builds no tree, so nothing above it can fold a class, indent a switch or ask what the word under the cursor is. It also predates most of the language: string interpolation, cascades, null safety, records and patterns all come out as plain text or as something else entirely.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {dart} from "@codincod/codemirror-lang-dart"

const view = new EditorView({
  parent: document.body,
  doc: `class Greeter {
  const Greeter(this.name);

  final String name;

  String greet() => 'Hello, $name!';
}
`,
  extensions: [basicSetup, dart()]
})

Coverage

Dart 3. Classes, mixins, enums with constructors and members, extensions and extension types, sealed and base hierarchies, typedefs both ways, generics with bounds, sound null safety, records and their types, patterns everywhere Dart 3 puts them, switch expressions and switch statements, if-case, cascades and null-aware cascades, collection if, for and spreads, async and both kinds of generator, initializing and super formals, metadata, and every one of the eight ways Dart writes a string.

The parts Dart added most recently are here too: digit separators, null-aware elements, dot shorthands, primary constructors and the new that names a constructor rather than calling one.

Measured against 4 465 Dart files from six open source projects (the Flutter framework and its tools, the Dart SDK's own libraries and front end, Riverpod, bloc, Dio and dart-lang/core), which is 58 MiB of code, of which 98.66% parse with no error node.

What the grammar cannot say

Three things about Dart are settled in src/tokens.ts rather than in the grammar, because no context free rule can decide them.

A ? makes a type nullable or opens a conditional. e as String? and e is String ? a : b disagree about one character, and the disagreement cannot be settled where it stands. Lezer settles a shift against a reduce by precedence rather than by trying both, because the two readings do not cover the same span and so are not the kind of ambiguity a conflict marker can hold open; whichever way the precedence goes, half the language breaks. The tokenizer answers it instead, by looking further along the line than the parser does: a colon outside every bracket means a conditional, and a semicolon, a comma, an assignment or a bracket nobody opened means a type. An angle bracket behind the cursor that nothing closed is what tells the ? of c ? JS<Ref?>(a) : d from the one that opens the conditional around it.

A string is eight different languages. The quote may be single or double, the delimiter one character or three, and an r in front turns every escape back into an ordinary character. Which of the eight is being read decides what ends the string and whether $ opens a hole holding an expression, and none of it can be read off the character at the cursor. It is not remembered in the tokenizer either, because the parser may fork and anything remembered would be remembered for both halves; it is asked of the parser instead, since each flavour's content token can only be shifted inside its own kind of string. That is also what makes 'a ${b("c")} d' work: inside the hole no content token can be shifted at all, so the quote there opens a string of its own.

Block comments nest, and a raw string opens with a letter. /* /* */ */ is one comment in Dart and two in most other languages, which is a thing a regular token cannot say: it has to count. And r'a' begins with something every other tokenizer is happy to call an identifier, so it has to be claimed before the grammar's own tokenizer is asked.

Two decisions about the tree are worth saying out loud.

Casing decides what is a type. Foo bar is a declaration and foo.bar is a member access, and after one identifier no amount of lookahead separates them, so the parser follows the convention every Dart codebase keeps: a word starting with a capital is read as a type first and an ordinary name second, and a word starting with a lower case letter the other way round. Both readings stay alive and the one that fits wins. Dart writes a private name with a leading underscore and generated code writes one with a dollar, so the letter that decides is the first one that is a letter: _MyWidgetState and _$FooImpl are types.

A cascade is a member access. a..b()..c = 1 binds looser than anything else in the real language, but a highlighter has no use for the difference, and reading .. at the precedence of . costs no level, no conflict and no lookahead. The tree says (a..b)() where Dart says a..(b()); every token in it is still coloured by what it is.

Development

npm test                    # build, then the parse specs and the editor tests
npm run corpus <directory>  # error nodes over a directory of real Dart

test/corpus.ts is what decides whether the grammar is finished. No corpus ships with the package, because none of that code is ours to redistribute; point it at a checkout of anything large and it will group the report by the project each file came from, so a package that drags the total down names itself.