@codincod/codemirror-lang-regex

[ CHANGELOG ]

This package implements regular expression support for the CodeMirror code editor, using a Lezer grammar written for this package. It reads a pattern as one named engine reads it, and it knows nineteen of them.

CodeMirror has never shipped a mode for the inside of a regular expression. In CodeMirror 5 and 6 alike, a regex literal in a JavaScript file is one token painted one colour, and a regex typed into a search box is plain text. That is fine until the pattern is the thing you are working on, which is the case on every regex tester ever built.

Written in part for CodinCod, a competitive coding platform, where it reads the patterns typed into its regex tester.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {regex} from "@codincod/codemirror-lang-regex"

const view = new EditorView({
  parent: document.body,
  doc: String.raw`(?<area>\d{3})-(\d{4})`,
  extensions: [basicSetup, regex()]
})

regex() on its own reads the pattern as a browser would, because a pattern typed into a browser is usually about to be handed to that browser's own RegExp. Name another engine when you know one:

regex({flavour: "pcre"})
regex({flavour: "python"})
regex({flavour: "php"})              // an alias for pcre
regex({flavour: "pcre", flags: "x"}) // free spacing, so `#` starts a comment
regex({flavour: "javascript", flags: "v"})
regex({flavour: "icu"})              // an alias for swift, objc and mysql
regex({flavour: "are"})              // an alias for tcl and postgres
regex({flavour: "cpp"})              // std::regex, which is not quite ECMAScript
regex({flavour: "xsd"})              // the patterns inside an XML schema
regex({flavour: "raku"})             // a second grammar, in the same package

The flags are the engine's own letters, so the same string you would pass the engine can be passed here. Four of them change what a pattern is rather than what it matches, and the rest are read past: x ignores whitespace and takes # as a comment, PCRE2's xx does that inside character classes as well, and JavaScript's u and v each read the pattern more strictly than the flagless dialect they replace.

A regular expression is not one language

(?P<year>\d+) is a named group in Python, a named group in PCRE and a syntax error in a browser. \d is a digit in thirteen of the flavours here and a literal d in grep. \v is vertical whitespace in PCRE and a vertical tab in JavaScript, so [\v] matches five characters on a server and one in a browser. [a-z&&[^aeiou]] is an intersection in Java and, in PCRE, a class holding two ampersands and a bracket. (?<=a)* compiles in Python and PCRE and is an error in every browser. \b is a word boundary in fourteen of the flavours here, a backspace in Tcl and PostgreSQL, where the boundary is \y and \B is a backslash, and nothing at all in an XML schema, which has no assertions and no anchors either: ^a$ there matches four characters.

So an engine has three ways to answer, and this grammar gives all three.

| | | | --- | --- | | it has this | the construct gets the node it deserves: \d is a CharacterClassEscape | | it reads this differently | no error, and no node: \d in grep is an Escape, a literal letter | | it refuses this | an error node, where a pattern will not compile at all |

The middle column is the one worth having. A loose engine almost never rejects anything, so a parser that only answered yes or no would call \p{L} fine in a flagless JavaScript pattern, where it matches a p and a brace and no letters whatsoever.

The nineteen

Sixteen of them are one grammar. Raku, Lua and Vim are the other three and are not, so each has a grammar and a parser of its own in the same package: [abc] is a character class in every other engine here and a non-capturing group in Raku, a Lua pattern has %a where everyone else has \w and no alternation at all, and Vim puts its assertions after the group rather than inside its opener. regex({flavour: "raku"}) and its two neighbours reach them, and everything an editor does with the result is the same.

| Flavour | Aliases | Notes | | --- | --- | --- | | javascript | js, ecmascript, node | three languages, chosen by the u and v flags | | pcre | pcre2, php, elixir, erlang, r | the widest of them: conditionals, recursion, callouts, verbs | | perl | perl5 | PCRE plus code blocks, minus callouts | | python | re, py | (?P<n>) and (?P=n), no properties, \z since 3.12 | | dotnet | csharp, c# | balancing groups, class subtraction with a dash | | java | kotlin, scala, groovy | class intersection and nesting, \Q...\E, possessive quantifiers | | icu | swift, objc, mysql, icu4c, icu4j | Java's neighbour: (?#, \U, POSIX classes, and no \pL | | cpp | c++, cxx, stdregex | says it reads ECMAScript, and refuses a third of what V8 takes | | ruby | onigmo, oniguruma | the absent operator, \O, subroutine calls, \h as a hex digit | | rust | | class set operations, no backreferences, no lookaround | | go | re2, golang | as Rust for what it refuses, and POSIX classes for what it takes | | ere | posix, egrep, awk | POSIX extended, with the parts GNU added | | bre | grep, sed | POSIX basic, where the groups are \(escaped\) | | are | tcl, postgres, postgresql | Spencer's advanced: \y is the word boundary, \b a backspace, ***= a literal | | emacs | elisp, emacs-lisp | escaped groups, bare quantifiers, syntax classes, symbol boundaries | | xsd | xml, xmlschema | a schema's own: \i and \c, no anchors, no groups but plain ones | | raku | perl6, rakudo | a grammar of its own: <[a..z]>, **, %, insignificant whitespace | | lua | luajit, luau | a grammar of its own: %a, %b(), %f[], no alternation | | vim | nvim, neovim, vi | a grammar of its own, in four levels of magic: \(a\|b\)\+, \(a\)\@= |

The differences between the sixteen are 98 Lezer dialects, and a flavour is a set of them: xsd switches on 11, bre on 17, are on 28, pcre on 53. The whole matrix is one line per feature in src/dialects.ts, written as "who spells this", because the interesting question is never what PCRE has.

Measured

Eight bodies of patterns, none of them written here, and two flavours measured on somebody else's. Two are read by test/corpus.ts: the first is the fixture set of regexpp, the regex parser ESLint uses, which records what V8 accepts and the reason V8 gives when it does not, and the second is CPython's Lib/test/re_tests.py, the re module's correctness suite since Python 1.5.

The other three grammars, and the three dialect sets whose engines are programs in their own right, are measured the same way: against the largest body of patterns their own engine will vouch for.

Raku's is roast, whose S05-* directories are the regex chapter: 2948 patterns come out of them, and whether each one is legal is a question only Rakudo can answer, so test/verdicts.raku asks it and test/roast.ts keeps the score.

Lua's is the test suite from lua.org, where pm.lua is the pattern matching file and the other thirty-two files use patterns as any program does; 256 come out, and test/verdicts.lua puts each one to Lua, which has no way to compile a pattern without matching one and so is asked with nine subjects.

Vim's corpus is Vim. 20,680 patterns come out of the syntax, indent, ftplugin and autoload files it ships, which is more than the rest of this measurement put together, and they are patterns people rely on daily rather than test cases. test/verdicts.vim asks Vim itself about each one, and test/vim.ts reads the magic level off the front of the pattern so that each is parsed as the file it lives in would have it read.

Emacs's corpus is Emacs, the same way Vim's is Vim. There is no Emacs installed on the machine this was written on, but there is one in the image our own code runner runs Emacs Lisp from, which is where both the engine and the lisp come from: 6821 patterns are read out of the 1655 files it ships, by walking each file the way the reader does and keeping the strings that a regexp function was handed or that a defvar named a regexp. test/verdicts.el asks Emacs about each one and test/emacs.ts keeps the score.

Emacs's own lisp holds nothing Emacs refuses, which would leave half of that measurement with nothing to measure, so the patterns are cut short as well. A pattern with its tail taken off is either still legal, and must still parse, or it is not, and should be caught. It is also what an editor looks at most of the time, since a pattern being typed is a pattern that has not finished yet.

ICU's corpus is ICU's own: regextst.txt, the file its authors run before a release, where a case is a pattern, a column of flags and a subject with the capture groups tagged in it. 2107 patterns come out of it, and the verdicts come from ICU4C itself: test/verdicts.cpp is thirty lines around uregex_open and test/icu.ts keeps the score. Two kinds of case are left out rather than counted, because neither is in this language: Q puts the whole pattern in quotes, and e turns on error-on-unknown-escapes, under which \q stops being a q.

ARE's corpus is both halves of the lineage. Tcl ships Henry Spencer's own suite for the engine he wrote, tests/reg.test, which says which flavour each case is in and leaves 290 that are advanced regular expressions rather than POSIX; and Postgres ships src/test/regress/sql/regex.sql, which is the same engine two decades on. Whether a pattern is legal is that engine's business, and Postgres is the half of it that answers questions, so test/verdicts.sql asks it and test/are.ts keeps the score.

The last two flavours have no suite of their own anywhere, so each is put to the patterns of the language it is nearest to, and its own engine says which of them it will take. std::regex is measured on ECMAScript's fixtures, because the C++ standard says std::regex reads ECMAScript; the answer is that a third of them will not compile, which is the argument for its being a column rather than an alias. XML Schema is measured on both the ECMAScript and the Python patterns, since a schema's regexes live inside <xs:pattern> and nobody keeps a file of them; 441 of the 1177 survive, and what survives is a fair sample of the subset a schema author is left with.

Both are asked the same way the others are. test/verdicts.cpp answers for std::regex as well as for ICU, since both are asked in C++. test/xsd.ts needs no asker at all: the question is already a document, so it writes a schema holding one type per pattern and xmllint names the lines it will not compile.

| | | | --- | --- | | patterns V8 accepts, parsed with no error | 499 / 499 | | patterns CPython compiles, parsed with no error | 363 / 363 | | patterns Rakudo compiles, parsed with no error | 2329 / 2329 | | patterns Lua accepts, parsed with no error | 243 / 243 | | patterns Vim compiles, parsed with no error | 20535 / 20535 | | patterns Emacs accepts, parsed with no error | 6821 / 6821 | | the same patterns cut short, where Emacs still accepts them | 3685 / 3685 | | patterns ICU accepts, parsed with no error | 2076 / 2076 | | patterns Postgres compiles, parsed with no error | 294 / 294 | | patterns std::regex compiles, parsed with no error | 387 / 387 | | patterns a schema compiles, parsed with no error | 441 / 441 | | patterns V8 rejects, caught as errors | 149 / 293 | | patterns CPython rejects, caught as errors | 32 / 40 | | patterns Rakudo refuses, caught as errors | 373 / 419 | | patterns Lua refuses, caught as errors | 13 / 13 | | patterns Vim refuses, caught as errors | 83 / 83 | | the same patterns cut short, where Emacs refuses them | 4817 / 4846 | | patterns ICU refuses, caught as errors | 5 / 31 | | patterns Postgres refuses, caught as errors | 40 / 71 | | patterns std::regex refuses, caught as errors | 242 / 405 | | patterns a schema refuses, caught as errors | 663 / 736 |

The first eleven are the pass mark, and 100% is the only acceptable number: an editor that paints working code as broken is worse than one that paints nothing at all.

The rest are watched rather than aimed at, because a grammar knows shapes and not meanings. Of the eight Python rejections that get through, every one is about meaning: \1 with no groups to refer to, a[b-a] with its range the wrong way round, ((.)\1+) referring to a group that is still open. Those are a linter's business, and this package is not a linter.

And for contrast, since it is the whole argument for having flavours at all: 139 of CPython's 363 working patterns are not JavaScript, 182 of the 499 patterns V8 compiles are not std::regex, and 736 of the 1177 patterns written for the other engines are not a schema's.

ICU's refusals are the clearest case for that being a linter's line rather than a parser's: of the 26 this lets through, every one is a name or a number rather than a shape. [\p{InBadBlock}] is a block nobody defined, x{4294967300} is a count that will not fit, (?<a>.)(?<a>.) names two groups the same, and (?<!(0123456789a){10000000})x is a lookbehind ICU declines to think about.

The 73 schema rejections that get through are three shapes, and two of them are names rather than shapes: \p{Emoji} and \p{Script=Hiragana} are properties a schema does not define, and [b-a] is the usual range the wrong way round. The third is [A--B], which is a difference in a browser's v mode and, in a schema, a range ending at a hyphen, which is a thing a schema will not have.

The 29 cut Emacs patterns that get through are two shapes: a repeat left open, ^[A-Z]\{2, and a \s with its syntax class not yet typed. Both are read as text, which is what Emacs does with most of what it does not recognise and what this grammar does where a flavour has not said otherwise.

What it will not tell you

Whether the pattern means anything. (a)\2 refers to a group that does not exist, a{3,2} counts backwards, [z-a] runs the wrong way, and all three are well formed. Counting the groups and comparing the ends of a range are a linter's job.

Whether a flag letter exists. (?i) and (?n) are read the same way, though only .NET has the second. Which letters an engine takes is a table this grammar would rather not carry twice.

Where an ARE's options are allowed to sit. (?i) has to be the first thing in a Tcl or Postgres pattern, and this reads one anywhere, the way it reads a (?i) anywhere in the nine other engines that have one. Every difference between the engines here is a token, and where a token may sit is not a question a token can answer.

Which escapes a schema names. XML Schema names the characters that may follow a backslash and refuses every other one, so \f is a form feed in twelve of the flavours here and an error in a schema. A backslash before punctuation is that punctuation everywhere in this grammar, which is a rule worth keeping for the other eighteen. Where libxml2 is looser than the schema grammar this follows the grammar, so a{2}{2} is a repeat on a repeat and an error here, though xmllint will take it.

Which magic level a Vim pattern is at halfway through. \v and its three siblings choose a level, and a Lezer dialect is chosen once for a parse, so a pattern that starts in one level and switches to another is read at the level it started with. The switch is in the tree as a Flags node, and vimMagic reads the level off the front of a pattern for you. 62 of Vim's own 20,680 patterns do this, and they are left out of the measurement above rather than counted as passes.

What it gives an editor

Syntax highlighting where the metacharacters are painted and the literal text is not, which is the way round that makes a pattern readable. Bracket matching that pairs ) with whichever of the fourteen group openers began it. Folding and indentation for a pattern written over several lines in free-spacing mode.

Completion, from the engine you named:

• the escapes it has, and only those, with a word on what each one matches
• the Unicode property names, general categories and scripts, after \p{
• the POSIX class names after [[:
• the names of the groups in the pattern, after \k<, \g<, (?& or (?P=

The tree

Literal characters are not in it. A pattern is mostly literal text, that text wants no highlighting, and a node per character would be a tree ten times the size of the pattern for nothing. What is left is the skeleton:

(?<area>\d{3})-(\d{4})

Pattern(
  NamedGroup("(?<", GroupName, ">", CharacterClassEscape, Quantifier, ")"),
  Group("(", CharacterClassEscape, Quantifier, ")"))

Group openers are node types named after the text they match, so (?:, (?<= and (*atomic: are all there to be styled, and a tree dump reads like the pattern it came from.

There are four parsers in the package and they agree about their node names, so a theme written for one paints the others and a tree dump reads the same way. Raku's has nodes the others have no need for, Separator and Adverb and RuleCall and Ratchet, and lacks the ones it has no use for.

regexLanguage is the undialected grammar, which is the core all of the engines started from, POSIX extended. regexLanguageFor(flavour, flags) gives one engine's language for use with LanguageSupport, parser is the parser on its own, and flavours, toFlavour and dialectFor are exported for a page that offers a choice of engine. The Raku half answers to raku, rakuRegexLanguage, rakuParser and rakuCompletion as well as to the flavour name.