@codincod/codemirror-lang-r

[ CHANGELOG ]

This package implements R language support for the CodeMirror code editor, using a Lezer grammar ported from R's own.

CodeMirror 6 has never shipped R. What stands in for it is the stream mode carried over from CodeMirror 5, which colours a keyword list and builds no tree, so nothing above it can fold a function, indent an argument list or ask what the name under the cursor is. There is one other Lezer grammar for R on npm, and it reads under a third of real R: no unary minus, no < or >, no :, no |>, no %in%, no braceless body, no \(x), no raw string.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {r} from "@codincod/codemirror-lang-r"

const view = new EditorView({
  parent: document.body,
  doc: `greet <- function(name = "world") {
  sprintf("Hello, %s!", name)
}
`,
  extensions: [basicSetup, r()]
})

Coverage

R 4.5. Every assignment R has, including :=; functions written with function and with \(x); if, for, while and repeat with braces and without; calls with named, empty and dotted arguments; [, [[ and their empty subscripts; ::, :::, $ and @; formulas; the native pipe and the ones packages define between percent signs; raw strings; backtick names; integer, double, hexadecimal and complex literals; roxygen comments; and ..., ..1 and the pipe placeholder.

Measured against 2 322 R files from eight open source projects (R itself, ggplot2, dplyr, tidyr, data.table, shiny, testthat and devtools), which is 15 MiB of code, of which 99.70% parse with no error node. The seven files that do not are not R: three are fixtures in R's own test suite that hold a deliberate parse error, two are shiny templates holding {{ }} placeholders, one is testthat's syntax-error fixture, and one is written in Latin-1.

What the grammar cannot say

The grammar is a port of src/main/gram.y from the R sources, which is an LALR grammar and so nearly a Lezer grammar already. Its precedence table is R's read from the other end, and every production answers to one there. Three things are settled in src/tokens.ts instead, because no context free rule can decide them.

A line break ends an expression when the expression is finished. R has no statement terminator. x <- 1 followed by a line holding -2 is two expressions, and x <- 1 + followed by a line holding 2 is one. The tokenizer offers a terminator of no width at every line break and the parser takes it only where an expression may end, which is nowhere inside a bracket and nowhere after an operator still waiting for its right hand side. That is the whole rule, and it needs one exception: a line may open with else, which R itself allows inside braces and refuses outside them.

A raw string carries its own delimiter. r"(a\d+)" holds a backslash and ends at )". r"---(x)---" ends only at the dashes it opened with, and how many there were is not something a regular expression can remember.

A string is cut into pieces. The escapes inside it are coloured apart from the text around them, so the tokenizer hands over a start, the runs of text, the escapes and an end. Each quote character has its own set of those tokens, which is how the tokenizer knows what it is looking for without keeping any state.

Testing

npm test                       # the parse specs, then highlighting and indentation
npm run corpus -- path/to/code # error nodes over a directory of real R files

No corpus ships with the package, because none of that code is ours to redistribute. test/corpus.ts takes a directory and reports the share of files that parse with no error node, grouped by directory so that a project which drags the total down names itself.