@codincod/codemirror-lang-perl

This package implements Perl language support for the CodeMirror code editor, using the Lezer grammar Glenn Rice wrote for codemirror-lang-perl, with the parse fixes below on top of it.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import { EditorView, basicSetup } from 'codemirror';
import { perl } from '@codincod/codemirror-lang-perl';

const view = new EditorView({
 parent: document.body,
 doc: `use strict;
use warnings;

$| = 1;

my @squares = map { $_ ** 2 } 1..10;
printf "%d\\n", $_ for @squares;`,
 extensions: [basicSetup, perl()]
});

API Reference

• perl() → LanguageSupport
• perlLanguage: LRLanguage

What changed

Four parse fixes over the published 0.1.8, each with grammar tests:

• the first dot of a range. A float may end in a decimal point, so 1..10 matched the float 1. and left .10 with nowhere to go. This was 40% of every failure we measured, because a numeric range is in a great deal of Perl: (0..3), for my $i (1..15), 1..2*$n. A float is read in a tokenizer here, which takes the point only when a second dot does not follow it, the way Perl's own lexer does, so the bare 1. is still a float;
• a variable name opening with a digit, which no Perl name does. It cost the modulus its right operand, so $n%4 and 5%2 read %4 and %2 as hashes;
• $|, which fell outside the characters a special variable is drawn from and so read as a dereference of nothing followed by a bitwise or;
• the digits 8 and 9, which two of the tokenizer's own predicates stopped short of. length8() did not parse and "\x39" highlighted as \x3 beside a stray 9.

All four are offered back to the original, in three pull requests: https://github.com/drgrice1/codemirror-lang-perl/pull/1, https://github.com/drgrice1/codemirror-lang-perl/pull/2 and https://github.com/drgrice1/codemirror-lang-perl/pull/3

Tests

npm test builds the parser and runs the grammar tests in test/*.txt.

npm run corpus <directory> parses every .pl, .pm, .t, .PL and .psgi file under a directory and reports how many produce no error node. Against the Perl 5.42 standard library, 1553 files and 18.7 MiB, that is 80.36%, where the original's published 0.1.8 reads 79.91%. Against 300 solutions from Rosetta Code, which is the shape of code this package was written for, 68.33% against 57.67%.

No corpus ships here, because none of that code is ours to redistribute.

Known gaps

An empty pattern does not parse. split //, $s is a match on nothing, and // is also the defined-or operator; the longer of the two wins the tokenizer, and by the time the pattern wants its delimiter the token has been decided. This is the largest single thing left, around a sixth of the failures we still see.

A * directly before an identifier reads as a typeglob rather than a multiplication, so 2*int(rand 2) fails where 2 * int(rand 2) parses. Telling those apart needs the parser state rather than the token.

One file of the Perl standard library, TAP/Parser/Result/Plan.pm, parses clean under 0.1.8 and does not here. Its documentation holds a 1..0 # SKIP: why bother?, and the tree diverges a little before that at a sub the parse takes for a called function. Eight other files of that library go the other way, from failing to clean.