@codincod/codemirror-lang-pascal

[ CHANGELOG ]

This package implements Pascal language support for the CodeMirror code editor, using a Lezer grammar written for this package.

No Lezer grammar for Pascal existed before this one. A Pascal file in a CodeMirror 6 editor was read by the stream mode carried over from CodeMirror 5, which recognises keywords, strings, numbers and comments, and has no tree to fold, indent or navigate by.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {pascal} from "@codincod/codemirror-lang-pascal"

const view = new EditorView({
  parent: document.body,
  doc: `unit Greeting;

interface

type
  TGreeter = class
  private
    FName: string;
  public
    constructor Create(const AName: string);
    procedure Say;
  end;

implementation

constructor TGreeter.Create(const AName: string);
begin
  FName := AName;
end;

procedure TGreeter.Say;
begin
  WriteLn('hello, ', FName);
end;

end.
`,
  extensions: [basicSetup, pascal()]
})

Coverage

Object Pascal as Free Pascal and Delphi compile it. Units, programs, libraries and packages; classes, records, interfaces, helpers and metaclasses; properties with their specifiers; generics on both the Delphi spelling and the Free Pascal one; variant records; procedural types; RTTI attributes; the whole directive vocabulary; inline variables; anonymous methods; assembler blocks; and every number and string spelling the two compilers accept.

Three other dialects are read as far as they go without costing Object Pascal anything. PascalABC.NET has match, yield, ranges as values, type classes and a variable that takes its type from its value. Pascaline has joins, fixed and an array written with its length. Standard Pascal passes a subprogram as a parameter.

The top level is deliberately looser than a compiler's. An editor is shown fragments as often as whole files, so a bare statement, a lone declaration and a unit cut off halfway all parse.

Measured against 36092 files of published Pascal, 433 MiB, 95.02% parse with no error node. By project:

| Project | Files | Clean | | --- | ---: | ---: | | Free Pascal | 22332 | 96.57% | | Lazarus | 4976 | 96.32% | | PascalABC.NET | 4405 | 90.62% | | Castle Game Engine | 2470 | 95.43% | | mORMot | 576 | 97.92% | | Indy | 803 | 95.89% | | Pascal-P6 | 530 | 47.55% |

The five Object Pascal projects together are 96.44% clean. Much of what is left in the other two is not meant to parse: 246 of Pascal-P6's 278 failures are its ISO conformance suite, files whose whole purpose is to be rejected, and PascalABC.NET keeps a directory of compiler error tests beside its samples. Set those aside, along with Free Pascal's must-not-compile tests, and 95.70% of the remaining 35653 files parse.

The extensions counted are .pas, .pp, .inc, .dpr, .lpr and .dpk, which is what lezer-survey knows Pascal by and needs no flag to use. An include file is a piece of a unit rather than a unit, and Free Pascal writes 6752 of them; a Delphi package is a file Delphi compiles. Both are the language and both are counted.

The corpus does not ship with this package. Every project in it has its own licence, so npm run corpus takes the path to your own copy.

What a tokenizer has to settle

Some things about Pascal cannot be written as grammar rules, and they are the whole of tokens.ts.

Case does not matter. BEGIN, Begin and begin are one word, and real code uses all three. The reserved words are therefore recognised by a function that lowercases a word before looking it up, rather than by a list of literals in the grammar file.

Most of the words are not reserved. read, write, name, index, message, default, private and the calling conventions are directives. They mean something in one position and are ordinary names everywhere else, and a single unit will often use one both ways. They come in through @external extend, which offers the parser the keyword and leaves the identifier standing wherever the keyword cannot go.

An angle bracket is two different things. TList<Integer> and a < b open on the same character. Where a type belongs there is no comparison to be had and the parser says so. Where an expression belongs the answer is further along the line, so the tokenizer reads ahead to find it.

An assembler block is not Pascal. asm ... end holds another language, with its own comment characters and its own uses for [ and %. It is taken whole by a tokenizer that looks only for the end that closes it.

A modifier says nothing about what it binds. static behind a header is a directive and ends with a semicolon; in front of one it belongs to the method or the field that follows. Free Pascal's [public, alias: 'x'] and an RTTI attribute are the same bracket, told apart by the word inside it. Both answers lie past what a parser is allowed to look at, so a tokenizer reads ahead and hands back a different token.

The branch of a conditional the compiler would not read is skipped. A compiler reads one arm of a {$ifdef} and an editor is shown both, so a grammar that reads what is in front of it sees one declaration written twice. Everything from an {$else} to the {$endif} that closes it is passed over, which is what the compiler does with the same file, and the only reading that yields a tree.

What the tree does not say

A subprogram body is a sibling of its header, never a child of another subprogram. Nothing before the body says whether there will be one: an interface entry, a forward, an external and a full definition all open the same way, and the answer can be thousands of lines off. A parser that goes looking nests every header in a unit's interface section inside the one above it, and buries implementation at the bottom. So a header ends the moment the next declaration starts, and a procedure nested in another stands beside it. Folding, indentation and highlighting want the body, and none of them wants the nesting.

A parenthesised list is not read for meaning. The same brackets spell a call, an index, an enumeration, a record constant and an array constant, and which one is meant depends on what the name in front of them refers to. This grammar knows what no name refers to, so it separates the five by what is inside the brackets and no further.

^A is a name, not a control character. Turbo Pascal wrote control characters as a caret and a letter, and Delphi still accepts #13^J. The same caret spells a pointer dereference, and PLooper^ and ^ALooper differ only in which side of the name it falls on. Real code dereferences constantly and writes control characters rarely, so the caret is always the operator.

A few PascalABC.NET spellings are left out. Tuples, slices, if used as an expression, a nullable T? and parameterless lambdas each collide with something Object Pascal writes, and Object Pascal is what the grammar is for. A tuple type and an enumeration are the same brackets around the same names; a nullable type and the conditional expression are the same question mark.