@codincod/codemirror-lang-ocaml

This package implements OCaml language support for the CodeMirror code editor: a Lezer grammar, highlighting, indentation, folding and completion for the standard library.

The grammar is written from scratch. OCaml has never had a Lezer grammar, and what a CodeMirror 6 editor could reach for until now was the mllike stream mode in @codemirror/legacy-modes, which reads one keyword list for OCaml, F# and Standard ML and has no idea where a definition ends.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {ocaml} from "@codincod/codemirror-lang-ocaml"

const view = new EditorView({
  parent: document.body,
  doc: `type shape =
  | Circle of float
  | Rect of {width : float; height : float}

let area = function
  | Circle r -> Float.pi *. r *. r
  | Rect {width; height} -> width *. height

let () =
  [Circle 1.0; Rect {width = 2.0; height = 3.0}]
  |> List.map area
  |> List.iter (Printf.printf "%.2f\\n")
`,
  extensions: [basicSetup, ocaml()]
})

What it reads

Measured against the stream mode it replaces, over 1011 files of OCaml from CodinCod's language-guessing pool: the stream mode carries a highlighting tag on 90.34% of the text in 8 colours, with 54.99% of it in the busiest one. This package tags 97.76% in 15 colours, with 37.22% in the busiest. Over the same 1011 files, 88.03% parse with no error node.

Over the OCaml compiler and standard library, Jane Street's core and the dune build system, 2996 files and 17.98 MiB, 99.00% parse with no error node.

Over 20000 files of published OCaml, 144 MiB out of a thousand repositories, 98.49%. The denominator there is the compiler's: ocamlc -stop-after parsing turns down 1137 of those files, most of them HOL Light and camlp4 writing terms between backticks, which is a preprocessor's syntax rather than the language's. Of the 18863 files it does parse, 285 still hold an error node, and every one of those is a thing to fix.

Four things took most of the work.

• A comment holds a language. (* (* *) *) nests, so the depth has to be counted rather than matched. A string inside a comment is read as a string and a character literal as a character, which is OCaml's own rule and not a nicety: (* the "*)" case *) and (* '"' *) both end where their author meant them to, and a tokenizer that stopped at the first star-paren would colour the rest of the file as a comment.
• An apostrophe is three things. 'a is a type variable, 'a' is a character, and x' is an ordinary name. The name is read first so the quote in it never reaches the tokenizer that would open a literal, the character is tried next because it is the only one with a closing quote, and what is left is a type variable. OCaml lexes the quote and the name as two tokens, so ' a is one variable and is read as one.
• The semicolon that ends a sequence. begin f (); g (); end may dangle its last separator, and so may a record, a list and the arm of a clause. The grammar cannot see that the statement after the separator is missing until it has read the token that closes the block, so the tokenizer decides: a semicolon standing in front of a closer, or in front of a word that cannot open an expression, is a different token from the one that separates. Spelling it in the grammar instead cost two and a half points of the corpus, because the parser weighed an empty statement against the bar that ends a clause and got the bar wrong in every file that wrote one.
• A capitalised word is a module or a constructor. OCaml spells the two the same, and the dot that follows one of them is the whole of the difference. That dot arrives too late for the grammar, so the tokenizer reads it: a capitalised word with a dot against it is a module segment and can be nothing else. Some x is a constructor, Set.empty is a module, and a theme can colour them apart.

Smaller things worth naming.

• Operators are not declared. >>=, |> and +. are ordinary names in OCaml and a library may write its own, so they are classified by their first character rather than listed. A leading !, ~ or ? binds tighter than application, which is what hands f !r what the reference holds rather than applying f to an exclamation mark.
• Binding operators are one word. let* and and+ are a keyword with an operator against it, and reading the two apart would leave the grammar with a let whose name was an operator. A percent is deliberately not one of the characters they may be spelled with: let%lwt hands the binding to a preprocessor and is a let with an extension after it.
• Attributes and extensions are read whole. [@inline], [@@deriving], [@@@warning], [%blob] and [%%if] reach the tree with their name apart from their payload, and the payload keeps whatever colour the code inside it earned. The name may be a reserved word, because the name comes first and nothing else can stand there.
• Quoted strings take no escapes. {|raw|} and {sql|select 1|sql} end at the delimiter they opened with and at nothing else.

What it does not read

• Labelled tuples. (~x, y) as a pattern and (x:int * y:string) as a type are OCaml 5.4 syntax that the compiler's own sources have started using. The parenthesised form collides with the label a function type already puts in the same place, and it is most of the gap in the 99.00% above.
• An arm mixing a value pattern and an exception pattern. | None | exception _ -> is legal and rare. Either kind alone works, and so does | exception (A | B) ->.
• Indexing operators. let (.%()) map e = ... defines one, and the name is read as an operator in brackets rather than as the whole of .%().
• A record path that starts lowercase and continues capitalised. {flag.Flag.Internal.count with ...} opens a module halfway along a field path.
• M.[...] in a pattern. A module opened for the length of a pattern works with parentheses, and a record works with braces; the list form is the one shape that would make every list pattern ambiguous with a constructor applied to one.
• A tree that groups a :: b as x and a | b, c the way the manual does. Both parse. OCaml binds as loosest and | looser than ,; this grammar binds the tuple loosest, which changes how the pattern nests and nothing else.

Measuring it yourself

npm run measure -- path/to/ocaml/files
npm run gaps -- path/to/ocaml/files

No corpus ships with the package, because none of that code is ours to redistribute. Point it at your own; it counts files with no error node and groups them by directory, so a project that drags the total down names itself.

Run the compiler over the same directory before believing the number. ocamlc -stop-after parsing is the honest denominator: a file it turns down is a file this grammar cannot be charged for, and a directory of OCaml gathered by extension holds more of those than anybody expects.