@codincod/codemirror-lang-jq

This package implements jq language support for the CodeMirror code editor: a Lezer grammar, highlighting, indentation, folding and completion for the builtins.

The grammar is written from scratch, following jq's own src/parser.y. There was a codemirror-lang-jq on npm before this one, and its author says plainly that it looks for keywords, numbers and strings rather than tokenizing for valid syntax. That shows up in the tree it builds: a program is a flat run of tokens with brackets grouped, so nothing in it says what a pipeline is, where a definition ends, or which of two dots is a field.

It reads the three implementations people write, not only the first. Where gojq, fq and jaq take something jq rejects, this takes it too, and the four places that happens are listed below.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {jq} from "@codincod/codemirror-lang-jq"

const view = new EditorView({
  parent: document.body,
  doc: `def counts:
  reduce .[] as $x ({}; .[$x] |= (. // 0) + 1);

.items
| map(select(.price > 10) | .name)
| counts
| to_entries
| sort_by(-.value)
| .[]
| "\\(.key): \\(.value)"
`,
  extensions: [basicSetup, jq()]
})

What it reads

Two corpora, and they answer different questions.

The first is 195 files and 0.37 MiB of jq written by people who write jq: jq's own src/builtin.jq and test modules, fq's interpreter, gojq's and jaq's builtins, JBOL, and jqjq, which is jq written in jq. 95.90% of those files parse with no error node, against 59.49% for the package this replaces. The average byte is placed 7.80 nodes deep rather than 1.81, across 87 kinds of node rather than 49; a grammar that agrees with everything sits under two.

The second is CodinCod's language-guessing pool, 1706 snippets of jq from Rosetta Code. 88.69% parse with no error node against 57.15%, and 99.91% of the text carries a highlighting tag against 59.09%, over 11 colours rather than 8.

The gap between those two numbers is the pool rather than the grammar, and jq itself says so. Compiling all 1706 snippets with jq 1.8.2 and comparing verdicts file by file:

| | Snippets | | --- | --: | | jq compiles it, this parses it clean | 1511 | | jq rejects it, this reports an error | 193 | | jq rejects it, this parses it clean | 2 | | jq compiles it, this reports an error | 0 |

There is no jq program in either corpus that jq accepts and this rejects. What the pool's remaining 11% holds is snippets that are not jq programs: 102 of them are shell transcripts with the jq invocation still around the filter, and the rest are listings of several programs at once, prose, and entries with a stray semicolon in them. jq will not compile any of them either.

What is jq's, and what is not

Four things here are read that jq rejects. Each is written by one of the other implementations, and none of them can change how a program jq accepts is read, because jq has no other meaning for the syntax.

• Hexadecimal numbers, 0xff, which fq takes.
• Backtick strings, `\d+`, which fq writes a pattern in.
• A format as a definition's name, def @json: …, which is how jaq defines one.
• A file of definitions with no query under it, which is what a jq module is, and which jq's own front end takes when it is imported rather than run.

What took the work

• A leading dot belongs to whatever reads furthest. .. recurses, .5 is a number, .name is a field, and a lone dot is the identity or the start of ."name". A field is one token, which is what makes . name not a field and .end a field even though end closes a conditional.
• A comment ends where jq says it ends. A backslash at the end of one carries it onto the next line, which is jq's lexer reading the backslash and the newline as two characters of comment. It costs a file in the corpus above, a program written for jq 1.6 that jq 1.8 no longer compiles for exactly this reason, and copying it is the point: an editor that disagreed with the compiler would colour a broken program as though it worked.
• A suffix is read once, not twenty-four times. jq's own grammar spells out every combination of .name, […], [:] and the ? that may follow each. Reading a term and then a run of suffixes builds the same tree from a fifth of the rules, and makes .a[0]? and .a?[0] both fall out.
• An object value takes a pipeline but not a comma. {a: .x | f, b: 1} is two keys, and a value that could hold a comma would make it one.
• A key is read before the language's own vocabulary is. {if: 1} names a key if, and so does every other word jq reserves.
• A string holds a query. "\(.a | f)" is parsed rather than skipped, so what is inside an interpolation is coloured like the code it is.

What it does not read

Nothing jq compiles, as far as 1901 files of jq can say. The four extensions above are the only places this is looser than jq, and they are deliberate.

Development

npm install
npm test            # the parse specs and the editor behaviour
npm run corpus      # how much of a directory of jq parses clean
npm run gaps        # what it could not parse, grouped by shape

corpus and gaps delegate to lezer-survey, which is expected to sit beside this checkout. Point either at a directory of .jq files.