@codincod/codemirror-lang-shell

[ CHANGELOG ]

This package implements shell language support for the CodeMirror code editor, using a Lezer grammar written for this package.

CodeMirror ships a shell mode in @codemirror/legacy-modes. It colours keywords, strings and comments from a word list and a small state machine, and it produces no tree, so nothing above the token level is available: no folding, no indentation, no structural selection. It also has to guess, and guesses wrong in both directions. echo a=b comes out as an assignment. The done in while true; do echo done; done comes out as a keyword, when it is the word the loop prints.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {shell} from "@codincod/codemirror-lang-shell"

const view = new EditorView({
  parent: document.body,
  doc: `#!/bin/sh
for f in *.txt; do
  echo "$(wc -l < "$f") $f"
done
`,
  extensions: [basicSetup, shell()]
})

shell() on its own reads the shell command language, which is what sh, bash, zsh and ksh all are. The two shells that are not take a name:

shell({shell: "fish"})   // or "csh", or "tcsh"

csh() and fish() are the same thing said shorter, and shellLanguageFor answers with the LRLanguage for a name if you are assembling the extensions yourself.

Coverage

The shell command language of IEEE 1003.1, with the bash extensions that are everywhere in practice: [[ ]], (( )), $(( )), arrays and associative arrays, process substitution, here-strings, &>, +=, the ${x/a/b} family of expansions, and the extended patterns @(a|b) that ksh has always had and bash has under shopt -s extglob.

zsh is read by the same grammar rather than by a switch, the way this package's AWK grammar reads gawk: the expansion flags in ${(kv@)h}, an expansion whose name is another expansion, $#array, several names in one for, a list in parentheses, the short forms that drop the do and the then, and the qualifier in *.txt(D). Somebody reading a shell script does not always know which shell wrote it, and reading more than one shell writes costs an editor nothing.

ksh93's compound types are read too. typeset -T Point_t=( ... ) puts commands where an array puts words, and which of the two a =( opens is decided by looking for a ; inside it, since that is a syntax error in an array and the punctuation between two commands everywhere else. A field reached through a subscript, ${people[i].name}, comes with them.

Measured against 754 shell scripts from Rosetta Code, 93.50% parse with no error node, and every zsh-labelled file does. Of the 49 that do not, most are not this shell: 13 are csh and 5 are rc. The rest are terminal sessions with the prompt still in them, program output saved as source, and files the wiki mangled, where a tab under a <<- became spaces or a quote became a typographic one.

The C shell

csh is not a dialect of the shell command language and does not share a grammar with it here. It shares the word layer, the quoting, the pipelines and the redirections, and replaces everything above them: if takes an expression rather than a list of commands, arithmetic is a command of its own spelled @, a loop ends with end rather than done, and there are no functions at all. So it has a grammar of its own in this package, csh.grammar, built from the same three questions and answering to a name of its own.

What it adds is an expression language, and it is the C operator table rather than anything the shell has: == and != compare, =~ matches a pattern, -e and its relatives ask about a file, and { cmd } stands for the status a command exits with. The precedence is C's, so $n % 15 == 0 is the remainder compared against nothing else.

Two of its questions have no counterpart on the sh side. A number is a number only inside an expression, because echo 1 passes a word and csh has no numbers anywhere else. And @ n++ ends the name before the ++ where echo a++ does not, which cannot be asked of the parser, since the word the operator would follow has not been reduced yet; it is asked of the text instead, which says the same thing.

Of the 13 csh files in the corpus, 12 parse with no error node. The one that does not is an alias whose body is a whole csh program inside a raw string, escaped a character at a time.

fish

fish gets a grammar of its own as well, and it is the shortest of the three, because fish threw out most of what makes a shell hard to parse. There is no word splitting. There are no heredocs. There is no ${}, no $(()), no backticks and no subshell: a parenthesis is a command whose output stands in for it, everywhere, and that is the only thing it is. Every block ends with end.

It has one question of its own. A [ opens a subscript where it touches the word in front of it and is the name of the test command where it does not: $argv[1] against [ -f $file ]. Both readings are open at once, so the tokenizer settles it and hands the parser one or the other, never both. A closing parenthesis is the exception it makes, since (seq 10)[-1] indexes what the command printed.

else if is one line and an else on its own is another, so the line ending tells a continued conditional from a nested one. Neither of the other two grammars here can be read that way, and fish can because it has no then.

Every fish file in the corpus parses with no error node, though there are only 21 of them; Rosetta Code files fish under two headings and one of them is a different language with the same name.

What a tokenizer has to settle

Shell is not a language of expressions with a few strings in it. It is a language of words, and nearly every hard question here is about where a word begins and ends. Seven of them are settled by asking the parser what it has room for.

A word is a run of pieces that touch. foo"bar"$baz is one argument and foo "bar" $baz is three, so the pieces of a word are parsed with nothing skipped between them. What closes a word is a token of no width, produced where the shell would end one: at whitespace, or at one of ; & | ( ) < >. The comment rule falls out of that and needs no code of its own. Nothing is skipped inside a word, so a#b is one word and a #b is a word and a comment.

A reserved word is only reserved where a command may start. if opens a conditional and echo if prints two letters, and the difference is nothing but position. A command cannot start until the one before it has been terminated, so the parser has room for done after a ; and not after echo, which is the whole rule. The same question settles {, }, [[, ]] and !, each of which a word would otherwise swallow.

An assignment is a word in the right place. x=1 sets a variable and echo x=1 prints one, and again only position tells them apart. The value has to touch the =, because x= cmd runs cmd with x set to nothing.

A newline ends a command where a command can end, and is skipped everywhere else. That one rule covers every place the shell allows a line to be broken without a marker: after |, &&, do, then, else, {, ( or a comma in an array.

A heredoc's body is nowhere near the << that promised it. <<EOF says the body starts on the next line, and one line may promise several. Rather than carry the promise from token to token, the body is read at the newline by looking back over the line that just ended, and is then skipped like whitespace.

A backslash before a newline is space between words and a letter inside one. grep foo \ continued on the next line passes two arguments, and echo ab\ continued on the next passes one, and the only difference is the space before the backslash. A skipped token cannot also be shifted, so the second case is produced where a word is already under way.

A parenthesis after a glob narrows it, in zsh. *.txt(D) matches the hidden files as well. A parenthesis ends a word everywhere else, so this is asked first, and only where a word has already begun; (cd /tmp) at the start of one is still a subshell.

Two more are settled by counting rather than by asking.

Three parentheses in a row are ambiguous, and how they close says which was meant. ((( n % 2 == 0 )) && echo even) is a subshell holding an arithmetic command; (((x += 2) <= 8)) is arithmetic holding a parenthesised sum. The shell decides this by parsing the arithmetic and taking the other reading when that fails, which a parser with no backtracking cannot do. Counting the parens gets the same answer.

A closing brace needs no semicolon after a compound command. f() { if x; then y; fi } is accepted by bash and f() { echo hi } is not, because } is a reserved word where a command may start and a plain word where an argument may. What the line ended with decides it, which is the question bash asks of the token it read last.

One is settled by parsing the same text twice.

A backtick substitution is closed by the character that opens it, so a grammar reading left to right cannot tell which of the two it has. It is one token, and the commands inside it are parsed afterwards, by handing the range between the backticks back to the same parser. Across the corpus that is 197 backticks and 2 that hold something this grammar cannot read, both of them Tcl filed under the wrong heading.

What the tree does not say

Three things are deliberately shallower than the language reference.

A bracket expression is text. * and ? are marked as the wildcards they are, and [a-z] is left alone, because [ is also the name of a command and [ -f x ] is a great deal more common than a glob.

Arithmetic and words are separate languages. Inside $(( )) a bare name is a variable and 2 is a number. Outside it, echo 42 passes a word, because the shell has no numbers anywhere else and colouring one there would be a lie.

A ${...} keeps its operator and loses the shape of what follows. ${x:-default} is an expansion with an operator in it; the default itself is read as the text and expansions it is made of, without being told apart from a pattern or a replacement.

What it gives an editor

Syntax highlighting, folding for groups, subshells, loop bodies, case statements and command substitutions, indentation that knows fi, done, esac, } and ;; close what came before them, and completion for the reserved words, the shell builtins and the variables the shell sets.

shellLanguage is exported for use with LanguageSupport. parser is the grammar on its own, without the highlighting props and without the re-entry into backticks; shellLanguage.parser is the one an editor runs.

cshLanguage, cshCompletion and cshParser are the same three things for the C shell, and fishLanguage, fishCompletion and fishParser for fish.

Testing it on your own code

npm run corpus -- path/to/your/scripts
npm run corpus:csh -- path/to/your/csh/scripts
npm run corpus:fish -- path/to/your/fish/scripts

It prints the files that failed, the first line of each that did, and the proportion that came out clean. The corpus itself is not included; Rosetta Code is CC BY-SA and cannot be redistributed under this licence.