@codincod/codemirror-lang-assembly

[ CHANGELOG ]

This package implements assembly language support for the CodeMirror code editor, using a Lezer grammar written for this package.

It is not written for one assembler. It reads 8080, Z80, NASM, FASM, MASM, the GNU assembler in both its syntaxes, 68000, ARM and AArch64, 6502, MIPS, RISC-V, PDP-8 and the IBM 360 assemblers, because it is written for the shape all of them share rather than the vocabulary none of them do.

CodeMirror 6 has never shipped assembly, and no Lezer grammar for it existed anywhere before this one. What a CodeMirror 6 editor could offer until now was the CodeMirror 5 stream mode for the GNU assembler, which knows AT&T syntax and nothing else.

Written in part for CodinCod, a competitive coding platform, where it colours the editor people solve puzzles in.

This code is released under an MIT license.

Usage

import {EditorView, basicSetup} from "codemirror"
import {assembly} from "@codincod/codemirror-lang-assembly"

const view = new EditorView({
  parent: document.body,
  doc: `	global	main
	section	.text
main:
	push	rbp
	mov	rbp, rsp
	xor	eax, eax
	lea	rdi, [fmt]
	call	printf@PLT	; say it
	leave
	ret

	section	.data
fmt	db	"Hello, World!", 10, 0
`,
  extensions: [basicSetup, assembly()]
})

Coverage

Measured against the 1966 assembly snippets of the language-guessing corpus behind CodinCod, which are drawn from Rosetta Code and carry every assembler listed above: 97.81% parse with no error node, and 100% of their non-whitespace characters come back with a highlight tag. The stream mode this replaces there tagged 30.39% of the same corpus.

The tag matters as much as the tree. A grammar that parses a file and leaves half of it black has failed at the only job a highlighter has, and one that colours everything by reading the file as one long string has failed at the other. Both numbers are in test/corpus.ts, which takes the path to your own directory of assembly files.

Of the files that do not parse cleanly, most are IBM 360 listings, where the commentary on a line is written in the operand field with nothing in front of it and only the column it starts in says where the operands stopped. A few are not assembly at all: the corpus files them under assembly because a Rosetta Code page did.

The shape every assembler agrees on

label:   mnemonic   operand, operand    ; comment

That is the whole grammar. Everything an assembler argues about sits inside those four slots, and a grammar written for one assembler is wrong about the next one, so this one is written for none of them and reads the slots.

The tree says less than a language grammar's would. There is no addressing-mode node, because two assemblers spell the same mode differently and a third gives that spelling to another mode. An operand is a run of atoms, and what the atoms mean is the assembler's business. What the tree does say is which slot every character stands in, which is what a highlighter needs and what an assembler programmer reads.

What a tokenizer has to settle

Four things cannot be written down as a grammar, and they are the whole of tokens.ts.

The first word of a line is not the first word of a statement. main: is a label, mov is a mnemonic and .text is a directive, and all three are a word at the start of a line. A colon settles it when there is one. When there is none, the word behind it does: bdos equ 5h declares bdos, because equ can only follow a name, and push a does not, because a cannot. That one lookahead is what reads FASM's message db 'x' and MASM's msg BYTE "x" and the CP/M assembler's wboot equ 0 without being told which assembler wrote them. The size words are the exception it has to know about, because mov byte ptr [x], 1 writes one of them where a declaration would: the ptr behind it says the line is an instruction after all.

A word means one thing in a statement and another in an operand. byte is a directive that reserves one and a size that qualifies one. a is a mnemonic-shaped word and it is the accumulator. The parser knows which slot it is standing in, so the tokenizer asks it rather than guessing, and hands back a different token for each. That is also what separates %macro, which opens a statement, from %eax, which is a register.

A sigil is not a sigil. $ is hexadecimal to a 6502 assembler, an immediate to gas, a register prefix to a MIPS one and the current address to NASM. % is a register, a binary literal, a remainder or a preprocessor word. # prefixes an immediate or opens a comment. The character behind the sigil and the slot it stands in decide which, and no token rule can see either. Hexadecimal wins the one real tie: $a0 is a MIPS register and a 6502 address, and an assembler that prefixes its addresses writes far more of them than one that prefixes its registers writes registers.

A comment character is also an operator. # opens a comment when a space follows and prefixes an immediate when a digit does; @ opens one when a space follows and joins a symbol to its relocation when it does not. Both of those are decided by the character behind them. * and / are decided by where they stand: each opens a comment in column one, to the 68000 assemblers and to the PDP-8's, and multiplies or divides everywhere else.

Lezer's own tokenizer runs before an external one and settles a token wherever it can, so *, / and % are missing from the operator token rule on purpose. Left there, they would never reach the tokenizer that has to decide what they are.

What the tree does not say

A register is a list, and the list is shared. A register is spelt like any other word, and only the assembler reading it knows that d0 names one and msg does not, so this package carries the registers of every instruction set it has been measured against and strips the prefix before the lookup. %eax, $t0 and eax all land on the same entry, which is what lets one grammar read an assembler that insists on a prefix and one that forbids it. The cost is a label named a or x read as a register, which is the trade every assembly highlighter makes.

A mnemonic is not in a list. There is no opcode table here and there will not be one. A mnemonic is the first word of a statement; that is what makes it a mnemonic, and it is why an instruction set nobody has heard of reads as well as x86 does.

A bracket does not span a line. Assembly has no expression that runs past the newline, so a ( left open ends at the line rather than swallowing the rest of the file. The same goes for a quote: an unterminated literal ends where the line does, because a lone apostrophe standing for a character is ordinary in half these assemblers.

A column is not counted. The IBM assemblers put the commentary of a line in the operand field with no marker in front of it, and only the column it starts in says the operands have stopped. Reading that would mean counting columns, which would be wrong for every other assembler on the list, so those comments are read as operands.

Prior art

There was no Lezer grammar to be compatible with, so this one is not compatible with anything. Two things were read while writing it: @defasm/codemirror, which highlights x86-64 in gas syntax by assembling it, and tree-sitter-asm, whose generic grammar makes the same bet this one does: that the shape is worth more than the vocabulary.