isotropic-lexer

A small, fast tokenizer. Turn text into a stream of tokens through a generator or a transform stream. It's like String.prototype.split if you could split on multiple things at once.

Why Use This?

• Synchronous or Streaming: Iterate a generator for immediate results, or pipe text through a web TransformStream for large or chunked input
• Line and Column Tracking: Every token carries its absolute offset, line index, and column index
• Recognizes the Gaps: Text that matches no lexeme is emitted as its own token, so the input is fully accounted for
• Configurable Line Endings: Single-character, multi-character, or disabled entirely
• Minimal Dependencies: Builds on isotropic-later and isotropic-make

Installation

npm install isotropic-lexer

How It Works

A lexer is created from a lexicon: the set of strings it should recognize as tokens. From that lexicon it compiles a single regular expression, once, and reuses it for every call.

Lexing produces a token for each recognized lexeme and a token for each run of text between recognized lexemes. Every token records where it is (index), what line it is on (lineIndex), and how far into that line it sits (columnIndex). When two lexemes could match at the same spot, the longer one is preferred, so a lexicon of '<' and '<=' recognizes '<=' as a single token rather than two.

Usage

import _lexer from 'isotropic-lexer';

{
    const lexer = _lexer({
        lexicon: [
            '+',
            '-',
            '*',
            '/',
            '(',
            ')'
        ]
    });

    for (const token of lexer.lex({
        text: '12 + (34 * 56)'
    })) {
        console.log(token);
    }

    // Operators and parentheses arrive as recognized tokens; the numbers and
    // spaces between them arrive as their own tokens. Each token reports its
    // index, lineIndex, and columnIndex.
}

Streaming Large Input

getTransformStream() returns a standard web TransformStream that accepts string chunks and emits the same tokens. Node's Readable.toWeb and a TextDecoderStream are all that is needed to feed it from a file or socket.

import _lexer from 'isotropic-lexer';
import {
    Readable
} from 'node:stream';
import {
    createReadStream
} from 'node:fs';

{
    const lexer = _lexer({
        lexicon: [
            'function',
            'return',
            '{',
            '}',
            '(',
            ')'
        ]
    });

    await Readable.toWeb(createReadStream('source.js')).pipeThrough(new TextDecoderStream()).pipeThrough(lexer.getTransformStream()).pipeTo(new WritableStream({
        write (token) {
            console.log(token);
        }
    }));
}

API

lexer(options)

Creates a lexer instance. The lexicon and line ending are fixed for the life of the instance, and the matcher is compiled once and shared by every call.

Parameters

• options (Object, optional): Configuration object.
  • lexicon (Array, optional): The strings to recognize as tokens. When more than one entry could match at the same position, the longest match is preferred. Empty and duplicate entries are ignored. Defaults to [], which recognizes nothing, so the entire input is returned as a single unmatched token.
  • lineEnding (String, optional): The substring that separates lines, used to compute lineIndex and columnIndex. May be more than one character, for example '\r\n'. An empty string disables line counting and treats the input as a single line. Defaults to '\n'.

Returns

• (Object): A lexer instance.

lexer.lex(options)

Returns a generator that yields one token for each recognized lexeme and one token for each run of text between lexemes.

Parameters

• options (Object): Configuration object.
  • complete (Boolean, optional): Whether text is the entire input. When false, a trailing run that might be the beginning of a longer lexeme is withheld from the output and reported as remainder instead, so a lexeme that straddles a boundary is never mis-tokenized. Defaults to true.
  • index (Number, optional): The absolute offset of the first character of text within the larger input. Defaults to 0.
  • lineBeginIndex (Number, optional): The absolute offset at which the current line began. Defaults to 0.
  • lineIndex (Number, optional): The line index of the first character of text. Defaults to 0.
  • text (String, required): The text to tokenize.

Returns

• (Generator): A generator of token objects. Each token has:

  • columnIndex (Number): The offset of the token from the start of its line.
  • index (Number): The absolute offset of the token.
  • lexeme (String): The matched text. Either a lexeme or the string between lexemes.
  • lineIndex (Number): The zero-based line index of the token.

  When iteration finishes, the generator's return value is the state needed to resume with another chunk: an object with index, lineBeginIndex, lineIndex, and remainder (the withheld trailing text, or undefined).

lexer.getTransformStream()

Creates a web TransformStream whose writable side accepts string chunks and whose readable side emits the same token objects that lex() yields. Partial lexemes are carried across chunk boundaries, so streaming a string in pieces produces the same recognized tokens as lexing the whole string at once.

Returns

• (TransformStream): A transform stream from strings to tokens.

Examples

Separating Recognized Tokens From the Text Between Them

Because unmatched runs are emitted as tokens, a Set built from the lexicon is enough to tell them apart.

import _lexer from 'isotropic-lexer';

{
    const lexicon = [
            'cat',
            'dog'
        ],
        lexiconSet = new Set(lexicon),
        lexer = _lexer({
            lexicon
        });

    for (const token of lexer.lex({
        text: 'the cat and the dog'
    })) {
        if (lexiconSet.has(token.lexeme)) {
            console.log('keyword:', token.lexeme);
        } else {
            console.log('text:', JSON.stringify(token.lexeme));
        }
    }
}

Tracking Lines and Columns

Every token reports its line and the column within that line, so a recognized lexeme can be located precisely in multi-line input.

import _lexer from 'isotropic-lexer';

{
    const lexer = _lexer({
        lexicon: [
            'TODO'
        ]
    });

    for (const token of lexer.lex({
        text: 'line one\nline TODO two\nlast line'
    })) {
        if (token.lexeme === 'TODO') {
            // lineIndex 1, columnIndex 5
            console.log(`found at line ${token.lineIndex}, column ${token.columnIndex}`);
        }
    }
}

Using a Custom Line Ending

lineEnding may be more than one character, which keeps lineIndex and columnIndex correct for input that uses Windows-style line breaks.

import _lexer from 'isotropic-lexer';

{
    const lexer = _lexer({
        lexicon: [
            'X'
        ],
        lineEnding: '\r\n'
    });

    for (const token of lexer.lex({
        text: 'aa\r\nbbXcc\r\ndd'
    })) {
        // The X token reports lineIndex 1.
        console.log(token);
    }
}

Lexing a String in Chunks by Hand

getTransformStream() is the easy path, but the same chunk-to-chunk behavior is available directly from lex() by feeding each chunk's returned state into the next call. Pass complete: false for every chunk except the last, prepend the previous remainder, and finish with a complete: true call.

import _lexer from 'isotropic-lexer';

{
    const chunks = [
            'foo',
            'bar'
        ],
        lexer = _lexer({
            lexicon: [
                'foobar'
            ]
        }),
        tokens = [];

    let state = {
        index: 0,
        lineBeginIndex: 0,
        lineIndex: 0,
        remainder: ''
    };

    chunks.forEach((chunk, chunkIndex) => {
        const generator = lexer.lex({
            complete: chunkIndex === chunks.length - 1,
            index: state.index,
            lineBeginIndex: state.lineBeginIndex,
            lineIndex: state.lineIndex,
            text: `${state.remainder ?? ''}${chunk}`
        });

        let result = generator.next();

        while (!result.done) {
            tokens.push(result.value);
            result = generator.next();
        }

        state = result.value;
    });

    // A single 'foobar' token, even though it arrived split across two chunks.
    console.log(tokens);
}

Contributing

Please refer to CONTRIBUTING.md for contribution guidelines.

Issues

If you encounter any issues, please file them at https://github.com/ibi-group/isotropic-lexer/issues