jlex

Install

npm i jlex

Usage

npx jlex --help
Usage: jlex [options] <filename>

A tiny wrapper around jison-lex that allows you to use jison-lex as a standalone (flex like) processor.

Options:
  -V, --version            output the version number
  -o, --output <fileName>  Output file name
  -v, --verbose            Enable verbose output
  -h, --help               display help for command
See https://github.com/ULL-ESIT-PL/jlex/blob/main/README.md for more help

Example

jlex is a tiny wrapper around jison-lex that allows you to use the script jlex as standalone (flex like) processor.

Assuming the following lexer in file examples/example.l:

comment [/][*](.|[\r\n])*?[*][/]
%%
\s+|{comment}         /* skip whitespace */
[0-9]+                return 'NUMBER';
[-+*/]                return 'OPERATOR';
<<EOF>>               return 'EOF';
.                     return 'INVALID';

Compile it with:

➜  jlex git:(main) ./jlex.js examples/example.l -v -o examples/example.js
📖 Reading lexer grammar from: examples/example.l
📝 Generated 11340 characters of lexer code
🔄 Applied transformation pattern: /var\s+lexer\s*=/
📝 Writing file: examples/example.js
✅ Successfully processed examples/example.l → examples/example.js
📊 Output size: 11.1KB

This produces a Common.JS module examples/example.js you can use with a simple require like in the file main.js below:

const lex = require("./example");
const input = process.argv[2] || "2\n-/* a comment*/\n3";
lex.setInput(input);

const results = [];

results.push({ type: lex.lex(), lexeme: lex.yytext, loc: lex.yylloc });
results.push({ type: lex.lex(), lexeme: lex.yytext, loc: lex.yylloc });
results.push({ type: lex.lex(), lexeme: lex.yytext, loc: lex.yylloc });
results.push({ type: lex.lex(), lexeme: lex.yytext, loc: lex.yylloc });

console.log(results); 

When you execute the former program, you get:

➜  jlex git:(main) ✗ node examples/main.js 
[
  {
    type: 'NUMBER',
    lexeme: '2',
    loc: { first_line: 1, last_line: 1, first_column: 0, last_column: 1 }
  },
  {
    type: 'OPERATOR',
    lexeme: '-',
    loc: { first_line: 2, last_line: 2, first_column: 0, last_column: 1 }
  },
  {
    type: 'NUMBER',
    lexeme: '3',
    loc: { first_line: 3, last_line: 3, first_column: 0, last_column: 1 }
  },
  {
    type: 'EOF',
    lexeme: '',
    loc: { first_line: 3, last_line: 3, first_column: 1, last_column: 1 }
  }
]

Using the lexer from a Jison grammar

In file examples/grammar.jison you'll find an example of setting the generated lexer to be used from a Jison grammar. The key is to set the lex attribute of the parser object to the generated lexer:

%%
const lexer = require("./example.js");
parser.lexer = lexer;

Compile the grammar with:

➜  jlex git:(main) ✗ npx jison examples/grammar.jison -o examples/parser.js

And use the parser:

➜  jlex git:(main) ✗ node
Welcome to Node.js v25.6.0.
Type ".help" for more information.
> p = require("./examples/parser.js")
{
  parser: { yy: {} },
  Parser: [Function: Parser],
  parse: [Function (anonymous)],
  main: [Function: commonjsMain]
}
> p.parse(`3
| -
| /* comment */
| 1`)
{
  type: 'OPERATOR',
  left: {
    type: 'number',
    value: 3,
    loc: { first_line: 1, last_line: 1, first_column: 0, last_column: 1 }
  },
  right: {
    type: 'number',
    value: 1,
    loc: { first_line: 4, last_line: 4, first_column: 0, last_column: 1 }
  },
  loc: { first_line: 2, last_line: 2, first_column: 0, last_column: 1 }
}

The Lexical Analyzer Object

Here is a description of the attributes of the lexer object:

{
  EOF: 1,
  parseError: [Function: parseError],
  setInput: [Function: setInput],
  input: [Function: input],
  unput: [Function: unput],
  more: [Function: more],
  reject: [Function: reject],
  less: [Function: less],
  pastInput: [Function: pastInput],
  upcomingInput: [Function: upcomingInput],
  showPosition: [Function: showPosition],
  test_match: [Function: test_match],
  next: [Function: next],
  lex: [Function: lex],
  begin: [Function: begin],
  popState: [Function: popState],
  _currentRules: [Function: _currentRules],
  topState: [Function: topState],
  pushState: [Function: pushState],
  stateStackSize: [Function: stateStackSize],
  options: { moduleName: 'example' },
  performAction: [Function: anonymous],
  rules: [ /^(?:\s+)/, /^(?:[0-9]+)/, /^(?:-)/, /^(?:$)/, /^(?:.)/ ],
  conditions: { INITIAL: { rules: [Array], inclusive: true } }
}

Writing a Jison compatible lexer by hand

See file examples/manual-lexer.js to see an example that illustrates how to write a Jison compatible lexer by hand.

To use with the grammar in the examples folder, set the parser.lexer to the hand-written one:

➜  jlex git:(main) ✗ git -P diff examples/grammar.jison 
diff --git a/examples/grammar.jison b/examples/grammar.jison
index 0b99d40..c9e974d 100644
--- a/examples/grammar.jison
+++ b/examples/grammar.jison
@@ -12,6 +12,7 @@ expr
       {
         $$ = {
           type: "OPERATOR",
+          lexeme: $2,
           left: $1,
           right: $3,
           loc: @2
@@ -29,5 +30,7 @@ expr
 
 %%
 
-const lexer = require("./example.js");
+//const lexer = require("./example.js");
+const lexer = require("./manual-lexer.js");
+
 parser.lexer = lexer;
\ No newline at end of file

Then compile the grammar:

➜  jlex git:(main) ✗ npx jison examples/grammar.jison -o examples/parser.js

and use it like this:

➜  jlex git:(main) ✗ node                                                  
Welcome to Node.js v25.6.0.
Type ".help" for more information.
> p = require("./examples/parser.js")
{
  parser: { yy: {} },
  Parser: [Function: Parser],
  parse: [Function (anonymous)],
  main: [Function: commonjsMain]
}
> p.parse("2*3")
{
  type: 'OPERATOR',
  lexeme: '*',
  left: {
    type: 'number',
    value: 2,
    loc: { first_line: 1, last_line: 1, first_column: 0, last_column: 1 }
  },
  right: {
    type: 'number',
    value: 3,
    loc: { first_line: 1, last_line: 1, first_column: 2, last_column: 3 }
  },
  loc: { first_line: 1, last_line: 1, first_column: 1, last_column: 2 }
}