Rale or Rantemma Advanced Lexing Evaluation

Is the official lexer of the Rantemma Language System (aka RLS).

No need to create a lexer, with Rale it's easy to parse any string into Token as you want or with default lexer parameters.

Exemple is generally in TypeScript

Create your own lexer

Unique Token

The unique token is the most simple kind of tokens, allow you to tokenize one character with a name.

const uniques: Unique[] = {
    {name: "semicolon", character: ";"}
}

Long Token

The long token is a token formed by RegExp. The regex match is executed character by character

const longs: Longs[] = {
    {name: "string", regex: /[a-z0-9]/i}
}

Wrapper Token

The wrapper token is a specific token that wrap other tokens beetween two defined character inside the same token.

const wrappers: Wrapper[] = {
    {name: "braces", begin: "{", end: "}"}
}

Create the instance

With tokens made before, we create the Lexer.

const lexer = new Rale(longs, uniques, wrappers, true); // the last value is for tokenizing unknown tokens.

Use your lexer

const query = "hey { test }";
lexer.parse(query);

That gave you :

[
  {
    "begin": {
      "index": 0,
      "coloumn": 0,
      "line": 0
    },
    "end": {
      "index": 3,
      "coloumn": 3,
      "line": 0
    },
    "type": "long",
    "name": "string",
    "content": "hey"
  },
  {
    "begin": {
      "index": 3,
      "coloumn": 3,
      "line": 0
    },
    "end": {
      "index": 4,
      "coloumn": 4,
      "line": 0
    },
    "type": "long",
    "name": "space",
    "content": " "
  },
  {
    "begin": {
      "index": 4,
      "coloumn": 4,
      "line": 0
    },
    "end": {
      "index": 12,
      "coloumn": 12,
      "line": 0
    },
    "type": "wrapper",
    "name": "curly_bracket",
    "wrapperContent": [
      {
        "begin": {
          "index": 5,
          "coloumn": 5,
          "line": 0
        },
        "end": {
          "index": 6,
          "coloumn": 6,
          "line": 0
        },
        "type": "long",
        "name": "space",
        "content": " "
      },
      {
        "begin": {
          "index": 6,
          "coloumn": 6,
          "line": 0
        },
        "end": {
          "index": 10,
          "coloumn": 10,
          "line": 0
        },
        "type": "long",
        "name": "string",
        "content": "test"
      },
      {
        "begin": {
          "index": 10,
          "coloumn": 10,
          "line": 0
        },
        "end": {
          "index": 11,
          "coloumn": 11,
          "line": 0
        },
        "type": "long",
        "name": "space",
        "content": " "
      }
    ]
  }
]

Enjoy!