Pascal Tokenizer

A simple and efficient tokenizer for the Pascal programming language, written in TypeScript.

Installation

Install the package using npm:

npm install pascal-tokenizer

Usage

Basic Usage

You can import the tokenizePascal function and optionally the PascalToken and TokenType types

Input example:

import { tokenizePascal, PascalToken, TokenType } from "pascal-tokenizer";

const pascalCode = `
program HelloWorld;
begin
  writeln('Hello, World!'); // Example comment
end.
`;

// Tokenize the code (comments skipped by default)
const tokens: PascalToken[] = tokenizePascal(pascalCode);
console.log(tokens);

// Example: Find the first keyword token
const firstKeyword = tokens.find((token) => token.type === "KEYWORD"); // You can use the imported TokenType for type checking

// Tokenize including comments
const tokensWithComments = tokenizePascal(pascalCode, false);
console.log(tokensWithComments);

Default output:

[
  { type: "KEYWORD", value: "program" },
  { type: "IDENTIFIER", value: "HelloWorld" },
  { type: "DELIMITER_SEMICOLON", value: ";" },
  { type: "KEYWORD", value: "begin" },
  { type: "IDENTIFIER", value: "writeln" },
  { type: "DELIMITER_LPAREN", value: "(" },
  { type: "STRING_LITERAL", value: "Hello, World!" },
  { type: "DELIMITER_RPAREN", value: ")" },
  { type: "DELIMITER_SEMICOLON", value: ";" },
  { type: "KEYWORD", value: "end" },
  { type: "DELIMITER_DOT", value: "." },
  { type: "EOF", value: "" },
];

Output with comments:

[
  { type: "KEYWORD", value: "program" },
  { type: "IDENTIFIER", value: "HelloWorld" },
  { type: "DELIMITER_SEMICOLON", value: ";" },
  { type: "KEYWORD", value: "begin" },
  { type: "IDENTIFIER", value: "writeln" },
  { type: "DELIMITER_LPAREN", value: "(" },
  { type: "STRING_LITERAL", value: "Hello, World!" },
  { type: "DELIMITER_RPAREN", value: ")" },
  { type: "DELIMITER_SEMICOLON", value: ";" },
  { type: "COMMENT_LINE", value: "// Example comment" },
  { type: "KEYWORD", value: "end" },
  { type: "DELIMITER_DOT", value: "." },
  { type: "EOF", value: "" },
];

Including Comments

By default, comments are skipped. If you want to include comments in the token stream:

const tokens = tokenizePascal(pascalCode, false); // false = don't skip comments

Token Structure

Each token has the following structure:

interface PascalToken {
  type: TokenType;
  value: string;
}

Token Types

The tokenizer recognizes the following token types:

| Token Type (TokenType) | Symbol / Example in Pascal | Description | | :----------------------- | :----------------------------------- | :----------------------------------------- | | Comments | | | | COMMENT_BLOCK_BRACE | { ... } | Comment delimited by braces | | COMMENT_STAR | (* ... *) | Comment delimited by parenthesis/asterisk | | COMMENT_LINE | // ... | Single-line comment (Delphi/FPC style) | | Operators | | | | OPERATOR_ASSIGN | := | Assignment | | OPERATOR_LESS_EQUAL | <= | Less than or equal to | | OPERATOR_GREATER_EQUAL | >= | Greater than or equal to | | OPERATOR_NOT_EQUAL | <> | Not equal to | | OPERATOR_RANGE | .. | Range (used in subranges, case) | | OPERATOR_PLUS | + | Addition | | OPERATOR_MINUS | - | Subtraction | | OPERATOR_MULTIPLY | * | Multiplication | | OPERATOR_DIVIDE | / | Division (real) | | OPERATOR_EQUAL | = | Equality (comparison) | | OPERATOR_LESS | < | Less than | | OPERATOR_GREATER | > | Greater than | | OPERATOR_POINTER | ^ | Pointer dereference / Pointer type | | OPERATOR_ADDRESSOF | @ | Memory address (@ operator) | | Delimiters | | | | DELIMITER_SEMICOLON | ; | Semicolon | | DELIMITER_COMMA | , | Comma | | DELIMITER_DOT | . | Dot (end of program, record access) | | DELIMITER_LPAREN | ( | Left parenthesis | | DELIMITER_RPAREN | ) | Right parenthesis | | DELIMITER_LBRACKET | [ | Left bracket (arrays) | | DELIMITER_RBRACKET | ] | Right bracket (arrays) | | DELIMITER_COLON | : | Colon (type declaration, labels) | | Literals | | | | STRING_LITERAL | 'Hello', 'A' | String literal (without quotes in value) | | BOOLEAN_LITERAL | true, false | Boolean value (case-insensitive) | | NUMBER_REAL | 1.23, -0.5, .5 | Number with decimal part | | NUMBER_INTEGER | 100, 0, -42 | Integer number | | Others | | | | KEYWORD | (See list below) | Language reserved word | | IDENTIFIER | myVariable, Counter, TMyObject | User-defined name | | EOF | | End Of File marker (internal) | | WHITESPACE | Can be used for external consumption | Unused. |

Keywords (KEYWORD) Currently Recognized (according to tokenizer.ts):

• program
• const
• var
• begin
• end
• if
• then
• else
• while
• do
• for
• to
• downto
• repeat
• until
• case
• of
• record
• array
• procedure
• function
• type
• integer
• real
• char
• string
• boolean
• uses
• label
• goto

(Note: true and false are specifically handled as BOOLEAN_LITERAL in your code, even though they might be in an initial keyword list).

Considerations

Please note that if you have writeln('Hello, World!'), the resulting token will be { type: "STRING_LITERAL", value: "Hello, World!" }. The single quotes (') will not be included directly in the value. This is intentional, so that you can render it on the screen as desired later on.

License

MIT