⬇️ Installation

To install the latest version of the library, simply run npm install @tsoding/tokenizer.js, or yarn add @tsoding/tokenizer.js, depending on your package manager.

📄 Usage

Using tokenizer.js is very easy. Here's an example of how to use it:

import tokenize from "@tsoding/tokenizer.js";

const str = "const test = 'hello';";
const tokens = tokenize(str);

The above code will return an array of tokens.

📝 Expansion

I plan to expand this library to do more things other than tokenizing variables. If you're willing to help, please contribute! Any help and/or feedback is welcome!