fsm-tokenizer

finite state machine tokenizer

Contents

• What is this?
• When should I use this?
• Install
• Use
• API
• Types
• Contribute

What is this?

This package is a tokenization utility. The underlying tokenizer is implemented as a state machine that produces tokens with positional information, as well as any other fields attached when the token was opened. Tokens are turned into events, with the tokenizer calling any defined resolvers in the process.

When should I use this?

This package can be used to tokenize a file, Uint8Array, string, or a list containing these values, like command-line arguments. Developers familiar with micromark will find that tokenizers and constructs (objects used to define how to parse text) in this package work similarly.

Install

This package is ESM only.

In Node.js with yarn:

yarn add @flex-development/fsm-tokenizer

In Deno with esm.sh:

import { tokenize } from 'https://esm.sh/@flex-development/fsm-tokenizer'

In browsers with esm.sh:

<script type="module">
  import { tokenize } from 'https://esm.sh/@flex-development/fsm-tokenizer'
</script>

Use

TODO: use

API

TODO: api

Please refer to the source code for documentation.

Types

This package is fully typed with TypeScript.

Contribute

See CONTRIBUTING.md.

This project has a code of conduct. By interacting with this repository, organization, or community you agree to abide by its terms.