Synchronous Tokenizer

A simple synchronous string tokenizer using Regex.

Installation

yarn add tokenize-sync

Usage

The package ships a single tokenize function that takes an (string) input and a (Object) ruleMap that maps (string) token names to Regexes.

import tokenize from 'tokenize-sync'

const ruleMap = {
  identifier: /^[a-z-]+$/i,
  number: /^\d+$/,
  whitespace: /^\s+$/
}

const input = 'test 12  foobar3'

const tokens = tokenize(input, ruleMap)

tokens === [{
  type: 'identifier',
  value: 'test',
  start: 0,
  end: 4
}, {
  type: 'whitespace',
  value: ' ',
  start: 4,
  end: 5
}, {
  type: 'number',
  value: '12',
  start: 5,
  end: 7
}, {
  type: 'whitespace',
  value: '  ',
  start: 7,
  end: 9
}, {
  type: 'identifier',
  value: 'foobar',
  start: 9,
  end: 15
},  {
  type: 'number',
  value: '3',
  start: 15,
  end: 16
}]

License

tokenize-sync is licensed under the MIT License. Documentation is licensed under Creative Common License. Created with ♥ by @rofrischmann.