Token Iterator

Iterate over regex-matched tokens in a string with next, peek, and hasNext.

Installation

npm install token-iterator

Usage

import { createTokenIterator } from 'token-iterator';

// Basic usage
const iter = createTokenIterator(/\w+/, 'hello world');
while (iter.hasNext()) {
  console.log(iter.next()); // 'hello', 'world'
}

// Skip a capture group (e.g. whitespace)
const iter2 = createTokenIterator(/(\s+)|\w+/g, 'foo  bar', { skipGroup: 1 });
iter2.peek();        // 'foo'  – does not consume
iter2.next('foo');   // 'foo'  – asserts and consumes
iter2.next(/\w+/);  // 'bar'  – regex assertion

API

createTokenIterator(pattern, input, options?)

| Parameter | Type | Description | |---|---|---| | pattern | RegExp | Token pattern (the g flag is added automatically if missing) | | input | string | Input string to tokenize | | options.skipGroup | number | Capture group index whose matches are silently skipped |

Returns a TokenIterator:

| Method | Returns | Description | |---|---|---| | next(expected?) | string | Returns the next token. Throws if exhausted or if expected (string or RegExp) doesn't match | | hasNext() | boolean | true if tokens remain | | peek() | string \| null | Next token without consuming it, or null if exhausted |

License

MIT — see LICENSE.