tokenizer.js

A modern async text tokenizer, for the modern web. Demo

Features

• Pure Javascript (Uses No JQuery or Frameworks).
  • But can be used with any.
• Based on Reactive Streams (RxJS).
• Fully Async, get tokenization using REST calls.
• Written in TypeScript (types bundled).
• Battle tested on all modern browsers (and IE11).
• Fully customizable using CSS.
• Hooks and callbacks included.
• Support for IME locales like (Hiragana, Kenji etc.)

Articles

• Supercharge your search bar with visual tokens!

Usage

<div id="input" contenteditable></div>

import {Tokenizer} from 'tokenizer.js';

let el = document.getElementById('input');
let tokenizer = new Tokenizer(el /* Target element */, {
        onChange: () => {
            // return a Promise resolving to 
            // tokens.
        }
    }
);

Demos

• Simple
• Custom Tokens

Config Options

 let tokenizer = new Tokenizer(el, {
    initialInput: [], //  Initial token state, when the Tokenizer is initialized, this shows up as initial tokens.

    isFocused: false, //Flag to indicate whether the tokenizer is in focus when initialized.


    // A callback called whenever the text in the tokenizer is changed. 
    //This should return a Promise resolving to the new tokenization state.
    onChange: (inputText: string, caretPosition: number, isCaretOnSeparator?: boolean) => {
        return new Promise<DisplayToken[]>();
    },

    // Callback called on each keydown inside the tokenizer.
    // Return `false` to stop propagation.
    onKeyDown: (event: KeyboardEvent) => true,

    // Callback called when focus state of the tokenizer is changed. 
    // The param `isFocused` indicates the new focus state.
    onFocusChanged: (isFocused: boolean) => void,

    // Callback called when wrapping state of the tokenizer changes. 
    // The param `isWrapped` represents whether the tokenizer is wrapping.
    onWrap: (isWrapped: boolean) => void,

    // Callback called when the caret position is changed. 
    // For eg. using arrow keys, backspace, click etc.
    onCaretPositionChanged: (inputText: string, caretPosiotion: number, isCaretOnSeparator: boolean) => void,

    // Callback called when a token is hovered over.
    onMouseOverToken: (tokenIndex: number, e: MouseEvent) => void,

    // Callback called when the user stop hovering over any tokens.
    onMouseOutTokenizer: () => void
 })

Public Methods

let tokenizer = new Tokenizer(el, config);

// Clear the tokenizer text.
tokenizer.clear();

// Blur the tokenizer (loose focus).
tokenizer.blur();

// Get coords of the token having the caret.
tokenizer.getCaretTokenPosition(); // {x: 50, y: 80} For eg.

// Gets the position of the token at [idx] index.
tokenizer.getTokenPosition(idx: number);

// Replace the text of the tokenizer with new text.
// This is followed by a automatic reTokenization.
tokenizer.updateText(text: string);

// Update the tokenizer display with a new set of `tokens`,
// puts the caret after `caretTokenIdx` token,
// if `reTokenization=true` reTokenization will follow.
tokenizer.updateDisplay(tokens: DisplayToken[], caretTokenIdx: number, reTokenization: bool);

// Gets the text equivalent of the tokenizer.
tokenizer.getInnerText() // 'Some string user entered'

// Whether the tokenizer has focus.
tokenizer.hasFocus() // true

// Gets the number of tokens.
tokenizer.getNumTokens() // 4

// Sets the caret at the start of the `idx` index token.
tokenizer.selectNodeByIdx(idx: number);

DisplayToken

This is a structure which represents a token. Its an object with these properties:

Prop name | type | description --- | --- | --- value | string | The text value of the token. className | string | The css class name to be applied to the token, the consumer can apply styles under this css class. isIncomplete | bool | Boolean flag to indicate whether the token is complete. isExtensible | bool | Flag to indicate whether the token is extensible, if true, cursor is not moved out of it on completion.