lindera-wasm

WebAssembly of Lindera

Demo Application

• https://lindera.github.io/lindera/demo/

npm

• https://www.npmjs.com/package/lindera-wasm — Lindera WASM

Dictionary

Pre-built dictionaries are available from GitHub Releases. Download a dictionary archive (e.g. lindera-ipadic-*.zip) and load it at runtime using the OPFS API. See the example application for a working demo of downloading and loading dictionaries.

Usage

Web Usage

The lindera-wasm package is built with wasm-pack's web target. Call the default-exported async init function (__wbg_init) once before using any API. Dictionaries are loaded at runtime from a local path or downloaded from GitHub Releases using the OPFS API.

import __wbg_init, { TokenizerBuilder, loadDictionaryFromBytes } from 'lindera-wasm';
import { downloadDictionary, loadDictionaryFiles } from 'lindera-wasm/opfs';

async function main() {
    await __wbg_init();

    // Download dictionary from GitHub Releases (first time only)
    await downloadDictionary("https://github.com/lindera/lindera/releases/download/v3.0.0/lindera-ipadic-3.0.0.zip", "ipadic");

    // Load dictionary from OPFS
    const files = await loadDictionaryFiles("ipadic");
    const dict = loadDictionaryFromBytes(
        files.metadata, files.dictTrie, files.dictValsIdx, files.dictVals,
        files.dictWordsIdx, files.dictWords, files.matrixMtx, files.charDef, files.unk
    );

    const builder = new TokenizerBuilder();
    builder.setDictionaryInstance(dict);
    builder.setMode("normal");
    const tokenizer = builder.build();

    const tokens = tokenizer.tokenize("すもももももももものうち");
    tokens.forEach(token => {
        console.log(`${token.surface}: ${token.details.join(", ")}`);
    });
}

main();

Bundler Usage (Webpack, Rollup, etc.)

Bundlers use the same lindera-wasm package. Modern bundlers (Vite, Webpack 5 with the asyncWebAssembly experiment) can consume the web-target build directly; call await __wbg_init() before using any API, exactly as in the web usage above. The dictionary loading approach is also the same.

Token Properties

Each token object has the following properties:

| Property | Type | Description | | -------- | ---- | ----------- | | surface | string | Surface form of the token | | byteStart | number | Start byte position in the original text | | byteEnd | number | End byte position in the original text | | position | number | Position index of the token | | wordId | number | Word ID in the dictionary | | isUnknown | boolean | Whether the token is an unknown word | | details | string[] | Morphological details array |

Tokens are plain JavaScript objects, not class instances, so read details directly (token.details[0]) and pass a token to JSON.stringify as-is.

For Vite Projects

You should exclude this package in the optimizeDeps:

// vite.config.js
import { defineConfig } from 'vite'

export default defineConfig({
  optimizeDeps: {
    exclude: [
      "lindera-wasm"
    ]
  },
})

For Browser Extension Development

Set the cors config in vite.config.js:

// vite.config.js
import { defineConfig } from 'vite'

export default defineConfig({
  server: {
    cors: {
      origin: [
        /chrome-extension:\/\//,
      ],
    },
  },
})

And set the content_security_policy to contain wasm-unsafe-eval in manifest.json:

"content_security_policy": {
  "extension_pages": "script-src 'self' 'wasm-unsafe-eval';"
}

Development

Install project dependencies

• wasm-pack : https://rustwasm.github.io/wasm-pack/installer/

Setup repository

# Clone the Lindera project repository
% git clone [email protected]:lindera/lindera.git
% cd lindera

Build project

% make wasm-build

Run tests

% make wasm-test

Build example web application

% cd example && npm install && npm run build && cp index.html dist/index.html

Run example web application

% cd example && npm run start