Sudachi WASM

WebAssembly build of Sudachi Japanese tokenizer for use in browsers and Node.js.

Quick Start

Build

# Build the WASM module
cargo build --target wasm32-unknown-unknown --release
# or, if you are in macOS, run `brew install llvm` and
# CC_wasm32_unknown_unknown=/opt/homebrew/opt/llvm/bin/clang cargo build --target wasm32-unknown-unknown --release

# Install wasm-bindgen-cli (if not already installed)
cargo install wasm-bindgen-cli

# Generate JavaScript bindings for web
wasm-bindgen ../target/wasm32-unknown-unknown/release/sudachi_wasm.wasm \
    --out-dir pkg \
    --target web

# Or for Node.js
wasm-bindgen ../target/wasm32-unknown-unknown/release/sudachi_wasm.wasm \
    --out-dir pkg-node \
    --target nodejs

JavaScript API

Functions

loadDictionary(xdicBytes: Uint8Array): number

Load a dictionary from .xdic file bytes. Returns a handle ID for use with other functions.

Parameters:

• xdicBytes: Uint8Array containing the dictionary file contents

Returns: Dictionary handle (number)

Example:

const response = await fetch("system.xdic");
const dictBytes = new Uint8Array(await response.arrayBuffer());
const handle = loadDictionary(dictBytes);

tokenize(handle: number, text: string, mode: number): TokenResult[]

Tokenize Japanese text using the loaded dictionary.

Parameters:

• handle: Dictionary handle from loadDictionary
• text: Japanese text to tokenize
• mode: Tokenization mode
  • 0: Mode A (short units - finest granularity)
  • 1: Mode B (middle units)
  • 2: Mode C (long units - coarsest granularity, default)

Returns: Array of token objects with properties:

• surface: Surface form of the token
• reading: Reading (pronunciation) of the token
• pos: Part of speech tag

Example:

const tokens = tokenize(handle, "選挙管理委員会", 0);
tokens.forEach((token) => {
    console.log(`${token.surface} (${token.reading}) - ${token.pos}`);
});

freeDictionary(handle: number): void

Free a dictionary handle and release its resources.

Parameters:

• handle: Dictionary handle to free

Example:

freeDictionary(handle);

Usage Examples

Browser (ES Modules)

See demo.html for a complete interactive example.

import init, {
    freeDictionary,
    loadDictionary,
    tokenize,
} from "./pkg/sudachi_wasm.js";

// Initialize the WASM module
await init();

// Load dictionary
const response = await fetch("system.xdic");
const dictBytes = new Uint8Array(await response.arrayBuffer());
const handle = loadDictionary(dictBytes);

// Tokenize with Mode C (long units)
const tokens = tokenize(handle, "東京スカイツリー", 2);

tokens.forEach((token) => {
    console.log(`${token.surface}\t${token.reading}\t${token.pos}`);
});

// Clean up
freeDictionary(handle);

Node.js

See example-node.js for a complete example.

const { loadDictionary, tokenize, freeDictionary } = require(
    "./pkg-node/sudachi_wasm.js",
);
const fs = require("fs");

// Load dictionary
const dictBytes = new Uint8Array(fs.readFileSync("system.xdic"));
const handle = loadDictionary(dictBytes);

// Tokenize
const tokens = tokenize(handle, "東京スカイツリー", 2);

tokens.forEach((token) => {
    console.log(`${token.surface}\t${token.reading}\t${token.pos}`);
});

// Clean up
freeDictionary(handle);

Running the Demo

To test the browser demo:

# Serve the directory with a local web server (needed for ES modules)
python3 -m http.server 8000

# Or use any other static file server
# npx serve .
# or
# npx http-server .

# Then open http://localhost:8000/demo.html in your browser

Tokenization Modes

Sudachi supports three tokenization modes with different granularities:

• Mode A (0): Short units (finest) - e.g., "選挙管理委員会" → ["選挙", "管理", "委員", "会"]
• Mode B (1): Middle units - e.g., "選挙管理委員会" → ["選挙", "管理", "委員会"]
• Mode C (2): Long units (coarsest) - e.g., "選挙管理委員会" → ["選挙管理委員会"]

Build Configuration

The project is configured to build for wasm32-unknown-unknown target:

• Uses wasm-bindgen for JavaScript interoperability
• On macOS, uses Homebrew LLVM's clang for WebAssembly support
• Configured in .cargo/config.toml

Files

• demo.html - Interactive browser demo
• example-node.js - Node.js usage example
• pkg/ - Web bindings (generated by wasm-bindgen)
• pkg-node/ - Node.js bindings (generated by wasm-bindgen)

Notes

• The dictionary file (.xdic) must be loaded before tokenization
• Multiple dictionaries can be loaded simultaneously with different handles
• Always call freeDictionary() when done to avoid memory leaks
• The WASM module is built for wasm32-unknown-unknown target for maximum compatibility
• For production use, consider compressing the dictionary file (it's quite large)

TypeScript Support

The generated bindings include TypeScript type definitions (.d.ts files), providing full type safety and IDE autocomplete support.