@pixagram/toxicity

Tiny offline toxicity classifier for the browser. Powers Pixagram's Toxic Comment Helper: a debounced, on-device hint while people type — for safety, playfulness and joy. No text ever leaves the device.

Under the hood: gravitee-io/bert-tiny-toxicity (BERT-tiny, 2 layers, binary toxic / not-toxic, trained on a 14-language toxicity dataset — works on English and French), re-quantized to int8 and run through onnxruntime-web (WASM).

Why re-quantize?

The upstream repo ships a model.quant.onnx of 29 MB — larger than the 17.6 MB fp32 model. Its recipe leaves the embedding table (89% of the parameters) in fp32. This package ships a correct quantization:

| | fp32 | upstream "quant" | this package | |---|---|---|---| | size | 17.60 MB | 29 MB | 4.49 MB | | verdict agreement vs fp32 | — | — | 100% | | mean ΔP(toxic) vs fp32 | — | — | 0.016 | | CPU latency vs fp32 | 1× | — | ~0.6× |

Recipe (reproducible via tools/quantize.py): weight-only int8 on Gather (the embedding table), dynamic int8 on constant-B MatMuls only (attention QKᵀ / probs×V stay fp32), pooler + classifier Gemm stay fp32. All resulting ops are supported by onnxruntime-web's WASM backend.

Install

npm i @pixagram/toxicity onnxruntime-web

onnxruntime-web is a peer dependency and is only import()ed the first time something is actually classified — apps that never trigger a check never download it.

Usage

import { getToxicityClassifier } from "@pixagram/toxicity";

const clf = getToxicityClassifier({
    // where onnxruntime-web's .wasm files are served from (same-origin!)
    wasmPaths: "/ort/",
    threshold: 0.7, // default — see below
});

const res = await clf.classify("t'es vraiment un gros débile");
// { toxic: true, score: 0.96, label: "toxic",
//   scores: { "not-toxic": 0.04, "toxic": 0.96 }, ms: 4.1 }

• load() is lazy, idempotent and shared between concurrent callers. Nothing heavy happens at import time.
• classify() results are LRU-cached by text, so a debounced check followed by a pre-broadcast re-check of the same text costs one inference.
• dispose() releases the WASM session (e.g. when the user disables the feature); the instance transparently re-loads on next use.

Default threshold: 0.7 (not 0.5)

BERT-tiny occasionally scores benign questions in the 0.5–0.65 band (measured: "Can you share which tool you used for this?" → 0.64), while genuinely toxic text lands ≥ 0.9 in both English and French. 0.7 separates these cleanly on the maintainer test suite. Tune per surface via { threshold }.

Bundler notes

Model + vocab resolve via new URL("../models/…", import.meta.url), which webpack 5 / Vite / Rollup turn into emitted same-origin assets. If your setup can't do that, pass modelUrl / vocabUrl explicitly. Copy onnxruntime-web's *.wasm into your static dir and point wasmPaths at it — keep everything same-origin so the "fully offline" guarantee holds.

License

Code © Pixagram SA. Bundled model weights are a quantization of gravitee-io/bert-tiny-toxicity, redistributed under OpenRAIL++ — see LICENSE.model and NOTICE (both shipped in this package). The OpenRAIL++ use restrictions apply to the weights.