pdq-wasm

v0.3.9

Published

4 months ago

WebAssembly bindings for Meta's PDQ perceptual image hashing algorithm

0High
0Medium
0Low

svnbjrn

pdq perceptual-hashing image-hashing wasm webassembly image-similarity content-moderation threat-exchange

PDQ WebAssembly

WebAssembly bindings for Meta's PDQ (Perceptual Hashing) algorithm, enabling fast perceptual image hashing in both browser and Node.js environments.

About PDQ

PDQ is a perceptual hashing algorithm developed by Meta (Facebook) for matching images that look similar to the human eye. It generates a 256-bit hash from an image that can be compared with other hashes using Hamming distance to determine similarity.

Key features:

Fast and efficient perceptual hashing
Robust to common image transformations (resize, crop, compress, etc.)
256-bit compact hash representation
Hamming distance-based similarity matching
WebAssembly performance (26KB binary, ~13KB JS wrapper)

Installation

npm install pdq-wasm

The package is published on npm and includes:

WebAssembly binaries (~26KB)
TypeScript definitions
Dual module system (CommonJS + ES Modules)
Browser utilities for duplicate detection
Browser and Node.js examples
Complete documentation

Quick Start

Node.js

const { PDQ } = require('pdq-wasm');

async function main() {
  // Initialize the WASM module (required once)
  await PDQ.init();

  // Create image data (grayscale or RGB)
  const imageData = {
    data: new Uint8Array(100).fill(128), // 10x10 gray image
    width: 10,
    height: 10,
    channels: 1, // 1 for grayscale, 3 for RGB
  };

  // Generate PDQ hash
  const result = PDQ.hash(imageData);
  console.log('Hash:', PDQ.toHex(result.hash));
  console.log('Quality:', result.quality);

  // Compare two images
  const hash1 = PDQ.hash(imageData);
  const hash2 = PDQ.hash(anotherImageData);

  const distance = PDQ.hammingDistance(hash1.hash, hash2.hash);
  console.log('Hamming distance:', distance);

  const similarity = PDQ.similarity(hash1.hash, hash2.hash);
  console.log('Similarity:', similarity.toFixed(2) + '%');

  // Check if images are similar
  const areSimilar = PDQ.areSimilar(hash1.hash, hash2.hash, 31);
  console.log('Similar?', areSimilar);
}

main();

TypeScript

import { PDQ, ImageData, PDQHashResult } from 'pdq-wasm';

async function hashImage(imageData: ImageData): Promise<PDQHashResult> {
  await PDQ.init();
  return PDQ.hash(imageData);
}

// Example with type safety
const imageData: ImageData = {
  data: new Uint8Array(300), // 10x10 RGB image
  width: 10,
  height: 10,
  channels: 3,
};

const result = await hashImage(imageData);
const hexHash: string = PDQ.toHex(result.hash);

Browser

The easiest way to use PDQ-WASM in the browser:

<script type="module">
  import { PDQ } from 'https://unpkg.com/[email protected]/dist/esm/index.js';

  async function main() {
    // Initialize - automatically loads WASM from CDN!
    await PDQ.init();

    // Now use PDQ as normal
    const canvas = document.getElementById('myCanvas');
    if (!canvas) {
      throw new Error('Canvas element with id="myCanvas" not found');
    }
    const ctx = canvas.getContext('2d');
    const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);

    // Extract RGB data (skip alpha channel)
    const rgbData = new Uint8Array(canvas.width * canvas.height * 3);
    let rgbIndex = 0;
    for (let i = 0; i < imageData.data.length; i += 4) {
      rgbData[rgbIndex++] = imageData.data[i];     // R
      rgbData[rgbIndex++] = imageData.data[i + 1]; // G
      rgbData[rgbIndex++] = imageData.data[i + 2]; // B
    }

    const result = PDQ.hash({
      data: rgbData,
      width: canvas.width,
      height: canvas.height,
      channels: 3
    });

    console.log('Hash:', PDQ.toHex(result.hash));
    console.log('Quality:', result.quality);
  }

  main();
</script>

Key Points:

🎉 Zero configuration required! Just call PDQ.init() and it automatically loads WASM from CDN
📦 No bundler setup needed for quick prototyping
🌐 Production ready with unpkg.com CDN (see self-hosting below for maximum control)

Using with npm + bundlers

If you've installed via npm, you can use it in your bundled application:

import { PDQ } from 'pdq-wasm';

// Option 1: Use CDN (easiest, zero config)
await PDQ.init();

// Option 2: Self-host for production (recommended)
await PDQ.init({
  wasmUrl: '/assets/pdq.wasm'
});

Self-hosting WASM files (Recommended for production)

While the CDN approach is convenient, for production we recommend self-hosting for better reliability and control:

Step 1: Copy WASM files to your static assets

# Copy from node_modules after npm install
cp node_modules/pdq-wasm/wasm/* public/assets/

Step 2: Configure bundler (optional - automate the copy)

npm install -D vite-plugin-static-copy

// vite.config.js
import { defineConfig } from 'vite';
import { viteStaticCopy } from 'vite-plugin-static-copy';

export default defineConfig({
  plugins: [
    viteStaticCopy({
      targets: [{
        src: 'node_modules/pdq-wasm/wasm/*',
        dest: 'assets'
      }]
    })
  ]
});

npm install -D copy-webpack-plugin

// webpack.config.js
const CopyPlugin = require('copy-webpack-plugin');

module.exports = {
  plugins: [
    new CopyPlugin({
      patterns: [{
        from: 'node_modules/pdq-wasm/wasm',
        to: 'assets'
      }]
    })
  ]
};

Step 3: Use your self-hosted files

await PDQ.init({ wasmUrl: '/assets/pdq.wasm' });

Available CDN Options

If you prefer to use a CDN:

unpkg (default): https://unpkg.com/[email protected]/wasm/pdq.wasm
jsDelivr: https://cdn.jsdelivr.net/npm/[email protected]/wasm/pdq.wasm
unpkg (latest): https://unpkg.com/[email protected]/wasm/pdq.wasm

You can also specify a custom CDN:

await PDQ.init({
  wasmUrl: 'https://your-cdn.com/pdq.wasm'
});

Module System Support

Package exports both CommonJS and ES Modules:

ES Module: pdq-wasm/dist/esm/index.js (for browsers and modern Node.js)
CommonJS: pdq-wasm/dist/index.js (for traditional Node.js)

The package.json exports field automatically selects the right version based on your environment.

Web Workers

✅ Full Web Worker Support - Hash images in background threads for better performance!

Quick Start: Workers

// worker.js
importScripts('https://unpkg.com/[email protected]/dist/browser.js');

// Initialize PDQ in the worker
await PDQ.initWorker({
  wasmUrl: 'https://unpkg.com/[email protected]/wasm/pdq.wasm'
});

// Handle messages from main thread
self.onmessage = async (event) => {
  const { file } = event.data;

  // ✅ Use generateHashFromBlob in workers
  const hash = await generateHashFromBlob(file);

  self.postMessage({ hash });
};

// main.js
const worker = new Worker('worker.js');

worker.onmessage = (event) => {
  console.log('Hash:', event.data.hash);
};

// Send file to worker
const file = document.querySelector('input[type="file"]').files[0];
worker.postMessage({ file });

Environment Detection

Not sure which API to use? The getEnvironment() helper detects your runtime and recommends the right API:

import { getEnvironment, generateHashFromBlob, generateHashFromDataUrl } from 'pdq-wasm/browser';

const env = getEnvironment();
console.log(`Running in: ${env.type}`); // 'browser', 'worker', 'node', or 'unknown'
console.log(`Recommended API: ${env.recommendedAPI}`);

// Automatically choose the right API
let hash;
if (env.supportsBlob) {
  // ✅ RECOMMENDED - Works in browsers AND workers
  hash = await generateHashFromBlob(file);
} else if (env.supportsDataUrl) {
  // ⚠️ Only works in browser main thread
  hash = await generateHashFromDataUrl(dataUrl);
}

API Comparison: Browser vs Workers

| Feature | generateHashFromBlob() | generateHashFromDataUrl() | |---------|-------------------------|---------------------------| | Browser Main Thread | ✅ Yes | ✅ Yes | | Web Workers | ✅ Yes | ❌ No (throws error) | | Uses | createImageBitmap + OffscreenCanvas | Image + Canvas (DOM) | | Input | Blob, File | Data URL, Blob URL | | Browser Support | Chrome 69+, Firefox 105+, Safari 16.4+ | All browsers | | Recommendation | ✅ Use this | Legacy fallback only |

Migration Guide: Fix Worker Errors

If you're getting errors like:

Error: generateHashFromDataUrl() requires browser main thread (needs DOM APIs).
For Web Workers, use generateHashFromBlob() instead...

Migration is simple:

// ❌ BEFORE (doesn't work in workers)
const dataUrl = URL.createObjectURL(file);
const hash = await generateHashFromDataUrl(dataUrl);

// ✅ AFTER (works everywhere)
const hash = await generateHashFromBlob(file); // file is Blob/File

Complete Worker Example

📖 See: examples/worker/README.md for a complete working example with:

ES module and classic worker support
Worker pool management
Progress tracking
Error handling

Browser Utilities

PDQ-WASM includes specialized browser utilities for common web application tasks. For complete documentation on browser utilities including duplicate detection, hash checking, and progress tracking:

📖 See: docs/BROWSER.md

Quick overview of available utilities:

import {
  getEnvironment,         // ⭐ NEW: Detect runtime environment and get API recommendations
  generateHashFromBlob,   // ✅ RECOMMENDED: Works in browsers AND workers
  generateHashFromDataUrl, // Legacy: Browser main thread only
  createHashChecker,      // Hash existence checking with caching
  hammingDistance,        // Hex hash comparison
  detectDuplicatesByHash  // Batch duplicate detection
} from 'pdq-wasm/browser';

Key Features:

getEnvironment: ⭐ NEW - Detect runtime environment and get API recommendations
generateHashFromBlob: ✅ RECOMMENDED - Worker-compatible image hashing (Blob/File input)
generateHashFromDataUrl: Canvas-based image hashing (browser main thread only)
createHashChecker: Chainable hash lookup with .cached() and .ignoreInvalid()
detectDuplicatesByHash: Batch duplicate detection with progress callbacks
hammingDistance: Convenient hex string comparison

Use Cases:

File upload deduplication
Photo gallery management
Content moderation
Offline-first apps with cached lookups
Real-time duplicate detection with progress UI

Security Note: As a best practice, only load WASM modules from trusted sources. The wasmUrl option dynamically loads and executes WebAssembly code in your application. Always verify that the URL points to a trusted CDN or your own infrastructure. For production applications, consider:

Using Subresource Integrity (SRI) hashes when supported
Hosting WASM files on your own domain with proper CORS headers
Implementing Content Security Policy (CSP) to restrict allowed sources

Examples

We provide comprehensive examples for both browser and Node.js environments:

Browser Example

Interactive web application demonstrating PDQ hashing in the browser:

cd examples/browser
npx serve .  # Or any static file server
# Open http://localhost:3000 in your browser

Features:

Upload and hash images
Compare two images visually
Real-time similarity scores
Uses Canvas API for image processing

Node.js Examples

Basic Usage

cd examples/nodejs
node basic-usage.js

Demonstrates:

Initializing PDQ
Hashing synthetic images
Hamming distance calculation
Format conversion

Image Comparison

npm install sharp  # Required for image loading
node image-comparison.js image1.jpg image2.jpg image3.jpg

Demonstrates:

Loading real image files
Batch comparison of multiple images
Finding similar images

More Examples

See the examples/ directory for:

Complete source code for all examples
API usage patterns
Common use cases (duplicate detection, content moderation)
Performance tips

Full documentation: examples/README.md

PostgreSQL Integration

For storing and querying PDQ hashes in PostgreSQL databases:

Schema design for hash storage
Efficient similarity queries using SQL
Using < and > operators on distances
Batch operations and performance optimization

Full guide: docs/POSTGRESQL.md

API Reference

Initialization

`PDQ.init(options?): Promise<void>`

Initialize the WASM module. Must be called before using any other PDQ methods.

await PDQ.init();

Hashing

`PDQ.hash(imageData): PDQHashResult`

Generate a PDQ hash from image data.

Parameters:

imageData: Object with properties:
- data: Uint8Array - Raw pixel data
- width: number - Image width in pixels
- height: number - Image height in pixels
- channels: 1 | 3 - Number of channels (1 for grayscale, 3 for RGB)

Returns: PDQHashResult

hash: Uint8Array - 32-byte (256-bit) PDQ hash
quality: number - Quality metric of the hash

Example:

const result = PDQ.hash({
  data: new Uint8Array(300),
  width: 10,
  height: 10,
  channels: 3,
});

Comparison

`PDQ.hammingDistance(hash1, hash2): number`

Calculate Hamming distance between two PDQ hashes.

Parameters:

hash1, hash2: Uint8Array - 32-byte PDQ hashes

Returns: number - Distance from 0 (identical) to 256 (completely different)

Example:

const distance = PDQ.hammingDistance(hash1, hash2);
console.log(`Distance: ${distance}/256`);

`PDQ.areSimilar(hash1, hash2, threshold?): boolean`

Check if two hashes are similar based on a threshold.

Parameters:

hash1, hash2: Uint8Array - 32-byte PDQ hashes
threshold: number - Maximum Hamming distance to consider similar (default: 31)

Returns: boolean - True if hashes are similar

Example:

if (PDQ.areSimilar(hash1, hash2, 20)) {
  console.log('Images are similar!');
}

`PDQ.similarity(hash1, hash2): number`

Get similarity percentage between two hashes.

Parameters:

hash1, hash2: Uint8Array - 32-byte PDQ hashes

Returns: number - Similarity percentage (0-100)

Example:

const similarity = PDQ.similarity(hash1, hash2);
console.log(`${similarity.toFixed(2)}% similar`);

`PDQ.orderBySimilarity(referenceHash, hashes, includeIndex?): SimilarityMatch[]`

Order an array of hashes by similarity to a reference hash. Returns hashes sorted from most similar to least similar.

Parameters:

referenceHash: Uint8Array - The reference hash to compare against
hashes: Uint8Array[] - Array of hashes to order
includeIndex: boolean - Whether to include original array index (default: false)

Returns: SimilarityMatch[] - Array of objects containing:

hash: Uint8Array - The hash
distance: number - Hamming distance from reference (0-256)
similarity: number - Similarity percentage (0-100)
index?: number - Original array index (if includeIndex is true)

Example:

const referenceHash = PDQ.hash(referenceImage).hash;
const candidateHashes = [hash1, hash2, hash3, hash4];

// Order by similarity
const ordered = PDQ.orderBySimilarity(referenceHash, candidateHashes);

// Most similar first
console.log('Most similar:', PDQ.toHex(ordered[0].hash));
console.log('Distance:', ordered[0].distance);
console.log('Similarity:', ordered[0].similarity.toFixed(2) + '%');

// With original indices
const withIndices = PDQ.orderBySimilarity(referenceHash, candidateHashes, true);
console.log('Original index:', withIndices[0].index);

Use Cases:

Find top-N most similar images
Rank search results by similarity
Deduplicate image collections
Build image recommendation systems

Logging and Error Handling