@neutrondev/kalla-oi-wasm-compress

A high-performance lossless compression library built with Rust and WebAssembly. Features adaptive algorithm selection that automatically chooses the optimal compression strategy based on file type and content entropy analysis.

Table of Contents

• Features
• Installation
• Bundler Configuration
  • Next.js (Turbopack)
  • Next.js (Webpack)
  • Vite
  • Webpack 5
  • Troubleshooting
• Quick Start
• API Reference
  • Initialization
  • Compression
  • Decompression
  • File Info
  • Metrics
  • Synchronous API
• Compression Algorithms
• Adaptive Strategy
• Binary Format Specification
• Web Worker Support
• Error Handling
• Performance
• Browser & Node.js Compatibility
• License

Features

• Adaptive Algorithm Selection - Automatically selects the best compression algorithm based on file type detection and entropy analysis
• Multiple Algorithms - Supports Brotli (best ratio), Deflate (balanced), and Store (passthrough for pre-compressed files)
• Smart File Detection - Detects file types via magic bytes and extension hints to skip compression on already-compressed formats
• Shannon Entropy Analysis - Calculates data entropy to determine compressibility for unknown file types
• Chunked Processing - Memory-efficient handling of large files up to 100MB with configurable chunk sizes
• Web Worker Support - Non-blocking compression for large files in browser environments
• Cross-Platform - Works seamlessly in both browser and Node.js environments
• Type-Safe - Full TypeScript support with comprehensive type definitions
• Zero Dependencies - Pure Rust/WASM implementation with no runtime dependencies

Installation

npm install @neutrondev/kalla-oi-wasm-compress

yarn add @neutrondev/kalla-oi-wasm-compress

pnpm add @neutrondev/kalla-oi-wasm-compress

bun add @neutrondev/kalla-oi-wasm-compress

Bundler Configuration

WASM files require special configuration in modern bundlers. Follow the guide for your setup:

Next.js (App Router with Turbopack)

Next.js with Turbopack requires copying WASM files to the public folder and using dynamic imports:

1. Copy WASM files to public folder:

# Add to your package.json scripts
"postinstall": "cp -r node_modules/@neutrondev/kalla-oi-wasm-compress/src/wasm public/wasm"

Or manually copy node_modules/@neutrondev/kalla-oi-wasm-compress/src/wasm/ to public/wasm/.

2. Create a client-side hook:

// hooks/use-compression.ts
'use client';

import { useCallback, useEffect, useRef, useState } from 'react';

export function useCompression() {
  const [isReady, setIsReady] = useState(false);
  const [error, setError] = useState<Error | null>(null);
  const moduleRef = useRef<typeof import('@neutrondev/kalla-oi-wasm-compress') | null>(null);

  useEffect(() => {
    let mounted = true;

    async function init() {
      try {
        // Dynamic import to avoid SSR issues
        const mod = await import('@neutrondev/kalla-oi-wasm-compress');

        // Initialize with explicit WASM path from public folder
        await mod.initWasm('/wasm/compression_wasm_bg.wasm');

        if (mounted) {
          moduleRef.current = mod;
          setIsReady(true);
        }
      } catch (err) {
        if (mounted) {
          setError(err instanceof Error ? err : new Error(String(err)));
        }
      }
    }

    init();
    return () => { mounted = false; };
  }, []);

  const compress = useCallback(async (data: Uint8Array, options?: Parameters<typeof import('@neutrondev/kalla-oi-wasm-compress').compress>[1]) => {
    if (!moduleRef.current) throw new Error('Compression not initialized');
    return moduleRef.current.compress(data, options);
  }, []);

  const decompress = useCallback(async (data: Uint8Array) => {
    if (!moduleRef.current) throw new Error('Compression not initialized');
    return moduleRef.current.decompress(data);
  }, []);

  return { isReady, error, compress, decompress };
}

3. Use in components:

'use client';

import { useCompression } from '@/hooks/use-compression';

export function MyComponent() {
  const { isReady, error, compress, decompress } = useCompression();

  if (error) return <div>Error: {error.message}</div>;
  if (!isReady) return <div>Loading compression...</div>;

  const handleCompress = async (file: File) => {
    const buffer = await file.arrayBuffer();
    const data = new Uint8Array(buffer);
    const result = await compress(data, { filename: file.name });
    console.log(`Compressed: ${result.savings * 100}% savings`);
  };

  return <button onClick={() => handleCompress(file)}>Compress</button>;
}

Next.js (Webpack - Pages Router or App Router without Turbopack)

1. Configure next.config.js:

// next.config.js
/** @type {import('next').NextConfig} */
const nextConfig = {
  webpack: (config, { isServer }) => {
    // Handle WASM files
    config.experiments = {
      ...config.experiments,
      asyncWebAssembly: true,
    };

    // Fix for WASM imports
    config.module.rules.push({
      test: /\.wasm$/,
      type: 'asset/resource',
    });

    return config;
  },
};

module.exports = nextConfig;

2. Use dynamic import with ssr disabled:

// components/compression-provider.tsx
'use client';

import dynamic from 'next/dynamic';

const CompressionComponent = dynamic(
  () => import('./compression-component'),
  { ssr: false }
);

export default CompressionComponent;

Vite

Vite has built-in WASM support but may need configuration for optimal loading:

1. Install vite-plugin-wasm (optional, for better compatibility):

npm install vite-plugin-wasm vite-plugin-top-level-await

2. Configure vite.config.ts:

// vite.config.ts
import { defineConfig } from 'vite';
import wasm from 'vite-plugin-wasm';
import topLevelAwait from 'vite-plugin-top-level-await';

export default defineConfig({
  plugins: [
    wasm(),
    topLevelAwait(),
  ],
  optimizeDeps: {
    exclude: ['@neutrondev/kalla-oi-wasm-compress'],
  },
});

3. Use in your app:

import { initWasm, compress, decompress } from '@neutrondev/kalla-oi-wasm-compress';

// Initialize once
await initWasm();

// Use compression
const result = await compress(data);

Webpack 5

1. Configure webpack.config.js:

// webpack.config.js
module.exports = {
  experiments: {
    asyncWebAssembly: true,
  },
  module: {
    rules: [
      {
        test: /\.wasm$/,
        type: 'asset/resource',
      },
    ],
  },
};

Troubleshooting

Error: "Module not found: Can't resolve './wasm/compression_wasm_bg.wasm'"

This occurs when the bundler can't resolve the WASM file path. Solutions:

1. Copy WASM to public folder and use explicit URL:

   await initWasm('/wasm/compression_wasm_bg.wasm');

2. Use dynamic imports to load the module only on the client side.

3. Configure your bundler to handle .wasm files (see configurations above).

Error: "WebAssembly is not defined" (SSR)

WASM only works in browser environments. Solutions:

1. Use dynamic imports with ssr: false (Next.js)
2. Check for browser environment before initializing:

   if (typeof window !== 'undefined') {
     await initWasm();
   }

Error: "CompileError: WebAssembly.instantiate()"

The WASM file may be corrupted or incorrectly served. Ensure:

1. WASM file is served with Content-Type: application/wasm
2. File is not being processed/transformed by the bundler incorrectly

Quick Start

import { initWasm, compress, decompress } from '@neutrondev/kalla-oi-wasm-compress';

// Initialize WASM module (required once before any operations)
await initWasm();

// Compress text data
const text = 'Hello, World! '.repeat(1000);
const data = new TextEncoder().encode(text);

const compressed = await compress(data, { filename: 'greeting.txt' });

console.log(`Original size: ${compressed.originalSize} bytes`);
console.log(`Compressed size: ${compressed.compressedSize} bytes`);
console.log(`Algorithm used: ${compressed.algo}`);
console.log(`Space savings: ${(compressed.savings * 100).toFixed(1)}%`);

// Decompress back to original
const decompressed = await decompress(compressed.data);
const originalText = new TextDecoder().decode(decompressed.data);

console.log(originalText); // "Hello, World! Hello, World! ..."

API Reference

Initialization

initWasm(wasmUrl?: string | URL): Promise<void>

Initializes the WebAssembly module. This function must be called once before using any compression functions. It is idempotent - subsequent calls return immediately.

import { initWasm } from '@neutrondev/kalla-oi-wasm-compress';

// Default initialization (auto-detects WASM location)
await initWasm();

// Custom WASM URL (useful for CDN deployment)
await initWasm('https://cdn.example.com/compression_wasm_bg.wasm');

isWasmInitialized(): boolean

Checks if the WASM module has been initialized.

import { isWasmInitialized } from '@neutrondev/kalla-oi-wasm-compress';

if (!isWasmInitialized()) {
  await initWasm();
}

Compression

compress(data: Uint8Array, options?: CompressOptions): Promise<CompressResult>

Compresses data using adaptive algorithm selection.

Parameters:

| Parameter | Type | Description | |-----------|------|-------------| | data | Uint8Array | Raw bytes to compress (max 100MB) | | options | CompressOptions | Optional compression settings |

CompressOptions:

| Option | Type | Default | Description | |--------|------|---------|-------------| | filename | string | undefined | Filename hint for file type detection (e.g., "data.csv") | | level | CompressionLevel | Default | Compression level: Fast, Default, or Best | | chunkSize | number | 4194304 (4MB) | Chunk size for large files (64KB - 16MB) | | forceAlgo | Algorithm | undefined | Force a specific algorithm instead of auto-detection |

CompressResult:

| Field | Type | Description | |-------|------|-------------| | data | Uint8Array | Compressed data with header | | algo | Algorithm | Algorithm that was used (Brotli, Deflate, or Store) | | originalSize | number | Original data size in bytes | | compressedSize | number | Compressed data size in bytes | | ratio | number | Compression ratio (compressed / original) | | savings | number | Space savings as decimal (0.0 - 1.0) |

Examples:

import { compress, CompressionLevel, Algorithm } from '@neutrondev/kalla-oi-wasm-compress';

// Basic compression with filename hint
const csvData = new TextEncoder().encode('name,age,city\nAlice,30,NYC\n'.repeat(1000));
const result = await compress(csvData, { filename: 'users.csv' });

// Maximum compression for archival
const archiveResult = await compress(data, {
  filename: 'important.json',
  level: CompressionLevel.Best
});

// Fast compression for real-time use
const fastResult = await compress(data, {
  level: CompressionLevel.Fast
});

// Force specific algorithm
const brotliResult = await compress(data, {
  forceAlgo: Algorithm.Brotli
});

// Custom chunk size for memory-constrained environments
const chunkedResult = await compress(largeData, {
  chunkSize: 1024 * 1024 // 1MB chunks
});

Decompression

decompress(data: Uint8Array): Promise<DecompressResult>

Decompresses data that was compressed with this library.

DecompressResult:

| Field | Type | Description | |-------|------|-------------| | data | Uint8Array | Decompressed original data | | algo | Algorithm | Algorithm that was used | | originalSize | number | Size after decompression | | compressedSize | number | Size before decompression |

import { decompress } from '@neutrondev/kalla-oi-wasm-compress';

const decompressed = await decompress(compressedData);
const text = new TextDecoder().decode(decompressed.data);

File Info

getFileInfo(data: Uint8Array): Promise<FileInfo>

Extracts metadata from compressed data without performing full decompression. Useful for displaying file information before decompression.

FileInfo:

| Field | Type | Description | |-------|------|-------------| | version | number | Format version (currently 1) | | algo | Algorithm | Compression algorithm used | | originalSize | number | Original uncompressed size | | isChunked | boolean | Whether data was chunked | | chunkSize | number | Chunk size if chunked, 0 otherwise |

import { getFileInfo } from '@neutrondev/kalla-oi-wasm-compress';

const info = await getFileInfo(compressedData);

console.log(`Format version: ${info.version}`);
console.log(`Algorithm: ${info.algo}`);
console.log(`Original size: ${info.originalSize} bytes`);
console.log(`Chunked: ${info.isChunked ? `Yes (${info.chunkSize} bytes)` : 'No'}`);

Metrics

getMetrics(): CompressionMetrics | null

Returns performance metrics from the last compression/decompression operation.

CompressionMetrics:

| Field | Type | Description | |-------|------|-------------| | inputBytes | number | Input data size | | outputBytes | number | Output data size | | timeMs | number | Processing time in milliseconds | | throughputMbps | number | Throughput in MB/s |

import { compress, getMetrics } from '@neutrondev/kalla-oi-wasm-compress';

await compress(largeData);
const metrics = getMetrics();

if (metrics) {
  console.log(`Processed ${metrics.inputBytes} bytes in ${metrics.timeMs.toFixed(2)}ms`);
  console.log(`Throughput: ${metrics.throughputMbps.toFixed(2)} MB/s`);
}

Synchronous API

After WASM initialization, synchronous versions are available for scenarios where async/await is not suitable.

compressSync(data: Uint8Array, options?: CompressOptions): CompressResult

import { initWasm, compressSync } from '@neutrondev/kalla-oi-wasm-compress';

await initWasm();

// Now sync API is available
const result = compressSync(data, { filename: 'data.txt' });

decompressSync(data: Uint8Array): DecompressResult

import { decompressSync } from '@neutrondev/kalla-oi-wasm-compress';

const result = decompressSync(compressedData);

Compression Algorithms

Brotli

Brotli is the default algorithm for text and structured data. It provides excellent compression ratios, especially for text content.

| Aspect | Details | |--------|---------| | Best for | Text, JSON, CSV, HTML, CSS, JavaScript, logs | | Compression ratio | Excellent (typically 70-90% reduction for text) | | Speed | Moderate (optimized for ratio over speed) | | Use case | Archival, network transfer, storage optimization |

Deflate

Deflate provides a balance between compression ratio and speed. It's the algorithm behind ZIP and gzip.

| Aspect | Details | |--------|---------| | Best for | General binary data, real-time compression | | Compression ratio | Good (typically 50-70% reduction) | | Speed | Fast | | Use case | Real-time compression, streaming |

Store (Passthrough)

Store mode passes data through without compression. Used automatically for already-compressed formats.

| Aspect | Details | |--------|---------| | Best for | JPEG, PNG, MP4, ZIP, PDF, and other compressed formats | | Compression ratio | 1:1 (no compression) | | Speed | Instant | | Use case | Avoiding double-compression overhead |

Adaptive Strategy

The library automatically selects the optimal compression strategy based on multiple factors:

File Type Detection

Files are detected using magic bytes (file signatures) and extension hints:

| File Type | Magic Bytes | Extensions | Algorithm | |-----------|-------------|------------|-----------| | Text/Structured | | .txt, .csv, .json, .xml, .html, .css, .js, .ts, .log | Brotli | | JPEG | FF D8 FF | .jpg, .jpeg | Store | | PNG | 89 50 4E 47 | .png | Store | | GIF | 47 49 46 38 | .gif | Store | | WebP | 52 49 46 46 ... 57 45 42 50 | .webp | Store | | MP4 | 00 00 00 .. 66 74 79 70 | .mp4, .m4v, .m4a | Store | | WebM | 1A 45 DF A3 | .webm | Store | | MP3 | FF FB, FF FA, 49 44 33 | .mp3 | Store | | ZIP | 50 4B 03 04 | .zip | Store | | GZIP | 1F 8B | .gz, .gzip | Store | | PDF | 25 50 44 46 | .pdf | Store | | XLSX | 50 4B 03 04 + xl/ | .xlsx | Store | | DOCX | 50 4B 03 04 + word/ | .docx | Store |

Entropy Analysis

For unknown file types, Shannon entropy is calculated to determine compressibility:

H = -Σ p(x) × log₂(p(x))

Where p(x) is the probability of each byte value.

| Entropy (bits/byte) | Interpretation | Algorithm | |---------------------|----------------|-----------| | 0.0 - 4.0 | Highly compressible (repetitive data) | Brotli | | 4.0 - 7.0 | Moderately compressible | Brotli/Deflate | | 7.0 - 8.0 | Low compressibility (random/encrypted) | Store |

Minimum Savings Threshold

Compression is only applied if it achieves meaningful space savings:

| Content Type | Minimum Savings | |--------------|-----------------| | Text/Structured data | ≥ 5% | | Unknown data | ≥ 2% | | Already compressed | Skipped (Store) |

Binary Format Specification

Compressed data uses a 27-byte header followed by the payload:

┌─────────────────────────────────────────────────────────────────┐
│                        HEADER (27 bytes)                        │
├────────┬──────┬───────────────────────────────────────────────┤
│ Offset │ Size │ Field                                          │
├────────┼──────┼───────────────────────────────────────────────┤
│ 0      │ 4    │ Magic bytes: "CWZ1" (0x43, 0x57, 0x5A, 0x31)  │
│ 4      │ 1    │ Version (currently 1)                         │
│ 5      │ 1    │ Flags (bit0=stored, bit1=chunked, bit2=dict)  │
│ 6      │ 1    │ Algorithm (0=Store, 1=Brotli, 2=Deflate)      │
│ 7      │ 4    │ Chunk size (u32 LE, 0 if not chunked)         │
│ 11     │ 8    │ Original size (u64 LE)                        │
│ 19     │ 4    │ Dictionary ID (u32 LE, reserved)              │
│ 23     │ 4    │ CRC32 checksum of original data               │
├────────┴──────┴───────────────────────────────────────────────┤
│                       PAYLOAD (N bytes)                        │
├───────────────────────────────────────────────────────────────┤
│ Compressed data (single block or multiple chunks)             │
└───────────────────────────────────────────────────────────────┘

Chunked Format

For large files, the payload contains multiple chunks:

┌─────────────────────────────────────────────────────────────────┐
│                     CHUNKED PAYLOAD                             │
├────────┬──────┬───────────────────────────────────────────────┤
│ For each chunk:                                                │
│ - 4 bytes: Compressed chunk size (u32 LE)                     │
│ - N bytes: Compressed chunk data                               │
└───────────────────────────────────────────────────────────────┘

Web Worker Support

For large files, use Web Workers to avoid blocking the main thread:

import { createCompressionWorker } from '@neutrondev/kalla-oi-wasm-compress/worker';

// Create and initialize worker
const worker = await createCompressionWorker();

// Compress in background (non-blocking)
const compressed = await worker.compress(largeData, {
  filename: 'large-file.csv',
  level: CompressionLevel.Best
});

// Decompress in background
const decompressed = await worker.decompress(compressed.data);

// Get file info
const info = await worker.getFileInfo(compressed.data);

// Clean up when done
worker.terminate();

Worker with Custom WASM URL

import { createCompressionWorker } from '@neutrondev/kalla-oi-wasm-compress/worker';

const worker = await createCompressionWorker('https://cdn.example.com/wasm/compression.wasm');

Error Handling

The library throws CompressionError with specific error codes:

import { compress, decompress, CompressionError, ErrorCode } from '@neutrondev/kalla-oi-wasm-compress';

try {
  const result = await decompress(invalidData);
} catch (error) {
  if (error instanceof CompressionError) {
    switch (error.code) {
      case ErrorCode.InvalidHeader:
        console.error('Invalid or corrupted compressed data');
        break;
      case ErrorCode.ChecksumMismatch:
        console.error('Data integrity check failed');
        break;
      case ErrorCode.SizeLimit:
        console.error('Input exceeds 100MB limit');
        break;
      case ErrorCode.RatioExceeded:
        console.error('Decompression bomb detected');
        break;
      default:
        console.error(`Compression error: ${error.message}`);
    }
  }
}

Error Codes

| Code | Name | Description | |------|------|-------------| | 1 | InvalidHeader | Missing or malformed header | | 2 | UnsupportedAlgo | Unknown compression algorithm | | 3 | TruncatedInput | Incomplete compressed data | | 4 | SizeLimit | Input exceeds 100MB maximum | | 5 | ChecksumMismatch | CRC32 verification failed | | 6 | OutputMismatch | Decompressed size doesn't match header | | 7 | CompressionFailed | Algorithm-specific compression error | | 8 | DecompressionFailed | Algorithm-specific decompression error | | 9 | RatioExceeded | Decompression ratio exceeds 10x (bomb protection) |

Performance

Benchmarks

Typical performance on modern hardware (M1/Intel i7):

| Operation | Data Type | Size | Time | Throughput | |-----------|-----------|------|------|------------| | Compress (Brotli) | Text/JSON | 1 MB | ~15ms | ~67 MB/s | | Compress (Deflate) | Binary | 1 MB | ~8ms | ~125 MB/s | | Compress (Store) | JPEG | 1 MB | ~1ms | ~1000 MB/s | | Decompress | Any | 1 MB | ~5ms | ~200 MB/s |

Optimization Tips

1. Use filename hints - Providing filename enables instant file type detection without entropy analysis
2. Match compression level to use case - Use Fast for real-time, Best for archival
3. Use Web Workers for large files - Prevents UI blocking in browsers
4. Adjust chunk size - Smaller chunks use less memory but have more overhead

Browser & Node.js Compatibility

Browser Support

| Browser | Minimum Version | |---------|-----------------| | Chrome | 57+ | | Firefox | 52+ | | Safari | 11+ | | Edge | 79+ |

Node.js Support

| Node.js | Support | |---------|---------| | 18.x | Full support | | 20.x | Full support | | 22.x | Full support |

ESM Only

This package is ESM-only and requires:

{
  "type": "module"
}

Or use .mjs extension for your files.

License

MIT License - see LICENSE for details.

Built with Rust and WebAssembly for maximum performance and reliability.