@shelby-protocol/clay-codes

WebAssembly bindings for Clay codes, a class of regenerating codes for distributed storage.

Installation

npm install @shelby-protocol/clay-codes

Usage

import {
  createDecoder,
  createEncoder,
  type EncoderOptions,
} from '@shelby-protocol/clay-codes';

const SAMPLE_OPTIONS: EncoderOptions = {
  // Total number of chunks (data + parity)
  n: 10,
  // Number of systematic data chunks
  k: 6,
  // Helper chunks contacted during repair
  d: 4,
  // Size of each chunk in bytes
  chunkSizeBytes: 1024,
};

// Input of Uint8Array of length k
const input = Array.from({ length: SAMPLE_OPTIONS.k }, () =>
  new Uint8Array(SAMPLE_OPTIONS.chunkSizeBytes),
);

// Encoder + erasure code
const encoder = await createEncoder(SAMPLE_OPTIONS);
const { chunks } = encoder.erasureCode(input);

// Create decoder
const decoder = await createDecoder({
  ...SAMPLE_OPTIONS,
  erasedChunkIndexes: [0, SAMPLE_OPTIONS.n - 1],
});

// Supply every surviving chunk in index order (systematic first, parity next)
const survivors = chunks.filter((_, idx) =>
  idx !== 0 && idx !== SAMPLE_OPTIONS.n - 1,
);
const recovered = decoder.decode(survivors);
console.log(recovered.systematic[0]);

Requirements

This package expects the compiled artifact at dist/clay.wasm. Generate it from the C sources with:

pnpm run build:wasm

The helper script builds natively on Linux hosts; on macOS/Windows it automatically spins up the repo's C toolchain container to run MACHINE=unknown_clang_wasm32 make and copies the wasm payload into dist/.

Development

# Install dependencies
pnpm install

# Run tests (requires dist/clay.wasm)
pnpm test

# Build
pnpm build

# Format code
pnpm fmt

# Lint code  
pnpm lint

WASM Integration

The package is designed to work in both Node.js and browser environments. The WASM module is loaded differently depending on the runtime:

• Node.js: Reads the WASM file from disk using fs/promises
• Browser: Uses fetch() with optional streaming compilation

API

createEncoder(options)

Creates a new Clay encoder instance.

Parameters:

• options: EncoderOptions object
  • n: Total number of chunks
  • k: Number of data chunks
  • d: Number of chunks to contact for repair
  • chunkSizeBytes: Size of each chunk in bytes

Returns: Promise<EncoderAPI>

EncoderAPI

• setChunk(idx, data): Copy a systematic chunk into the encoder prior to running
• run(): Execute the Clay encoding routine, filling both data and parity slots
• getChunk(idx): Retrieve either a data or parity chunk after encoding
• erasureCode(input): High-level helper that stages k chunks (either an array of Uint8Arrays or a flat buffer), runs the encoder, and returns a ChunkCollection
• free?(): Optional cleanup method (if implemented)

createDecoder(options)

Constructs a Clay decoder capable of rebuilding erased chunks.

Parameters:

• options: DecoderOptions object (extends EncoderOptions)
  • n: Total number of chunks
  • k: Number of data chunks
  • d: Number of chunks to contact for repair
  • chunkSizeBytes: Size of each chunk in bytes
  • erasedChunksMask: Bit-mask of erased chunk indexes (optional)
  • erasedChunkIndexes: Array of erased chunk indexes (optional)
  • availableChunkIndexes: Array of surviving chunk indexes (optional)

Provide at least one of the erased/available hints so the decoder knows which chunks must be reconstructed.

Returns: Promise<DecoderAPI>

DecoderAPI

• setChunk(idx, data): Stage a surviving chunk (systematic or parity) inside the decoder
• run(): Perform decoding once at least k chunks have been staged
• getChunk(idx): Read back a reconstructed or intact chunk after decoding
• configure(options?): Reconfigure the decoder for a new erasure pattern without reallocating WASM. Returns available chunk indexes in sorted order. If options is omitted, reuses the last configuration.
• decode(available, config?): High-level helper that stages the surviving chunks (array of Uint8Array), runs decoding, and returns a ChunkCollection. The optional config parameter allows reconfiguration for a different erasure pattern
• free?(): Optional cleanup method (if implemented)

Decoder Reconfiguration:

The decoder can be reconfigured for different erasure patterns without re-instantiating the WASM module:

const decoder = await createDecoder({
  n: 4,
  k: 2,
  d: 3,
  chunkSizeBytes: 128,
  erasedChunkIndexes: [0, 3],
});

// Decode with initial erasure pattern
let recovered = decoder.decode([chunk1, chunk2], { erasedChunkIndexes: [0, 3] });

// Decode with new erasure pattern - the config parameter reconfigures the decoder
recovered = decoder.decode([chunk0, chunk3], { erasedChunkIndexes: [1, 2] });

The erasure pattern can be specified in three equivalent formats:

• erasedChunksMask: Bit mask where bit i set to 1 means chunk i is erased
• erasedChunkIndexes: Array of erased chunk indexes
• availableChunkIndexes: Array of available chunk indexes (complement is treated as erased)

Note: Coding parameters (n, k, d, chunkSizeBytes) are fixed per encoder/decoder instance and cannot be reconfigured. This is because these parameters determine the WASM memory layout and buffer sizes. To use different parameters, create a new encoder/decoder instance.

Error Handling

The package exports custom error classes for better error handling:

• InvalidChunkIndexError: Thrown when a chunk index is invalid (negative, non-integer, exceeds bounds, or exceeds 32-bit mask width)
• DuplicateChunkIndexError: Thrown when duplicate chunk indexes are provided

Both error classes include the problematic index as a public readonly property:

import { InvalidChunkIndexError, DuplicateChunkIndexError } from '@shelby-protocol/clay-codes';

try {
  decoder.configure({ erasedChunkIndexes: [1, 1, 3] });
} catch (e) {
  if (e instanceof DuplicateChunkIndexError) {
    console.error(`Duplicate chunk index: ${e.index}`);
  }
}

Supporting Types

• ChunkCollection: Returned by the high-level helpers and exposes .chunks (all output in index order), .systematic, and .parity
• flattenSystematic(collection, chunkSizeBytes): Utility that returns a flat Uint8Array containing just the systematic portion of a ChunkCollection