@qvac/diffusion-cpp

v0.25.0

Published

3 days ago

stable-diffusion.cpp addon for qvac image/video generation

0High
0Medium
0Low

mafintosh

prdn

diffusion-cpp

Native C++ addon for image, video, interactive world-walk, and ESRGAN inference through qvac-ext-stable-diffusion.cpp, built for the Bare Runtime.

The package exposes four JS entry points:

| API | Entry point | Use case | | ---------------------- | ------------------------------------------- | ------------------------------------------------------------------------------- | | ImgStableDiffusion | @qvac/diffusion-cpp | Text-to-image, image-to-image, FLUX.2 reference fusion, optional ESRGAN upscale | | VideoStableDiffusion | @qvac/diffusion-cpp/video or named export | Wan and LTX text-to-video / image-to-video | | WorldStableDiffusion | @qvac/diffusion-cpp/world | ABot-World interactive walk: block-by-block generation under keyboard input | | EsrganUpscaler | named export from @qvac/diffusion-cpp | Standalone PNG/JPEG upscaling |

Supported Models

| Family | Mode | Notes | | ---------------------------- | -------------------------------------- | --------------------------------------------------------------------------------- | | SD1.x / SD2.x | image | All-in-one checkpoints through files.model | | SDXL | image | All-in-one checkpoints or split encoders when required | | SD3 | image | Supports split CLIP-L / CLIP-G / T5-XXL inputs | | FLUX.2 [klein] | image, img2img, multi-reference fusion | Split diffusion + Qwen3 LLM + VAE | | Wan 2.1 | text-to-video, image-to-video | Single diffusion expert; I2V requires CLIP vision | | Wan 2.2 TI2V-5B Turbo Q5_K_S | text-to-video | Community-distilled GGUF with Wan 2.2 VAE; use scripts/download-model-wan2.2.sh | | LTX-2 / LTXAV | text-to-video + audio | Gemma text encoder, video VAE, audio VAE, embedding connectors | | ABot-World | interactive world walk | Causal block-by-block generation under keyboard input; see ABot-World guide | | ESRGAN | upscale | Standalone or post-generation image upscale |

Supported Platforms

| Platform | Architecture | Status | GPU backend | | -------- | ------------ | ------ | -------------- | | macOS | arm64, x64 | Tier 1 | Metal | | Linux | arm64, x64 | Tier 1 | Vulkan | | Android | arm64 | Tier 1 | Vulkan, OpenCL | | iOS | arm64 | Tier 1 | Metal | | Windows | x64 | Tier 1 | Vulkan |

Dependencies:

qvac-ext-stable-diffusion.cpp
ggml
Bare Runtime >= 1.24.0
CMake >= 3.25 and a C++20-capable compiler

Building From Source

See build.md for prerequisites, platform setup, cross-compilation, and troubleshooting.

npm install -g bare bare-make
npm install
npm run build

CUDA builds can be generated explicitly:

npm run build:cuda

Downloading Models

Download scripts populate packages/diffusion-cpp/models/ with the files used by the examples.

| Script | Model set | | ------------------------------------- | ----------------------------- | | ./scripts/download-model.sh | FLUX.2 [klein] 4B image model | | ./scripts/download-model-sd2.sh | SD2.x example model | | ./scripts/download-model-sd3.sh | SD3 example files | | ./scripts/download-model-sdxl.sh | SDXL example files | | ./scripts/download-model-wan.sh | Wan 2.1 T2V 1.3B | | ./scripts/download-model-wan-14b.sh | Wan larger T2V variant | | ./scripts/download-model-wan-i2v.sh | Wan 2.1 I2V 14B + CLIP vision | | ./scripts/download-model-ltx.sh | LTX-2.3 video + audio files | | ./scripts/download-model-abot.sh | ABot-World set into test/model/abot (from the P2P model registry, no credentials) |

The FLUX.2 [klein] default image example uses:

| Role | File | | --------------- | --------------------------- | | Diffusion model | flux-2-klein-4b-Q8_0.gguf | | Text encoder | Qwen3-4B-Q4_K_M.gguf | | VAE | flux2-vae.safetensors |

LTX-2.3 requires more companion files:

| Role | Default file | | -------------------- | --------------------------------------------------------- | | Diffusion model | LTX-2.3-22B-distilled-1.1-Q5_K_M.gguf | | Text encoder | gemma-3-12b-it-UD-Q4_K_XL.gguf | | Video VAE | ltx-2.3-22b-distilled_video_vae.safetensors | | Audio VAE | ltx-2.3-22b-distilled_audio_vae.safetensors | | Embedding connectors | ltx-2.3-22b-distilled_embeddings_connectors.safetensors |

Downloads are resumable where supported by the script.

Examples

| Command | Description | | -------------------------------------------- | ------------------------------------------- | | npm run example | Load/unload the default FLUX.2 model | | npm run generate | FLUX.2 text-to-image | | npm run generate:sd2 | SD2.x text-to-image | | npm run generate:sd3 | SD3 text-to-image | | npm run generate:sdxl | SDXL text-to-image | | bare examples/generate-fusion.js | FLUX.2 multi-reference fusion | | bare examples/img2img-flux2.js | FLUX.2 single-reference img2img | | bare examples/img2img-sd3.js | SD3 SDEdit img2img | | bare examples/generate-video-wan.js | Wan 2.1 text-to-video | | bare examples/img2vid-wan.js | Wan 2.1 image-to-video | | npm run generate:video | Wan text-to-video | | npm run generate:ltx | LTX-2.3 text-to-video with audio | | npm run generate:ltx-coffee | 9-second LTX Ingredients coffee example | | npm run generate:esrgan | Image generation followed by ESRGAN upscale | | bare examples/standalone-esrgan-upscale.js | Standalone ESRGAN upscale | | npm run walk:world | ABot-World browser demo (generate + walk a world) |

Outputs are written to packages/diffusion-cpp/output/.

Image API

const path = require('bare-path')
const ImgStableDiffusion = require('@qvac/diffusion-cpp')

const MODELS_DIR = path.resolve(__dirname, './models')

const model = new ImgStableDiffusion({
  files: {
    model: path.join(MODELS_DIR, 'flux-2-klein-4b-Q8_0.gguf'),
    llm: path.join(MODELS_DIR, 'Qwen3-4B-Q4_K_M.gguf'),
    vae: path.join(MODELS_DIR, 'flux2-vae.safetensors')
  },
  config: {
    threads: 4,
    device: 'gpu',
    diffusion_fa: true
  },
  opts: { stats: true },
  logger: console
})

await model.load()

const response = await model.run({
  prompt: 'a majestic red fox in a snowy forest, golden light, photorealistic',
  width: 512,
  height: 512,
  steps: 20,
  guidance: 3.5,
  seed: 42
})

const images = []
await response
  .onUpdate((data) => {
    if (data instanceof Uint8Array) images.push(data) // PNG bytes
  })
  .await()

require('bare-fs').writeFileSync('output.png', images[0])
await model.unload()

Constructor Files

All file paths must be absolute.

| Key | Required | Description | | ------------------------------- | -------: | --------------------------------------------------------------------------- | | files.model | yes | Main model. All-in-one checkpoint for SD, diffusion model for split layouts | | files.clipL | no | CLIP-L text encoder for SD3 / split layouts | | files.clipG | no | CLIP-G text encoder for SDXL / SD3 | | files.t5Xxl | no | T5-XXL text encoder for SD3 | | files.llm | no | Qwen3 LLM text encoder for FLUX.2 [klein] | | files.vae | no | Separate VAE | | files.esrgan | no | ESRGAN model for post-generation upscale | | files.highNoiseDiffusionModel | no | Wan 2.2 high-noise expert path; normally used by VideoStableDiffusion |

Passing any separate text encoder (llm, t5Xxl, clipL, clipG) makes the wrapper route files.model to stable-diffusion.cpp's diffusion_model_path. All-in-one checkpoints are routed to model_path.

Image Config

config is part of the constructor object. There is no second constructor argument.

| Key | ----------------------- | threads | device | main-gpu | type | rng | sampler_rng | vae_decode_only | vae_tiling | flash_attn | diffusion_fa | mmap | offload_to_cpu | backend | params_backend | max_vram | stream_layers | prediction | flow_shift | diffusion_conv_direct | vae_conv_direct | backendsDir | lora_apply_mode | upscaler_tile_size | Type | Default | Description | | ----------------------------------------- | ----------------- | ------------------------------------------------------------------------- | | number | auto | CPU threads for loading / CPU ops | | 'gpu' | 'cpu' | 'gpu' | Prefer GPU backends or force CPU | | number | 'integrated' | 'dedicated' | unset | Pin the GPU selected by stable-diffusion.cpp | | weight type | auto | Override weight quantization | | 'cpu' | 'cuda' | 'std_default' | 'cuda' | Context RNG; cuda means Philox and is not GPU-specific | | RNG type | auto | Sampler RNG override | | boolean | false | Load only VAE decoder weights; leave false for img2img/fusion/hires paths | | boolean | false | Tile VAE decode to reduce peak VRAM | | boolean | false | Enable flash attention globally | | boolean | true | Enable diffusion-model flash attention; important for FLUX/LTX memory use | | boolean | backend default | Memory-map weights when supported | | boolean | backend default | Keep weights on CPU/offload as supported by backend | | string | auto | Runtime backend for all modules or per-module assignments | | string | runtime backend | Parameter residency on a backend, CPU RAM, or disk | | number | string | 0 | VRAM budget in GiB for graph-cut segmented execution | | boolean | false | Stream diffusion layers from CPU RAM when graph cutting is active | | prediction type | auto | Required for FLUX img2img/fusion routing; use 'flux2_flow' for FLUX.2 | | number | model default | Flow-matching noise schedule shift | | boolean | true | Use direct convolution in diffusion model | | boolean | true | Use direct convolution in VAE | | string | package prebuilds | Custom ggml backend directory | | string | auto | LoRA application mode | | number | 128 | ESRGAN tile size |

main-gpu is resolved against the addon's own ggml device enumeration and then pinned through sd_ctx_params_t.backend. If an explicit request cannot be satisfied ('integrated' with no integrated GPU, 'dedicated' with no discrete GPU, or an out-of-range index), the addon falls back to CPU instead of silently choosing another GPU. Mobile targets reject main-gpu because they are single-GPU devices. An explicit backend assignment takes precedence over main-gpu.

backend controls where graphs execute. params_backend controls where model weights remain between uses. For example, backend: 'diffusion=cuda0,te=cpu' runs the diffusion model on CUDA and the text encoder on CPU.

params_backend: 'diffusion=cpu' keeps diffusion weights in CPU RAM and stages them to the runtime backend. params_backend: 'diffusion=disk' reloads those weights from the model file on demand and releases them after use — on every job, not only the first — so a disk-backed module also disables eager weight loading for the whole context. Disk is not selected automatically.

offload_to_cpu: true supplies a *=cpu default. An explicit params_backend entry written in module=backend form overrides that default for that module only: params_backend: 'te=disk' with offload_to_cpu: true keeps TE weights on disk while other parameters remain in CPU RAM.

A bare entry or an assignment to *, all, or default sets the whole-spec default rather than a per-module override, and the last default wins. For example, params_backend: 'cuda0' with offload_to_cpu: true puts every module on cuda0 and offloads nothing. The addon logs when the final whole-spec default differs from CPU. Equivalent defaults such as all=cpu do not produce an error. Write params_backend: 'diffusion=cuda0' to move one module and leave the rest offloaded.

A nonzero max_vram enables graph-cut segmentation even without stream_layers. Positive values cap the VRAM budget in GiB. Negative values use detected free VRAM while reserving the absolute value as headroom, and 0 disables graph cutting. Backend assignments such as max_vram: 'cuda0=6,vulkan0=4' apply one budget per device.

stream_layers adds diffusion-layer prefetch and eviction only when graph cutting is active and the diffusion parameter backend is CPU, for example:

config: {
  backend: 'cuda0',
  params_backend: 'diffusion=cpu',
  max_vram: -1,
  stream_layers: true
}

It does not stream from disk. Use params_backend: 'diffusion=disk' for on-demand reads from the model file.

stream_layers is forwarded to the engine as configured; the engine itself skips streaming when its prerequisites are unmet. The addon reports the cases it can prove before engine initialization: max_vram is unset, 0, or an all-zero assignment such as 'cuda0=0'. Per-backend and automatic negative budgets are resolved by the engine using the selected runtime backend and its free memory. That message, and the main-gpu and params_backend notices above, are the only diagnostics emitted at the default verbosity: 0; set verbosity: 2 to also see the effective backend, params_backend and max_vram assignments the addon passes to the engine.

The 16-case Linux hardware matrix is available in scripts/validate-layer-streaming.sh. It expects the MiniMax-H3 files under /home/shared/models/minimax-h3-q2 by default. Override that location with H3_MODELS_DIR. The runtime backend defaults to vulkan0; override it with BACKEND, for example BACKEND=cuda0. Then run the script from the package directory. Each case requires a non-empty AVI and checks the engine log for the expected graph-cut, streaming, CPU RAM, or disk behavior.

Image Generation Parameters

| Key | Type | Description | | ----------------------------- | ----------------- | ------------------------------------------------------------------- | | prompt | string | Required non-empty prompt | | negative_prompt | string | Negative prompt | | width, height | number | Positive multiples of 8; FLUX img2img defaults omitted axes to 1024 | | steps | number | Diffusion step count | | cfg_scale | number | Classifier-free guidance for SD/SDXL/SD3 and FLUX img2img examples | | guidance | number | Distilled guidance for FLUX.2 | | sampling_method / sampler | string | Sampler name; omit for auto-selection | | scheduler | string | Scheduler name; omit for auto-selection | | seed | number | -1 for random | | batch_count | number | Number of images | | lora | string | Absolute path to a LoRA adapter | | upscale | boolean | object | Post-generation ESRGAN upscale; requires files.esrgan | | vae_tiling | boolean | Per-job VAE tiling | | vae_tile_size | number | string | VAE tile size, e.g. 512 or '512x512' | | vae_tile_overlap | number | Tile overlap fraction | | cache_mode | string | Step-caching algorithm | | cache_preset | string | Cache preset: slow, medium, fast, ultra | | cache_threshold | number | Cache reuse threshold | | eta | number | DDIM/TCD stochasticity | | clip_skip | number | Skip last N CLIP layers |

Do not force sampling_method: 'euler_a' for FLUX.2 models. Leave the sampler unset unless you know the model family requires an override.

Image-to-Image and FLUX.2 Fusion

init_image accepts PNG/JPEG bytes and selects image-to-image mode.

FLUX.2 uses in-context conditioning: the reference image is VAE-encoded into separate latent tokens and the target starts from pure noise.
SD1.x / SD2.x / SDXL / SD3 use SDEdit: the input is noised according to strength and denoised with the prompt.

For FLUX.2 img2img, load the context with files.llm and config.prediction: 'flux2_flow'.

const inputImage = require('bare-fs').readFileSync('assets/source.jpg')

const response = await model.run({
  prompt: 'a cinematic portrait of the same person, professional lighting',
  init_image: inputImage,
  width: 1024,
  height: 1024,
  cfg_scale: 1.0,
  guidance: 3.5,
  steps: 20,
  seed: 42
})

FLUX.2 multi-reference fusion uses init_images, an array of PNG/JPEG buffers. It is mutually exclusive with init_image and requires config.prediction: 'flux2_flow'.

const response = await model.run({
  prompt: 'blend @image1 and @image2 into one scientist in a black studio',
  init_images: [image1Bytes, image2Bytes],
  width: 624,
  height: 624,
  cfg_scale: 1.0,
  guidance: 3.5,
  steps: 10,
  seed: 10
})

@image1, @image2, etc. are prose anchors for the text encoder. The FLUX.2 Qwen3 text encoder does not receive vision tokens; visual fusion happens in the DiT through attention over reference latents. Keep increase_ref_index unset or false for FLUX.2-klein fusion.

Video API

const path = require('bare-path')
const fs = require('bare-fs')
const VideoStableDiffusion = require('@qvac/diffusion-cpp/video')

const MODELS_DIR = path.resolve(__dirname, './models')

const model = new VideoStableDiffusion({
  files: {
    model: path.join(MODELS_DIR, 'wan2.1_t2v_1.3B_fp16.safetensors'),
    t5Xxl: path.join(MODELS_DIR, 'umt5_xxl_fp16.safetensors'),
    vae: path.join(MODELS_DIR, 'wan_2.1_vae.safetensors')
  },
  config: {
    threads: 4,
    device: 'gpu',
    diffusion_fa: true,
    offload_to_cpu: true,
    vae_tiling: true
  },
  logger: console
})

await model.load()

const response = await model.run({
  mode: 'txt2vid',
  prompt: 'a colorful bird flapping its wings',
  negative_prompt: 'blurry, low quality, static, jittery, watermark',
  width: 480,
  height: 832,
  video_frames: 81,
  fps: 16,
  steps: 30,
  cfg_scale: 6.0,
  flow_shift: 3.0,
  seed: 42
})

let avi = null
await response
  .onUpdate((data) => {
    if (data instanceof Uint8Array) avi = data // MJPG AVI bytes
  })
  .await()

fs.writeFileSync('wan_t2v_seed42.avi', avi)
await model.unload()

The default export from @qvac/diffusion-cpp/video and the named VideoStableDiffusion export from @qvac/diffusion-cpp are the same class.

Video Files

| Key | Model family | Description | | ------------------------------- | ------------ | --------------------------------------------------------------- | | files.model | all video | Wan single/low-noise expert or LTX diffusion transformer | | files.t5Xxl | Wan | UMT5-XXL text encoder | | files.vae | Wan / LTX | Wan VAE or LTX video VAE | | files.clipVision | Wan I2V | OpenCLIP ViT-H/14; required for mode: 'img2vid' on Wan | | files.llm | LTX | Gemma text encoder | | files.audioVae | LTX | Audio VAE decoder for synchronized audio | | files.embeddingsConnectors | LTX | Text-embedding connector weights; also marks the context as LTX |

Video Parameters

| Key | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------ | | mode | Required: 'txt2vid' or 'img2vid' | | prompt, negative_prompt | Text conditioning | | width, height | Wan: multiples of 16. LTX: multiples of 32 | | video_frames | Wan: (4*k + 1). LTX: (8*k + 1), max 257 | | fps | AVI framerate metadata, default 16 for Wan examples and 24 for LTX | | steps, cfg_scale, sampling_method, scheduler, seed | Sampling controls | | flow_shift | Per-job flow-shift override; Wan 2.1 T2V 1.3B works well at 3.0 | | init_image | First frame for img2vid; required by that mode | | control_frames, vace_strength | Optional VACE guidance | | temporal_tiling | LTX-only temporal VAE tiling to reduce peak VRAM | | cache_mode, cache_preset, cache_threshold | Step-cache controls |

Video output is a single MJPG AVI Uint8Array. For LTX-2 models loaded with audioVae, the AVI also contains a second IEEE-float PCM stream at 48 kHz. VLC handles these files well.

Wan 2.2

The supported Wan 2.2 workflow is the community-distilled TI2V-5B Turbo Q5_K_S GGUF for text-to-video. The downloader does not include the CLIP vision encoder required by the wrapper's image-to-video path. Download its complete text-to-video model layout before running the example:

# TI2V-5B Turbo Q5_K_S (4 steps, CFG 1)
./scripts/download-model-wan2.2.sh
npm run generate:wan22

The download is approximately 16.4 GB: 3.56 GB for the Q5_K_S diffusion model, 11.4 GB for the fp16 UMT5-XXL encoder, and 1.41 GB for the Wan 2.2 VAE. The backend runs one process on one device with optional CPU offload. Use dimensions that are multiples of 32 for this TI2V model so the emitted AVI dimensions match the requested dimensions.

MiniMax-H3 Text-to-Audio-Video

MiniMax-H3 is supported for prompt-only video generation. Download the matching FL2VA denoiser, Qwen3-VL text encoder, video VAE, and audio VAE:

./scripts/download-model-minimax-h3.sh --q4

Set H3_MODELS_DIR when running npm run generate:h3-coffee with model files stored outside the package.

The initial integration intentionally rejects init images, control frames, and reference images. Use a 32-pixel spatial grid and a 17*k + 5 frame count. H3 is distilled: cfg_scale must be 1.0, and the output stream is always 24 FPS with its native stereo audio.

const VideoStableDiffusion = require('@qvac/diffusion-cpp/video')
const model = new VideoStableDiffusion({
  files: {
    model: '/models/minimax-h3/minimax_h3_fl2va_pruned-Q4_K.gguf',
    llm: '/models/minimax-h3/qwen3vl_32b_minimax_h3-Q4_K_M.gguf',
    vae: '/models/minimax-h3/vae/minimax_h3_video_vae_fp16.safetensors',
    audioVae: '/models/minimax-h3/vae/minimax_h3_audio_vae_fp32.safetensors'
  },
  config: { device: 'gpu', diffusion_fa: true, offload_to_cpu: true }
})
await model.load()
const response = await model.run({
  mode: 'txt2vid',
  prompt: 'A cinematic close-up of a person enjoying coffee in warm morning light.',
  width: 960,
  height: 544,
  video_frames: 124,
  fps: 24,
  steps: 8,
  cfg_scale: 1.0
})

LTX-2 Text-to-Video With Audio

const model = new VideoStableDiffusion({
  files: {
    model: path.join(MODELS_DIR, 'LTX-2.3-22B-distilled-1.1-Q5_K_M.gguf'),
    llm: path.join(MODELS_DIR, 'gemma-3-12b-it-UD-Q4_K_XL.gguf'),
    vae: path.join(MODELS_DIR, 'ltx-2.3-22b-distilled_video_vae.safetensors'),
    audioVae: path.join(MODELS_DIR, 'ltx-2.3-22b-distilled_audio_vae.safetensors'),
    embeddingsConnectors: path.join(
      MODELS_DIR,
      'ltx-2.3-22b-distilled_embeddings_connectors.safetensors'
    )
  },
  config: {
    threads: 4,
    device: 'gpu',
    diffusion_fa: true,
    vae_tiling: true,
    vae_conv_direct: true
  },
  opts: { stats: true },
  logger: console
})

const response = await model.run({
  mode: 'txt2vid',
  prompt: 'a claymation cat playing jazz on a piano',
  negative_prompt: 'blurry, low quality, static, jittery, watermark, distorted audio',
  width: 512,
  height: 320,
  video_frames: 241,
  fps: 24,
  steps: 10,
  cfg_scale: 1.0,
  temporal_tiling: true,
  seed: 42
})

LTX distilled variants are designed for low step counts and low CFG values. For full dev weights, use higher steps and a larger CFG.

ABot-World Interactive Walk

ABot-World is a causal world model: it generates video block-by-block under live keyboard input instead of one batch call, exposed via @qvac/diffusion-cpp/world (WorldStableDiffusion). Worlds are created natively from a prompt + first-frame image (createScene()), then walked with WASD/IJKL (step()), streaming decoded PNG/JPEG frames. The four model files ship in the QVAC P2P registry.

# models (P2P registry, no credentials) + browser demo
npm install -g @qvac/registry-client   # then see docs/abot-world.md for the 4 downloads
export ABOT_MODELS_DIR=~/abot-models ABOT_KV_CACHE=1
npm run walk:world                     # open http://127.0.0.1:8787

const WorldStableDiffusion = require('@qvac/diffusion-cpp/world')

const world = new WorldStableDiffusion({
  files: { model: ditGguf, taehv: taehvGguf, scene: scenePack },
  config: { seed: 42, kvCache: true }
})
await world.load()
const response = await world.step({ W: true }) // one generated block forward
await response
  .onUpdate((data) => {
    if (data instanceof Uint8Array) frames.push(data) // PNG/JPEG frames
  })
  .await()
await world.unload()

Hardware requirements (interactive walk, Q8 DiT + KV cache):

| tier | GPU | host RAM | disk | measured | |---|---|---|---|---| | minimum @ 832x480 | 24 GB card, >= 20 GB VRAM free (16.3 GB steady + ~2.7 GB transient at block 0) | 8 GB | 14 GB | RTX 5090: 1.78 s/block | | low-VRAM @ 448x256 | ~6 GB | 8 GB | 14 GB | laptop RTX 4050 | | optimal | RTX 5090-class, dedicated (no co-tenant VRAM); optional dual-GPU split ABOT_BACKEND="diffusion=cuda0,vae=cuda1" | 16 GB | NVMe | 6.7-6.8 fps generation |

Performance vs the PyTorch reference (same 5090, same walk): the QVAC build generates 6.7 fps vs ~28.5 fps for the reference's fp8 Triton pipeline — about 4.2x slower per block, the main known trade-off of this implementation today. In exchange it starts 36x faster (first frame in 1.5 s vs ~54 s), uses 14x less host RAM (3.1 GB vs 43.6 GB peak), one CPU core instead of two-plus, 14 GB on disk instead of ~38 GB, and needs no Python runtime.

Model fidelity: the weights are converted to GGUF (DiT and umT5 at Q8_0, VAE and taehv at F16), so outputs are not bit-exact vs the PyTorch reference — parity is held by cosine-similarity gates instead (golden walk replays 0.993-0.99995, scene packs >= 0.997, ~32-38 dB walk-level PSNR), and walks are visually indistinguishable in validation.

Full guide — building, the demo server (local and over-SSH playback), the world API for app developers, resolutions, performance details and troubleshooting: docs/abot-world.md.

ESRGAN Upscaler

const { EsrganUpscaler } = require('@qvac/diffusion-cpp')
const fs = require('bare-fs')

const upscaler = new EsrganUpscaler({
  files: {
    esrgan: '/absolute/path/to/RealESRGAN_x4plus_anime_6B.pth'
  },
  config: {
    device: 'gpu',
    upscaler_tile_size: 128
  },
  logger: console
})

await upscaler.load()

const response = await upscaler.upscale(fs.readFileSync('input.png'), {
  repeats: 1
})

const images = []
await response
  .onUpdate((data) => {
    if (data instanceof Uint8Array) images.push(data)
  })
  .await()

fs.writeFileSync('upscaled.png', images[0])
await upscaler.unload()

repeats controls how many ESRGAN passes are applied. One pass typically scales by 4x; two passes scale by 16x.

Response Streams and Stats

All three wrappers return a QvacResponse.

Progress updates are JSON strings like {"step":1,"total":20,"elapsed_ms":...}. The stream is multi-phase: sampler sequences (one per image batch item / video expert, total = its step count) and, when vae_tiling is enabled, VAE tile passes (total = tile count). Each sequence restarts at step: 0, so a bar renderer should key on total changes rather than assume a single monotonic sequence. Model weights normally load eagerly at load(), so generation emits no loader ticks. Two configurations load lazily instead and do emit loader ticks inside generation: mobile targets, and any params_backend naming disk for one or more modules — the latter on every job, since disk-backed weights are released after each phase. In those cases the first job's conditionerMs also absorbs weight-load time.
Image generation and ESRGAN emit PNG Uint8Array values.
Video generation emits one MJPG AVI Uint8Array.
If opts.stats is enabled, a stats event is emitted before completion.

Image stats include load time, generation time, cumulative steps/images/pixels, dimensions, and seed. Video stats additionally include cumulative videos, frames, fps, hasAudio, and audioSampleRate. ESRGAN stats include upscale timing, output dimensions, repeats, and the backend device actually used.

Cancellation and Unload

Each wrapper has cancel() and unload() methods. Only one job may run per model instance at a time; overlapping run() calls fail with a busy error.

const response = await model.run({ prompt: '...', steps: 30 })
await model.cancel()
await response.await().catch(() => {})
await model.unload()

During ESRGAN upscale, cancellation is honored between repeat passes.

Operational Notes

Native C++ logs are process-global. Configure them once with require('@qvac/diffusion-cpp/addonLogging').setLogger(...).
Leave sampler and scheduler unset for normal use; the addon preserves stable-diffusion.cpp auto-detection for model-specific defaults.
diffusion_fa defaults to true and is important for FLUX/LTX memory use.
For FLUX.2 img2img/fusion, set config.prediction: 'flux2_flow' so the JS wrapper and native layer select the in-context conditioning path.
Wan I2V requires files.clipVision; LTX img2vid does not.
Wan dimensions must be multiples of 16; LTX dimensions must be multiples of 32.
Wan frame counts use (4*k + 1); LTX frame counts use (8*k + 1).
LTX audio is muxed into AVI as IEEE-float PCM at 48 kHz.
On Linux the shipped prebuild must not hard-link any GPU loader (libvulkan/libOpenCL/libcuda) — an integration test asserts this. For local custom builds that legitimately do (e.g. SD_CUDA=ON), skip it with QVAC_SKIP_PREBUILD_LINK_CHECK=true.

Credits

Test Images

assets/von-neumann.jpg — John von Neumann (1956). Source: U.S. Department of Energy, File ID: HD.3F.191. This image is in the Public Domain as a work of the U.S. Federal Government.

assets/claude-shannon.jpg — Claude Shannon. Source: Bell Labs / Wikimedia Commons. Licensed under Creative Commons Attribution-ShareAlike (CC BY-SA). Attribution must be preserved; any redistribution of this image or a derivative must be released under a compatible CC BY-SA license.

License

Apache-2.0 — see LICENSE for details.