Shado

Shado is a packed-data and rendering toolkit for Babylon Lite and Babylon.js. Define a GPU struct once, then use the same layout for TypeScript objects, packed arenas, AssemblyScript reducers, Babylon shader inputs, and instanced rendering.

It is intentionally a library rather than a game engine. The higher-level rendering helpers are optional and the core schema/arena APIs can be used on their own.

Install

npm install @knervous/shado @babylonjs/lite

Babylon Lite is the preferred WebGPU renderer. Install full Babylon.js for WebGL fallback or features that are not yet available in Lite:

npm install @babylonjs/core
npm install @babylonjs/loaders @babylonjs/serializers
npm install --save-dev assemblyscript binaryen

Babylon loaders are needed for model preprocessing and scene-loader examples. AssemblyScript and Binaryen are only needed for runtime compilation or the @knervous/shado/asc APIs; precompiled reducers do not require them at runtime.

Define a packed struct

import { Shado, field, gpuStruct } from '@knervous/shado';

@gpuStruct({ name: 'ActorPool', useWasm: false })
class ActorPool extends Shado {
  @field('mat4') transform!: Float32Array;
  @field('vec4') color!: Float32Array;
  @field({ arrayOf: 'vec3' }) velocities!: Float32Array;
}

await ActorPool.initialize(engine, {
  backend: engine.isWebGPU ? 'storage' : 'datatex',
  wasm: false,
});

const pool = new ActorPool(engine);
pool.color = new Float32Array([1, 0.4, 0.1, 1]);
pool.setVarArray('velocities', [0, 1, 0, 1, 0, 0]);

Use storage with native WGSL on WebGPU. Keep datatex as the WebGL fallback or for a deliberately GLSL-authored compatibility material.

Babylon Lite

The default native Lite path uses public storage, shader-material, scene callback, and thin-instance APIs. It does not replace mesh functions or call private draw methods. Projected compute scatter uses one isolated, feature-detected compatibility bridge because Lite currently keeps its generic WebGPU device and storage handles opaque.

import {
  ShadoActor,
  ShadoLiteInstanceContainer,
  createShadoLiteMaterial,
} from '@knervous/shado/lite';

await ShadoLiteInstanceContainer.initialize(engine, {
  extra: ShadoActor,
  backend: 'storage',
  wasm: false,
});

const actors = new ShadoLiteInstanceContainer<ShadoActor>(engine);
actors.addInstances(1_000);
createShadoLiteMaterial(engine, scene, mesh, actors);

The lossless legacy arena is the default. Opt in to the first packed transform/appearance projection when the actor batch has stable world bounds:

const projected = createShadoLiteMaterial(engine, scene, mesh, actors, {
  projection: {
    encoding: 'packed',
    domain: {
      origin: [-64, -64, -64],
      extent: [128, 128, 128],
      scaleRange: [0, 8],
    },
  },
});

Packed positions and scale clamp to this domain, so re-home actors before they leave it. Use encoding: 'split-f32' as the exact-value component-stream control. Projection mode enables adaptive struct-span compute scatter by default. It shares the full-Babylon delta ABI and falls back to one complete affected-stream write if Lite's runtime handles are unavailable. Set computeScatter: false to force that fallback.

Enable computeScatterGPUTiming to collect timestamp-query results, then inspect actual publication work and timing:

projected.getLastProjectionPublication();
projected.getLastProjectionGPUTiming();

Assign the material before adding mesh to a Lite scene. Use @knervous/shado/renderer to select Lite or full Babylon.js before importing renderer-specific features. See BABYLON_RENDERER_STRATEGY.md for the application loading contract.

For implementation status, upload benchmarks, and the WebGPU-first ECS/SoA roadmap, see SHADO_RENDER_DATA_SCALING.md. For actor populations larger than the active renderer/simulation working set, see the bounded OPFS deferred storage slab proof.

Full Babylon compute scatter

Full Babylon exposes public compute and storage-buffer APIs, so random sparse projection updates can be scattered on the GPU:

import { BabylonActorProjectionPipeline } from '@knervous/shado/babylon';

const projection = new BabylonActorProjectionPipeline(engine, {
  encoding: 'packed',
  domain: {
    origin: [-64, -64, -64],
    extent: [128, 128, 128],
    scaleRange: [0, 8],
  },
});

projection.bind(material);
await projection.publishWhenReady(actors.children, {
  dirtyFlags: actors.getStructDirtyFlags('instances'),
});
actors.clearStructDirtyFlags('instances');

publishWhenReady() compiles/warm-ups whole-row and encoded-field-span kernels and is suitable for loading or asynchronous update code. The planner scatters only packed position/scale or rotation words when that costs less than a full transform row. Use synchronous publish() in frame code after warm-up. If a kernel is unexpectedly unavailable, it safely falls back to one full affected-stream write. Enable Babylon's GPU timing measurements to read the last per-stream scatter duration from getLastGPUTiming().

2D sprites and MSDF text

The full-Babylon render entry point includes a locked orthographic 2D path with compact instanced sprite records, tiled CPU visibility, exact screen picking, per-sprite pixel-size LOD, and optional WebGPU-owned motion and visibility. ShadoSprite2DRenderer uses an array-texture ShadoTextureAtlas; positions and sizes are two-dimensional world units.

import { ShadoSprite2DRenderer, createSolidColorAtlas } from '@knervous/shado/render';

const atlas = createSolidColorAtlas(scene, {
  hero: [0.2, 0.8, 1, 1],
  marker: [1, 0.65, 0.1, 1],
});
const sprites = new ShadoSprite2DRenderer(scene, atlas, {
  alphaMode: 'cutout',
  alphaCutoff: 0.35,
  tileSize: 8,
});

sprites.upsertMany([
  {
    id: 'hero-1',
    textureKey: 'hero',
    position: [4, -2],
    size: [1, 1.6],
    rotationDeg: 15,
    minPixelSize: 1,
  },
]);
sprites.setViewFromOrthographicCamera(camera);
const hit = sprites.pickScreen(scene.pointerX, scene.pointerY);

On WebGPU, a stable population can keep position and velocity in storage buffers. Population or CPU position/visibility mutations return authority to the CPU; call enableGpuMotion() again after completing those changes. Per-sprite LOD thresholds remain active in both the compacted and all-instance GPU paths. GPU-owned positions intentionally disable synchronous picking; readCpuPositions() provides explicit, versioned asynchronous readback.

sprites.enableGpuMotion({
  seed: 42,
  speed: 0.8,
  cadenceMs: 2_000,
  bounds: [-50, -30, 50, 30],
});
scene.onBeforeRenderObservable.add(() => {
  sprites.stepGpuMotion(performance.now(), scene.getEngine().getDeltaTime() / 1_000);
});

ShadoText2DRenderer accepts a Babylon MSDF FontAsset-compatible object and uses the same view contract. It supports kerning, advance-only whitespace, newlines, width wrapping, alignment, pivots, rotation, color, layer ordering, LOD, and screen picking.

The existing ShadoDynamicEntityRenderer remains the compatibility and world-space path. Its explicit sprite presentations are ground, billboard-y, billboard-screen, and slab. Omitted alphaMode preserves the 1.x blended-opacity result through premultiplied blending; choose cutout explicitly for depth-writing opaque sprites. Dynamic-entity picking now follows the resolved presentation and pivot instead of always intersecting a box.

Published controls

@shadoPublish puts a friendly, validated facade in front of packed numeric fields without changing their GPU layout. Enum values map to zero-based field indices by default, while labels, descriptions, groups, and sockets remain available to inspectors and generated UI.

import { field, gpuStruct, ShadoActor, shadoPublish } from '@knervous/shado';

@gpuStruct({ name: 'Character' })
class Character extends ShadoActor {
  @shadoPublish({
    name: 'armor',
    label: 'Armor set',
    description: 'One material family across the whole actor.',
    values: ['armorless', 'leather', 'chain', 'plate'],
  })
  @field('f32')
  armorClass!: number;

  @shadoPublish({
    name: 'mainHand',
    socket: 'r_point',
    values: ['none', 'sword', 'staff'],
  })
  @field('f32')
  weaponClass!: number;
}

actor.published.armor = 'chain'; // armorClass becomes 2
actor.published.mainHand = 'sword'; // weaponClass becomes 1
console.table(actor.published.$describe());

Use $get(name), $set(name, value), $describe(), or toJSON() when the property name is dynamic. Invalid enum values throw before touching packed state. Advanced adapters can provide fromInternal and toInternal.

Precompiled WASM

await ActorPool.initialize(engine, {
  backend: engine.isWebGPU ? 'storage' : 'datatex',
  wasm: {
    mode: 'precompiled',
    module: await fetch('/actor-pool.wasm').then(response => response.arrayBuffer()),
  },
});

Model preprocessing

The CLI can package a Babylon-readable model, bake dual-quaternion VAT data, emit schema wrappers, and build a manifest for runtime loading.

npx shado pack models --config ./shado.config.mjs
npx shado wrappers build --config ./shado.config.mjs
npx shado manifest models --config ./shado.config.mjs

At runtime, compressed artifacts can be fetched directly from a CDN or GitHub:

import { deserializeShadoModel } from '@knervous/shado/preprocess/runtime';

const model = await deserializeShadoModel(
  {
    manifestUrl: 'https://example.com/shado/models.json',
    modelName: 'actor',
  },
  { animation: true, vat: 'auto' }
);

Responsive runtime VAT baking

For source GLBs that must be compiled in the browser, the reusable headless bake worker loads each GLB into a Babylon NullEngine, samples its skeleton, and packs its VAT without touching the render scene or UI thread. The roster showcase runs three independent bake workers concurrently and transfers the source GLB and finished atlas buffers instead of cloning them.

The matrix decomposition, dual-quaternion packing, atlas layout, and float16 conversion use a bundled AssemblyScript WASM kernel. Runtime validation selects relaxed-SIMD, fixed-width SIMD128, or the scalar compatibility kernel in that order. Pass kernel: 'scalar', 'simd', or 'relaxed-simd' to packVatMatrices when profiling a specific variant; normal baking should leave selection automatic.

const packedVat = await bakeVatWithHeadlessWorker(
  '/shado/vat-bake-worker.js',
  await (await fetch('/models/human.glb')).arrayBuffer(),
  { useHalf: true, detectScale: false }
);

The lower-level instance-container path remains available when the source is already loaded in the render scene. It samples there with regular task yields, then transfers packing work to a worker:

await actors.attachMeshes(scene, meshes, skeleton, {
  vat: 'bake',
  vatQuality: 'full', // medium, low, or rigid can back separate LOD draw bins
  vatOptions: {
    execution: 'worker',
    yieldEveryFrames: 6,
    useHalfDQ: true,
    detectScale: false, // only for rigs known to contain rigid bone transforms
    animationGroups,
  },
});

full blends weighted bone influences across two frames. medium keeps the weighted blend but samples one frame, low samples one dominant bone at one frame, and rigid skips VAT generation and renders the rest mesh.

Use detectScale: true (the default) for unknown content. Disable it only when the source rig is known to contain rigid bone transforms.

Custom actor materials

ShadoInstanceContainer owns the generated instancing and DQ/VAT shader. Applications can add material behavior at typed, stable insertion points by overriding getGLSLHooks(); generated source remains an implementation detail.

import {
  ShadoActor,
  ShadoInstanceContainer,
  field,
  gpuStruct,
  type ShadoInstanceGLSLHooks,
} from '@knervous/shado';

@gpuStruct({ name: 'ExampleActor' })
class ExampleActor extends ShadoActor {
  @field('f32') heat!: number;
}

const HEAT_MATERIAL: ShadoInstanceGLSLHooks = {
  vertexInstance: `shadoColor.rgb *= mix(vec3(1.0), vec3(1.2, 0.5, 0.2), inst.heat);`,
};

class ExampleActors extends ShadoInstanceContainer<ExampleActor> {
  protected override getGLSLHooks() {
    return HEAT_MATERIAL;
  }
}

Hooks are available for vertex/fragment declarations, per-instance vertex setup, post-position logic, and final surface composition. Keeping each shader strategy in its own module makes it composable and avoids source search-and-replace.

Variant supermesh -> module draws

Models that ship every equipment variant as its own submesh and hide the unworn ones per instance skin the whole wardrobe for every actor and display a fraction of it, because the hiding happens after skinning. Cost scales with instances x total wardrobe vertices no matter how little is visible, which is why this layout tends to stall around a hundred actors.

splitMeshesIntoModules regroups those submeshes into one mesh per variant, and ShadoModuleDrawSet gives each one a compact actor list so it draws only the actors wearing it. The arena, the visibility pass, the actor records and the appearance array are untouched - only draw ownership moves.

import { splitMeshesIntoModules, ShadoModuleDrawSet } from '@knervous/shado';

const geometry = splitMeshesIntoModules(sourceMeshes, {
  groupKey: (mesh, index) => `${piece(index)}:${variant(index)}`,
  preserveAttributes: [{ kind: 'submeshData', stride: 2 }],
});

const draws = new ShadoModuleDrawSet(engine, geometry);
draws.registerThinInstanceAttribute('matrix', 16);

// once per frame, after visibility
const stats = draws.refresh(container.visibleActorIndices, showsModule);

// in each module material's per-draw bind, after binding the arena
mesh.forcedInstanceCount = draws.bindSelection(moduleIndex, effect);

A constant groupKey collapses everything into one module, which is byte-for-byte the supermesh you started with - land that first, then turn the real key on. refresh skips the upload for buckets that did not change and switches empty modules off, which also keeps them out of GPU picking. It returns submittedVertices / baselineVertices / vertexWorkReduction so a migration can be proven rather than asserted.

A real humanoid wardrobe (43 submeshes, 17 variants across 7 pieces) went from 9,406 vertices skinned per actor to 3,762. p95 frame ms on the same scene:

| actors | merged supermesh | module draws | | -----: | ---------------: | -----------: | | 1,000 | 18.25 | 18.36 | | 10,000 | 66.97 | 18.12 | | 20,000 | 134.98 | 30.03 |

Equivalent at low counts: this changes the slope, not the constant. See SUPERMESH_MODULE_MIGRATION.md for the step-by-step migration and its traps, and src/showcase/ShadoSupermeshModuleDemo.ts for a runnable slice. The sandbox route /hum-wardrobe runs the same pattern on a real 43-submesh humanoid at up to 10,000 actors, with an overlay that reports actor count and draw count side by side: both 256 actors and 10,000 actors draw 17 times.

PVS-reduced world lights

Authored point lights can independently opt into an offline bake, the dynamic runtime, or both. Runtime rows are stored as Shado SoA state and packed as two vec4 values (position/range and radiance/radius). They are not Babylon light objects and do not consume the engine's per-material light uniforms.

const coordinator = await ShadoWorldVisibilityCoordinator.create(world);
const lights = new ShadoWorldLightBuffer(scene, world, coordinator);

const frame = coordinator.reduceWorld(planes, camera);
lights.reduce(planes, frame, camera, { activePhaseMask: 0xffff_ffff });

const material = new ShadoMaterial(scene, mesh, atlas, actors, {
  worldLights: lights,
});

// Moving spell/fixture lights update one 32-byte GPU row.
lights.updateLight('spell-orb:light', { position: [x, y, z], intensity: 12 });

The complete world may contain far more lights than a draw evaluates. The WASM pass intersects PVS, cell policy, frustum, range, enabled state, and phase, then uploads only compact row indices. maxRuntimePointLights is therefore a quality/performance ceiling on the active list, not an authoring limit.

Headless video capture

Render a scene to a video file, or stream it live, with no browser process. Babylon's real WebGPU engine runs on Dawn in Node, so what is captured is the engine the game uses rather than an approximation.

Shado includes [email protected], the official dawn-gpu/node-webgpu binding. Its current Dawn build exposes optional features used by Babylon Lite, including primitive-index for detailed GPU picking. Linux CI hosts need a Vulkan loader and implementation; on Ubuntu install libvulkan1 and mesa-vulkan-drivers before starting a headless session.

npm install @knervous/shado @ffmpeg-installer/ffmpeg

# a turntable of a model
npx shado-video --input model.glb --out spin.mp4 --seconds 6 --materials

# a scene script: animations, rigs, a camera on a path
npx shado-video --script my-scene.mts --out clip.mp4 --materials --gif

The encoder is a package dependency, never something found on PATH, but it is an optional peer — it ships ~35MB of prebuilt binaries and video is one of twenty subpaths, so installing it is a deliberate step. The tools check for it before doing any work and name the install command if it is missing.

A capture is three seams that know nothing about each other: a FrameSource produces frames, a ShadoVideoEncoder compresses them into fragmented MP4, and a VideoSink carries the bytes.

import { createSessionFrameSource, orbitCamera, renderVideo } from '@knervous/shado/video';
import { createFfmpegEncoder, createFileSink } from '@knervous/shado/video/node';
import { createPreviewSession } from '@knervous/shado/devtools';

const session = await createPreviewSession({ width: 1280, height: 720 });
await session.loadGlb(bytes);
const camera = await session.frameCamera({ zoom: 2.4 });

await renderVideo({
  source: await createSessionFrameSource(session, {
    width: 1280,
    height: 720,
    camera,
    onFrame: orbitCamera(camera, { seconds: 6 }),
  }),
  encoder: createFfmpegEncoder(),
  sink: createFileSink('spin.mp4'),
  seconds: 6,
  fps: 30,
});

Because the container is fragmented MP4, the same byte stream is a valid file and is playable as it arrives — so swapping createFileSink for createHttpSink, createWebSocketSink or createDataChannelSink changes one argument. pacing: 'realtime' paces to the wall clock for streaming a running app; the default runs as fast as the renderer manages and still produces exact timestamps.

Frames are converted to yuv420p in a compute shader and read back pipelined, which together took a 1440p capture from 2.8s to 1.4s — see VIDEO_CAPTURE.md for the measurements, the scene-script API, transports, and the GIF and colour-management notes.

npm run demo:live then http://localhost:8787 is a runnable example: Babylon renders headless on the server, encodes in real time, and streams into a plain <video src="/live.mp4">. The page contains no script at all.

Package entry points

• @knervous/shado — schemas, arenas, backings, decorators, Babylon helpers, and actor instancing.
• @knervous/shado/babylon — Babylon peer exports and resolution helpers.
• @knervous/shado/core — renderer-neutral schemas, arenas, and packed runtime.
• @knervous/shado/lite — native Babylon Lite storage and instanced rendering.
• @knervous/shado/renderer — typed renderer and feature gates.
• @knervous/shado/render-data — projected actor codecs, upload plans, and compute-scatter WGSL.
• @knervous/shado/storage — bounded deferred-storage slabs backed by OPFS.
• @knervous/shado/asc — optional AssemblyScript compilation helpers.
• @knervous/shado/msdf — MSDF shader registration and nameplate helpers.
• @knervous/shado/render — lean dynamic-entity containers, renderers, atlases, reducers, and picking.
• @knervous/shado/preprocess — Node-side model and shader preprocessing.
• @knervous/shado/preprocess/runtime — browser-safe artifact loading and decompression.
• @knervous/shado/world — authored/compiled world contracts, PVS reduction, mutable light SoA state, collision, and runtime package loading.
• @knervous/shado/devtools — Node-only headless Babylon on Dawn: preview sessions, model/scene renders, pipelined GPU capture, RGBA-to-yuv420p conversion, configurable glTF loaders, PNG output, and pipeline comparison.
• @knervous/shado/video — runtime-neutral video capture: frame sources, the capture driver, WebCodecs encoding, and HTTP/WebSocket/WebRTC sinks.
• @knervous/shado/video/node — the ffmpeg encoder, file sink, and GIF output.

Examples and development

• sandbox/ is the full React/Vite validation app. It covers WebGL/WebGPU, DQ/VAT actor rendering, preprocessed assets, WASM reducers, picking, MSDF nameplates, and lean dynamic entities. Its demo content is only partly redistributable: /hum-wardrobe and /supermesh-scale run from a clean clone, while /, /msdf, /world, /world-editor and /test need assets that are not published here. See NOTICE.md for what is bundled, under which terms, and what is missing.
• playground/ contains a paste-ready Babylon.js Playground example that imports the npm package and downloads its model/VAT artifacts from raw GitHub URLs.

npm install
npm run typecheck
npm test
npm run build

cd sandbox
npm install
npm run build
npm run dev

See RELEASE_NOTES.md for release history and upgrade notes.

License

MIT