@driftengine/capture

A video to a playable scene, on the player's own device — no service, no account, and a runtime that never fetches.

66,373 bytes gzipped, measured by scripts/size-gate.test.mjs against scripts/fixtures/size/capture-only.ts. Optional: nothing in @driftengine/core imports it.

What is here so far

The models a capture runs, each written as a definition: a function of a model's weights that @driftengine/texture's graphFromWeights builds into a graph the engine's one neural runtime runs, on the CPU as a reference or on the device through @driftengine/core's createGraphRunner.

| Export | What it is | | -------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | depthAnything3(config, views, h, w) | Depth Anything 3 at views images of h × w pixels: logits per view and a pose encoding for all of them | | DEPTH_ANYTHING_3.small, .base | The two accepted sizes, from each checkpoint's configuration at its pinned revision | | depthAnything2(config, h, w), DEPTH_ANYTHING_2_SMALL | Depth Anything V2 Small, the lighter fallback: relative inverse depth from one view, in Transformers' layout | | decodeDepth, decodeCamera | The logits as depth and a confidence of at least one; a pose encoding as world-to-camera and a pinhole | | mobileSamEncoder(config), mobileSamDecoder(config, tokens, refine?), MOBILE_SAM | MobileSAM in two graphs: an image to its embedding once, and a prompt's tokens — and the last mask, to refine — to four masks and their predicted quality | | samPromptTokens, samTokenCount, samMasksToImage, stabilityScore | A prompt of points and a box as the decoder's tokens; the masks brought to the original image; the upstream's stability score | | sam21Encoder(config), sam21Decoder(config, tokens, options), Sam21Tracker, SAM_21_TINY | SAM 2.1: masks from a point or a box on an image, and objects tracked through a video by a memory of the frames before them, the loop on the host and its networks as graphs | | owlv2Image(config), owlv2Text(config, queries), owlv2Logits, owlv2Detections, OWLV2_BASE | OWLv2: an image's patches once, a set of text queries once, joined on the host into a logit a patch and query and those into boxes above a threshold | | clipTokenizer(merges), owlv2Tokens | CLIP's byte-level tokenizer from the merges a converted file carries, and queries padded as the text graph takes them | | eachFrame(source, visit), selectFrames(source, { budget, every? }), browserFrameSource(clip) | A clip through the host's own FrameSource, read one frame at a time; the frames a capture keeps, spaced by equal motion; a browser's video element behind one | | prepareDepthFrame, prepareSamFrame, prepareSam2Frame, prepareOwlv2Frame | A frame as each model's own upstream prepares it, down to the resize each library uses | | createDepthEstimator(config, weights, run) | A clip's frames to depth, confidence and a camera a view, a graph kept per prepared size | | svd3, svdN, symmetricEigen, cholesky, levenbergMarquardt, schurSolve | The decompositions structure from motion needs, in double precision, each held to its own identity | | detectFeatures, describeFeatures, matchFeatures | Corners, a description of each turned by its own orientation, and the matches between two frames — mutual, and unambiguous from both sides | | relativePose(a, b, count, intrinsics, random, out), triangulate(...) | The pose between two views by seeded RANSAC over both models, and the points behind the matches; a pair with no parallax is reported as a turn rather than fitted with a translation | | renderTestScene(scene, camera, out, depth?), lookAt | Analytic scenes ray cast on the CPU, with the true depth beside the colour: the fixture every later stage is held to | | estimatePoses(frames, options, out) | The clip's camera path: tracks, a two-view or depth-model start, every frame placed by what it sees, and a bundle adjustment — with per-pose confidence, parallax: false where the clip only turned, and a scale that is metric only when a length was given | | optimiseGaussians(targets, poses, options) | A clip's frames and its camera path fitted into Gaussians, out as @driftengine/splats' own SplatSource: Adam over six families, scale as a logarithm and opacity as a logit, densified by which Gaussians the frames pull on and pruned by what has faded, with a degree-1 band on request | | rasteriseGaussians(set, camera, out), projectGaussian, accumulateGradients, imageLoss | The reference rasteriser the fit descends: the projection @driftengine/splats' shader performs, low-pass included, and every analytic gradient behind it — checked against a central difference of itself | | splatColour, sh1Basis, SH_C1 | The degree-1 band, evaluated exactly as that shader evaluates it, so a capture fitted here is shaded there the same way | | structuralSimilarity(a, b, width, height) | How alike two pictures are, structure included — what a fit is judged by, where a squared difference is only what it descends | | renderDepth(set, camera, depth, coverage) | How far away a cloud is, a pixel at a time: each splat's own depth weighted by how much of the pixel it lit, with the coverage beside it — because a depth averaged out of a tenth of a pixel's light is not a measurement | | createVolume(dims, origin, spacing), fuseDepth(views, volume) | Many views averaged into one truncated signed distance, weighted by how squarely each saw the surface. Space in front of a surface is carved out, space behind it is left alone, and a surface one view saw is kept at the weight one view is worth | | marchVolume(volume, options?) | That volume's crossing as a mesh, by tetrahedra rather than cubes so it is closed by construction, facing outwards — and open where the capture never looked, rather than capped with a surface nobody measured | | decimate(mesh, budget, options?) | The mesh brought to a triangle budget by quadric-error collapse: a wall flattens, a corner keeps its detail, the border of an open capture is held, and a collapse that would turn a face inside out is refused | | collisionMesh(mesh, options?) | The drawn mesh cleaned for collision: triangles with no area taken out, because a plane needs a normal, and the vertices nothing uses any more taken with them. What comes back is what meshShape takes | | propHulls(mesh, options?) | A prop as convex hulls, asked for inside COLL's own caps rather than checked against them after the expensive part — a room is a mesh body and a prop is hulls, and the difference is that one can be thrown | | delight(views, mesh, options, out) | A surface's material, separated from the light it was photographed under: what stays the same across views is the material, what moves is the light, and an intensity jump the colour does not follow is a shadow. Every vertex carries a confidence, and the module states plainly what that confidence grades and what it does not — on the synthetic scene a cast shadow that darkens a floor 4.8× in the photographs comes back 2.2× in the albedo | | renderLit(scene, camera, light, out), nearestHit | The lit fixture those measurements are made against: a triangle mesh, one directional light, a hard shadow ray and a normalised specular lobe — enough to pose the question and no more | | segmentGeometry(mesh, options?) | The mesh cut into surfaces with no model loaded at all: runs of triangles that share an edge and a plane, largest first. A floor, its walls and the top of a table are findable without knowing what any of them is, and that is most of what a scene needs | | liftMasks(views, mesh, labels), promptGrid(width, height, across, down, random) | A segmentation model's masks voted onto the mesh across every view that saw a triangle, and the seeded lattice it is prompted on — a triangle no view claimed is left unassigned rather than given to its neighbour | | labelMasks(view, detections, queries, overlap?) | A detector's boxes put names to a segmenter's mask indices, by how far each box and mask agree — the mask's own pixels against the box, so a box drawn around a room does not name every cushion in it. A mask that agrees with nothing keeps no name, and no name is scenery | | proposeEntities(regions, options?), SCENERY_COMPONENTS, WALKABLE_COMPONENTS, MOVABLE_COMPONENTS | Each region as a proposal, and the word is load-bearing: a region nobody labelled becomes scenery — drawn, solid, and doing nothing. The largest surface a character could stand on gets a navigation surface; a named region small enough to shift is offered as movable | | captureFile(scene), proposalScene, readProposals, PROPOSAL_COMPONENT | Every stage as one .drft: the mesh, its materials and decode program, the cloud, a prop's hulls, the way across the scene and the proposals. The proposals travel under a component this package declares, not under a consumer's transform, whose field ids belong to that consumer — a capture proposes in its own words and an editor decides what a scene ends up holding | | miniatureCheckpoint, miniatureCheckpoint2, miniatureImages, miniatureSamCheckpoint, miniatureSam21Checkpoint, miniatureOwlv2Checkpoint | Seeded miniatures of each layout — every tensor name and shape — for tests and for checking a port of yours |

A definition runs twice. Once over a checkpoint, when tools/capture-weights/ converts it, and again over the converted file whenever a clip needs the graph at a new size: every device kernel bakes its shapes, and a clip's aspect ratio decides the patch grid. Every weight the source holds is read or set aside by name, so a forgotten bias is refused rather than run.

Each model answers as its upstream's own code does. The tests hold each definition to numbers the upstream's modules wrote for its miniature at the manifest's pinned revision — Depth Anything 3's own repository, Transformers for V2, SAM 2.1 and OWLv2, and MobileSAM's own repository — within a few parts in a million; tools/capture-weights/reference/ is the hand-run oracle that produced them.

Weights

None ship with this package. tools/capture-weights/manifest.json pins each accepted model to a commit and a SHA-256, and docs/SHIPPING.md §12 says what a game ships with one: the converted file, the model's licence, and its attribution, which NOTICE carries too.