mljs

This is a JS library that provides ML capabilities on browsers using ONNX runtime web.

Features

Below is a list of features available:

1. Spatial image

Spatial image

This lets you convert images into spatial image on browsers. To do this, you need to embed your image element into a div element:

<div id="target_div">
    <img src="/your/image/src">
</div>

and simply pass the div element to the library:

const div_container = document.getElementById("target_div");
spatial_scene.convert_single_image(div_container, 4);

To achieve this, there are two models used:

1. Depth Anything V2
2. MI-GAN

The ONNX binaries for these models can be found at this HuggingFace repository

Project structure

.
├── dist/
│   ├── index.js                          // Code generated by npm
│   └── ort-wasm-simd-threaded.jsep.wasm  // The ONNX runtime web WASM binary. This is copied from ONNX runtime NPM module.
├── src/
│   ├── demo                              
│   │   └── spatial-image                 // A directory containing a simple demo for the spatial scene feature
│   │       ├── index.html
│   │       └── script.js
│   ├── docs                            // Docs for each feature
│   ├── features                        // Contains the wrapper code that calls `models/` & `ui/`. Each feature such as spatial image is implemented here.
│   │   └── spatial-scene.ts
│   ├── models                          // Handles model inference.
│   │   ├── base-model.ts
│   │   ├── depth-estimation.ts
│   │   └── inpaint.ts
│   ├── package_loader                  // TODO.
│   │   └── TODO
│   ├── ui                              // Contains all UI-related code. For example, the code that animates the images from spatial image is written here.
│   │   └── animation.ts
│   └── utils                          // Just some utility functions....
│       ├── ort_tensor_manipulator.ts
│       └── type_converter.ts
├── index.ts                          // The file that esbuild refers to when building the package.
├── package.json
└── tsconfig.json

Building the package

To build the package, simply run npm run build

Dependencies

TODO

How to use the library for testing purposes

Currently, as package_loader hasn't been implemented yet, you need to do the following to use the library:

1. Create a directory called model_binaries in the root directory & put the models there.
2. Run the browser with web security disabled. For Chrome:

# On Windows
"C:\Program Files\Google\Chrome\Application\chrome.exe" --disable-web-security --disable-gpu --user-data-dir="C:\chromeTemp" --enable-features=SharedArrayBuffer

# On Linux
google-chrome --disable-web-security --user-data-dir="/temp/dev/" --enable-features=SharedArrayBuffer