babylon-mmd

Babylon.js loader and runtime for MikuMikuDance (MMD) models and animations. It supports PMX/PMD model formats, VMD/VPD animation formats, and provides a runtime for physics, IK, morphs, and more.

Playground Demo

Music: メランコリ・ナイト

Model: YYB Hatsune Miku_10th

Motion / Camera: https://www.nicovideo.jp/watch/sm41164308

https://github.com/user-attachments/assets/64207280-560a-49b5-ba50-bba7959dc79a

Music: ピチカートドロップス

Model: YYB Piano dress Miku

Stage: ガラス片ドーム

Motion: https://www.nicovideo.jp/watch/sm31508557

Camera: https://www.nicovideo.jp/watch/sm36273873

Documentation

Please refer to the documentation for detailed instructions on how to use it.

Usage

Here is how to load and animate an MMD model using babylon-mmd. For more details, please refer to the Documentation.

// side effects that register the loader
import "babylon-mmd/esm/Loader/pmxLoader";

// side effects that register the animation runtime
import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeCameraAnimation";
import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeModelAnimation";

async function build(canvas: HTMLCanvasElement, engine: Engine): Scene {
    // If you don't want full SDEF support on shadow / depth rendering, you can comment out this line as well. While using SDEF can provide similar results to MMD, it comes with a higher cost.
    SdefInjector.OverrideEngineCreateEffect(engine);

    const scene = new Scene(engine);
    scene.ambientColor = new Color3(0.5, 0.5, 0.5);

    const camera = new MmdCamera("mmdCamera", new Vector3(0, 10, 0), scene);

    const directionalLight = new DirectionalLight("DirectionalLight", new Vector3(0.5, -1, 1), scene);
    directionalLight.intensity = 1.0;
    
    CreateGround("ground1", { width: 60, height: 60, subdivisions: 2, updatable: false }, scene);
    
    const mmdWasmInstance = await GetMmdWasmInstance(new MmdWasmInstanceTypeMPR());
    const physicsRuntime = new MultiPhysicsRuntime(mmdWasmInstance);
    physicsRuntime.setGravity(new Vector3(0, -98, 0));
    physicsRuntime.register(scene);
    
    // MMD runtime for solving morph, append transform, IK, animation, physics
    const mmdRuntime = new MmdRuntime(scene, new MmdBulletPhysics(physicsRuntime));
    mmdRuntime.register(scene);
    
    // For synced audio playback
    const audioPlayer = new StreamAudioPlayer(scene);
    audioPlayer.source = "your_audio_path.mp3";
    mmdRuntime.setAudioPlayer(audioPlayer);
    
    // You can also run the animation before it loads. This will allow the audio to run first.
    mmdRuntime.playAnimation();

    // create a youtube-like player control
    new MmdPlayerControl(scene, mmdRuntime, audioPlayer);
    
    const vmdLoader = new VmdLoader(scene);

    const cameraAnimation = await vmdLoader.loadAsync("camera_motion", "your_camera_motion_path.vmd");
    const cameraRuntimeAnimationHandle = camera.createRuntimeAnimation(cameraAnimation);
    camera.setRuntimeAnimation(cameraRuntimeAnimationHandle);
    mmdRuntime.addAnimatable(camera);

    const assetContainer = await LoadAssetContainerAsync("path/to/your_file.pmx", scene);
    assetContainer.addAllToScene();
    const mmdMesh = assetContainer.meshes[0] as MmdMesh;

    const mmdModel = mmdRuntime.createMmdModel(mmdMesh, {
        materialProxyConstructor: MmdStandardMaterialProxy
    });
    const modelMotion = await vmdLoader.loadAsync("model_motion", "your_model_motion_path.vmd");
    const modelRuntimeAnimationHandle = mmdModel.createRuntimeAnimation(modelMotion);
    mmdModel.setRuntimeAnimation(modelRuntimeAnimationHandle);

    return scene;
}

Implementation status

Parser

• [x] PMX Parser
• [x] PMD Parser
• [x] VMD Parser
• [x] VPD Parser
• [x] Optimized Custom PMX like format
• [x] Optimized Custom VMD like format

PMX / PMD Loader

Mesh

• [x] Geometry load
• [x] Build Bone / MorphTarget

Material / Shading

• [x] Basic material parameters load
• [x] Custom MMD toon fragment shader
• [x] Spherical Deformation support
• [x] WebGL 1.0 / 2.0 support
• [x] WebGPU support

VMD / VPD Loader

• [x] Basic animation load / custom retargeting
• [x] Load animation using the Babylon.js Animation container
• [x] Babylon.js Animation Humanoid to MMD bone retargeting

Animation Runtime

• [x] Basic animation load
• [x] MMD morph system support
• [x] Solve Append transform
• [x] Solve IK
• [x] Play audio / sync with animation
• [x] Basic animation player UI
• [x] Animation blending in custom animation runtime
• [x] Force Humanoid Model Support

Physics Runtime

• [x] Solve Rigid body / Joint
• [x] WASM based runtime for threaded bullet physics and MMD IK

Not planned features

• PMX 2.1 support (because 2.1 spec not implemented in MMD)
• Self shadow, Ground shadow spec support

Support this project

Advanced technical support is available for those supporting the project

Contact me via email or discord

email: [email protected]

discord: noname0310

References

PMX (Polygon Model eXtended) 2.0, 2.1 File Format Specifications

blender_mmd_tools

Saba: OpenGL Viewer (OBJ PMD PMX)

three.js MMDLoader