babylon-mmd

mmd loader and runtime for Babylon.js

Screenshots

fig3.mp4

Music: ピチカートドロップス

Model: YYB Piano dress Miku

Stage: ガラス片ドーム

Motion: https://www.nicovideo.jp/watch/sm31508557

Camera: https://www.nicovideo.jp/watch/sm36273873

https://playground.babylonjs.com/#028YR6#1

Music: メランコリ・ナイト

Model: YYB Hatsune Miku_10th

Motion / Camera: https://www.nicovideo.jp/watch/sm41164308

How to use

First, install the package.

npm add @babylonjs/core @babylonjs/havok babylon-mmd

Here is the code to build a scene with a simple MMD model and play a VMD animation.

// side effects that register the loader
import "babylon-mmd/esm/Loader/pmxLoader";

// side effects that register the animation runtime
import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeCameraAnimation";
import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeModelAnimation";

async function build(canvas: HTMLCanvasElement, engine: Engine): Scene {
    // If you don't want full SDEF support on shadow / depth rendering, you can comment out this line as well. While using SDEF can provide similar results to MMD, it comes with a higher cost.
    SdefInjector.OverrideEngineCreateEffect(engine);

    const scene = new Scene(engine);

    const camera = new MmdCamera("mmdCamera", new Vector3(0, 10, 0), scene);

    const hemisphericLight = new HemisphericLight("light1", new Vector3(0, 1, 0), scene);
    hemisphericLight.intensity = 0.4;
    hemisphericLight.specular = new Color3(0, 0, 0);
    hemisphericLight.groundColor = new Color3(1, 1, 1);

    const directionalLight = new DirectionalLight("DirectionalLight", new Vector3(0.5, -1, 1), scene);
    directionalLight.intensity = 0.8;
    
    CreateGround("ground1", { width: 60, height: 60, subdivisions: 2, updatable: false }, scene);
    
    // Use havok physics engine for rigid body/joint simulation
    const havokInstance = await HavokPhysics();
    const havokPlugin = new HavokPlugin(true, havokInstance);
    scene.enablePhysics(new Vector3(0, -9.8 * 10, 0), havokPlugin);
    
    // MMD runtime for solving morph, append transform, IK, animation, physics
    const mmdRuntime = new MmdRuntime(new MmdPhysics(scene));
    mmdRuntime.register(scene);
    
    // For synced audio playback
    const audioPlayer = new StreamAudioPlayer();
    audioPlayer.source = "your_audio_path.mp3";
    mmdRuntime.setAudioPlayer(audioPlayer);
    
    // You can also run the animation before it loads. This will allow the audio to run first.
    mmdRuntime.playAnimation();

    // create a youtube-like player control
    new MmdPlayerControl(scene, mmdRuntime, audioPlayer);
    
    const vmdLoader = new VmdLoader(scene);

    mmdRuntime.setCamera(camera);
    const cameraMotion = await vmdLoader.loadAsync("camera_motion_1", "your_camera_motion_path.vmd");
    camera.addAnimation(cameraMotion);
    camera.setAnimation("camera_motion_1");

    const model = await SceneLoader.ImportMeshAsync(undefined, "your/root/path/", "your_file_name.pmx", scene).then((result) => result.meshes[0]);
    const mmdModel = mmdRuntime.createMmdModel(model);
    const modelMotion = await vmdLoader.loadAsync("model_motion_1", "your_model_motion_path.vmd");
    mmdModel.addAnimation(modelMotion);
    mmdModel.setAnimation("model_motion_1");

    return scene;
}

Use optimized custom format

The optimized custom formats, BPMX (BabylonPMX) and BVMD (BabylonVMD), handle most of the validation and parsing process required for loading at the conversion stage.

As a result, you can provide users with an incredibly fast loading experience.

Furthermore, it is also useful when you want to protect the original model/motion files since they cannot be loaded in MMD.

PMX to BPMX

To perform pmx conversion, please visit below link and load the desired models directory. Then select the model you want to convert and click on "convert".

pmx converter

then you can load the converted files like below.

// side effects that register the loader
import "babylon-mmd/esm/Loader/Optimized/bpmxLoader";

const model = await SceneLoader.ImportMeshAsync(undefined, "your/root/path/", "your_file_name.bpmx", scene).then((result) => result.meshes[0]);

VMD to BVMD

To perform vmd conversion, please visit below link and load the desired motions. Then, simply click on "convert". By default, it is possible to merge one camera motion and one model motion together.

vmd converter

then you can load the converted files like below.

const bvmdLoader = new BvmdLoader(scene);
const motion = async bvmdLoader.loadAsync("motion_1", "your_motion_path.bvmd");

Documentation

Please refer to the documentation for more detailed instructions on how to use it.

Documentation

Implementation status

Parser

PMX Loader

Mesh

Geometry load
Build Bone / MorphTarget

Material / Shading

VMD Loader

Basic animation load / custom retargeting
Load animation using the Babylon.js Animation container
Babylon.js Animation container Humanoid to MMD bone mapping

Animation Runtime

Physics Runtime

Solve Rigid body / Joint
WASM based runtime for threaded bullet physics and MMD IK

Not planned features

PMX 2.1 support (because 2.1 spec not implemented in MMD)
Self shadow, Ground shadow spec support

References

PMX (Polygon Model eXtended) 2.0, 2.1 File Format Specifications

blender_mmd_tools

Saba: OpenGL Viewer (OBJ PMD PMX)

three.js MMDLoader

vtange/babylon-mmd