/SysMocap

A real-time motion capture system for 3D virtual character animating.

Primary LanguageJavaScriptMozilla Public License 2.0MPL-2.0

banner

SysMocap

GitHub Actions release forks stars

English Version | 中文版本

A cross-platform real-time video-driven motion capture and 3D virtual character rendering system for VTuber/Live/AR/VR.

Available for Windows, macOS (packaged) & Linux (source code only)

Download Now (zipped, without installation)

(This is a multi-language software, including English. )

Highlights

🌟 Beautiful GUI with Material Design 3 Color System

UI

🌟 Easy to used. You can import 3D models just with drags.

add-model-drag.mp4

🌟 Support WebXR API on Mocap Forwarding (HTTPS only)

webxr-ar-demo.mp4

🌟 Model viewer with bones & dressing controller

Model viewer

🌟 Support OBS live-streaming

OBS

🌟 Support full-body motion capture

Full-body

Full-body-animotion

🌟 Support Auto Skeleton Detection for All VRM files and Mixamo Format FBX files

fbx-mixamo-animotion

🌟 Support Any Skeleton Structure with Manual Mapping

bdd-animotion

🌟 Does not require a discrete graphics card and runs smoothly even on eight-year-old computers (i7-4790k/GTX770/16G RAM)

🌟 Powered by Mediapipe and Kalidokit with Web Technologies

More Effect Demonstration

🌟 Facial

Facial-animotion

🌟 Half-body

Half-body-animotion

🌟 Half-body with Hands

Half-body with Hands animotion

🌟 Full-body

Full-body-animotion

System architecture

System architecture

How to use

Run on your computer from source code (need lastest Node.js):

git clone https://github.com/xianfei/SysMocap.git
cd SysMocap
npm i
npm start

Bugs

  • You tell me

Notice

  1. HTTP & HTTPS will use same port in Mocap Data Forward.

Required Skeleton Node in glTF/glb/FBX Model File for Macap:

(If not same as above, you need rebind them manually.)

  • Hips (Main Node, both Position and Rotation. Ratation only for other nodes)

  • Neck

  • Chest

  • Spine

  • RightUpperArm

  • RightLowerArm

  • LeftUpperArm

  • LeftLowerArm

  • LeftUpperLeg

  • LeftLowerLeg

  • RightUpperLeg

  • RightLowerLeg

Star History

Star History Chart

Development progress

To-Do

  • Settings page and global settings utils

  • Add play/pause button and progress bar when mocap from video

  • Support bones binding for glTF/glb

  • Support rendering glTF/glb model

  • Support binding when bones' name is non-uniformed

  • Model library add user's custom 3D model

  • Live plug-in / interface for Open Broadcast Software

  • Output video ( using such as libffmpeg )

  • Support per-frame rendering without drop frame

  • Support c-s architecture for online video mocap ( on cloud )

  • Support Material Designed 3 Color System (color picking)

  • Mocap data forwarding via network

  • Adapt for Linux and macOS

Thanks

You can see all dependencies in package.json

Cite

@INPROCEEDINGS{9974484,
  author={Song, Wenfeng and Wang, Xianfei and Gao, Yang and Hao, Aimin and Hou, Xia},
  booktitle={2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)}, 
  title={Real-time Expressive Avatar Animation Generation based on Monocular Videos}, 
  year={2022},
  volume={},
  number={},
  pages={429-434},
  doi={10.1109/ISMAR-Adjunct57072.2022.00092}}