Please visit our project page for more details.
📢 July 27, 2024: First release - motion transfer of generated or inverted motions, using a pretrained model. Coming soon: visualization of deep features, evaluation.
This code has been tested in the following environment:
- Ubuntu 20.04.6 LTS
- Python 3.8
- conda3 or miniconda3
- CUDA capable GPU (one is enough)
Setup conda env:
conda env create -f environment.yml
conda activate MoMo
pip install git+https://github.com/openai/CLIP.gitData should be located in the ./dataset folder.
We use the HumanML3D dataset. If you are only doing inference, a lightweight version is sufficient. The lightweight version saves disk space and is quicker to get.
Clone HumanML3D, then link the data folder to our repository:
cd ..
git clone https://github.com/EricGuo5513/HumanML3D.git
unzip ./HumanML3D/HumanML3D/texts.zip -d ./HumanML3D/HumanML3D
ln -s ../../HumanML3D/HumanML3D dataset/HumanML3D # adjust "../../" if neededFollow the instructions in HumanML3D, then link the result folder to our repository:
ln -s ../../HumanML3D/HumanML3D dataset/HumanML3D # adjust "../../" if neededDownload the pretrained model using the script below. The model will be saved in the ./save/ directory.
bash prepare/download_model.shTo generate motions using a pretrained model use the following command:
python -m sample.transfer --model_path ./save/official_model/model000500000.pt --text_leader "a person is walking" --text_follower "a person walks with hands high above his head" "a person stands, then walks like a gorilla" --num_repetitions 1 --assign_root_rot If you want to use real motions via inversion, make sure you have the full version of the dataset and use the following arguments:
--leader_motion_path dataset/HumanML3D/new_joint_vecs/006473.npy --follower_motion_path dataset/HumanML3D/new_joint_vecs/013122.npy Argument values are given as examples.
Where
--text_leader: the text controlling the generation of the leader motion.--text_follower: the text controlling the generation of the appearnce motions (one or more). The number of generated follower motions will match the number of given text controls.--leader_motion_path: a path to a motion to be used as a leader. Overrides--text_leaderif both are given.--leader_motion_path: paths to a motions to be used as followers (one or more). Overrides--text_followerif both are given.--num_repetitions: the number of instances of leader+followers to be generated. For each instance, a separte transfer is applied.
The rest of the arguments, and many others, can be found in utils/sample_utils.py.
Additional useful arguments:
--seed: a seed for random generation.--device: an index of a specific GPU to be used.--motion_length: motion length in seconds. Network has trained on up to 9.8 seconds.
Result files:
results.npyfile with xyz positions of the generated animationtransfer_samples_00_to_##.mp4- stick figure animations of all leader and output motions. The leftmost motion is the leader, followed by the output transferred results on its right. Each row represents a different repetition.follower_samples_00_to_##.mp4- stick figure animations of all follower motions. Each row represents a different repetition.
Example of file transfer_samples_00_to_##.mp4:
Example of file follower_samples_00_to_##.mp4:
🔜 Coming soon
🔜 Coming soon
The following MDM functionalities can be used as in the MDM repository.
Follow these instructions under the Text to Motion detail. For example:
python -m sample.generate --model_path ./save/official_model/model000500000.pt --num_repetitions 3 --text_prompt "a person is dancing wildly" --motion_length 8🔜 Coming soon