- fork from original repository
- Install required libraries (in requirements.txt)
- Get preprocess.py and mfcc_rev.py from Obama lip thing
- Change StandardError to Exception for Python3
- instead of --comp_snapshot, just use --resume
PyTorch implementation of the cross-modality generative model that synthesizes dance from music.
Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, Jan Kautz
Dancing to Music
Neural Information Processing Systems (NeurIPS) 2019
[Paper] [YouTube] [Project] [Blog] [Supp]
- Beat-Matching
1st row: generated dance sequences, 2nd row: music beats, 3rd row: kinematics beats
- Multimodality
Generate various dance sequences with the same music and the same initial pose.
- Long-Term Generation
Seamlessly generate a dance sequence with arbitrary length.
- Photo-Realisitc Videos
Map generated dance sequences to photo-realistic videos.
python train_decomp.py --name Decomp
python train_comp.py --name Decomp --decomp_snapshot DECOMP_SNAPSHOT
python demo.py --decomp_snapshot DECOMP_SNAPSHOT --comp_snapshot COMP_SNAPSHOT --aud_path AUD_PATH --out_file OUT_FILE --out_dir OUT_DIR --thr THR
-
Flags
aud_path
: input .wav fileout_file
: location of output .mp4 fileout_dir
: directory of output framesthr
: threshold based on motion magnitudemodulate
: whether to do beat warping
-
Example
python demo.py --decomp_snapshot checkpoint/Stage1.ckpt --resume checkpoint/Stage2.ckpt --aud_path tigerinside.mp3 --out_file test.mp4 --out_dir out_frame
If you find this code useful for your research, please cite our paper:
@inproceedings{lee2019dancing2music,
title={Dancing to Music},
author={Lee, Hsin-Ying and Yang, Xiaodong and Liu, Ming-Yu and Wang, Ting-Chun and Lu, Yu-Ding and Yang, Ming-Hsuan and Kautz, Jan},
booktitle={NeurIPS},
year={2019}
}
Copyright (C) 2020 NVIDIA Corporation. All rights reserved. This work is made available under NVIDIA Source Code License (1-Way Commercial). To view a copy of this license, visit https://nvlabs.github.io/Dancing2Music/LICENSE.txt.