God Mode Animation: 2D Game Animation Generation Model

I tried to train text to video and image to video models to generate 2D game animations. I used public game animation data and 3D mixamo model rendered animations to train the animation generation models. I'm open sourcing the model, training data, training code and data generation code. More details can be found in my blog here.

text to animation, image to animation

text to game based on animation model

You can try the games here

a dino jumping cacti in the desert

Donald Trump jumping trash can in new york city

Harry Potter jumping tree in Hogwarts castle

Tylor Swift jumping microphone in hotel room

Trained Models

Motion	Epochs	Steps	Model Type	HuggingFace Model
Sword Wield	36	2035	VC2 T2V	model
Sword Wield	38	2145	VC2 T2V	model
Spin Kick	32	1815	VC2 T2V	model
Spin Kick	34	1925	VC2 T2V	model
Run Jump	30	3379	VC2 T2V	model
Run Jump	34	3815	VC2 T2V	model
Run	19	1080	DC I2V	model

Training DataSet

Motion	Model Type	Data
Sword Wield	VC2 T2V	link
Spin Kick	VC2 T2V	link
Run Jump	VC2 T2V	link
Run	DC I2V	link

How to Train I2V Model

Clone the repository.

git clone https://github.com/lyogavin/godmodeanimation.git

Install the necessary dependencies.

cd godmodeanimation/i2v
pip install -r requirements.txt

Prepare the datasets by unzipping the files and placing them in the /root/ucf_ds directory.
Download the pretrained model in the /root/vc2/model.ckpt directory.

Run the training script.

train_t2v_run_jump.sh # for rum jump model
train_t2v_spinkick.sh # for spin kick model
train_t2v_sword_wield.sh # for sword wield model

How to Train DC I2V Model

DC I2V model is based on DynamiCrafter. Please follow the instructions in the DynamiCrafter repository to train the DC I2V model.

Render Mixamo Animation as 2D Game Animation Videos

Install Python blender: See here for details.

Render the animation:

cd godmodeanimation/data_generation
python render_mixamo_videos.py # --help to see more options

Replicate Public Model

We created Replicate public models for Sword Wield, Spin Kick, Run Jump and Run. You can try it from Replicat platform, find the details in Replicate.

Citing God Mode Animation

If you find This work useful in your research and wish to cite it, please use the following BibTex entry:

@software{godmodeanimation2024,
  author = {Gavin Li},
  title = {2D Game Animation Generation: All you need is repeat the same motion 1000 times},
  url = {https://github.com/lyogavin/godmodeanimation/},
  version = {1.0},
  year = {2024},
}

Contribution

Welcome contributions, ideas and discussions!

If you find this work useful or interesting to you, please ⭐ or buy me a coffee! It's very important to me! 🙏

pfisher/godmodeanimation