💡 We also have other video generation project that may interest you ✨.
Open-Sora-Plan
PKU-Yuan Lab and Tuzhan AI etc.
MagicTime
Shenghai Yuan, Jinfa Huang and Yujun Shi etc.
- ⏳⏳⏳ Evaluate more Text-to-Video Generation Models via ChronoMagic-Bench.
- ⏳⏳⏳ Release the code of the "Multi-Aspect Data Preprocessing", which is used to process the dataset. The code is being organized.
- ⏳⏳⏳ Support evaluating customized videos. The code and instructions are being organized.
[2024.06.28]
🔥 We released the ChronoMagic-Pro and ChronoMagic-ProH datasets. The datasets include 460K and 150K time-lapse video-text pairs respectively and can be downloaded at HF-Dataset-Pro and HF-Dataset-ProH.[2024.06.27]
🔥 We release the arXiv paper and Leaderboard for ChronoMagic-Bench, and you can click here to read the paper and here to see the leaderboard.[2024.06.26]
🔥 We release the testing prompts, reference videos and generated results by different models in ChronoMagic-Bench, and you can click here to see more details.[2024.06.25]
🔥 All codes & datasets are coming soon! Stay tuned 👀!
ChronoMagic-Bench can reflect the physical prior capacity of Text-to-Video Generation Model.
In contrast to existing benchmarks, ChronoMagic-Bench emphasizes generating videos with high persistence and strong variation, i.e., metamorphic time-lapse videos with high physical prior content.
Backbone | Type | Visual Quality | Text Relevance | Metamorphic Amplitude | Temporal Coherence |
---|---|---|---|---|---|
UCF-101 | General | ✔️ | ✔️ | ❌ | ❌ |
Make-a-Video-Eval | General | ✔️ | ✔️ | ❌ | ❌ |
MSR-VTT | General | ✔️ | ✔️ | ❌ | ❌ |
FETV | General | ✔️ | ✔️ | ❌ | ✔️ |
VBench | General | ✔️ | ✔️ | ❌ | ✔️ |
T2VScore | General | ✔️ | ✔️ | ❌ | ❌ |
ChronoMagic-Bench | Time-lapse | ✔️ | ✔️ | ✔️ | ✔️ |
We specifically design four major categories for time-lapse videos (as shown below), including biological, human-created, meteorological, and physical videos, and extend these to 75 subcategories. Based on this, we construct ChronoMagic-Bench, comprising 1,649 prompts and their corresponding reference time-lapse videos.
We visualize the evaluation results of various open-source and closed-source T2V generation models across ChronoMagic-Bench.
See numeric values at our Leaderboard 🥇🥈🥉
We recommend the requirements as follows.
git clone https://github.com/PKU-YuanGroup/ChronoMagic-Bench.git
cd ChronoMagic-Bench
conda create -n chronomagic python=3.10
conda activate chronomagic
# install base packages
pip install -r requirements.txt
# install flash-attn
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/csrc/layer_norm && pip install .
cd ../../../
rm -r flash-attention
- This project wouldn't be possible without the following open-sourced repositories: CoTracker, InternVideo2, UMT, FETV, VBench, ShareGPT4Video and LAION Aesthetic Predictor.
- The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
- The service is a research preview. Please contact us if you find any potential violations.
If you find our paper and code useful in your research, please consider giving a star ⭐ and citation 📝.
@misc{yuan2024chronomagicbenchbenchmarkmetamorphicevaluation,
title={ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation},
author={Shenghai Yuan and Jinfa Huang and Yongqi Xu and Yaoyang Liu and Shaofeng Zhang and Yujun Shi and Ruijie Zhu and Xinhua Cheng and Jiebo Luo and Li Yuan},
year={2024},
eprint={2406.18522},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2406.18522},
}