/EvalCrafter

[CVPR 2024] EvalCrafter: Benchmarking and Evaluating Large Video Generation Models

Primary LanguageJupyter Notebook

EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024) ๐ŸŽฅ๐Ÿ“Š

Project Page   ยท   Leaderboard   ยท   Gallery   ยท   Paper@ArXiv   ยท   EvalBoard

ECTV Dataset   ยท   Human Feedback   ยท   Prompt List

Welcome to EvalCrafter, a comprehensive evaluation toolkit for AI-generated videos. Our innovative framework assesses generative models across visual, content, and motion qualities using 17 objective metrics and subjective user opinions, providing a reliable ranking for state-of-the-art text-to-video generation models. Dive into the world of unified and user-centric evaluation with EvalCrafter! ๐Ÿš€๐ŸŒ๐Ÿ“Š

๐Ÿ”ฅ 2023/10/22: Release prompt list at Prompt List! You can generate the resulting video and send it to vinthony@gmail.com for evaluation!

๐Ÿ”ฅ 2024/01/10: Code and docker released!

๐Ÿ”ฅ 2024/01/24: EvalCrafter Text-to-Video (ECTV) dataset released, around 10000 AI-generated videos in total, take a look at here!

๐Ÿ”ฅ 2024/03/07: Dive into the exciting world of AI-Created Videos at our Gallery ๐ŸŒŒโœจ!

๐Ÿ”ฅ 2024/04/07: Release the code of EvalBoard ๐ŸŒŒโœจ for video comparison !

๐Ÿ”ฅ 2024/04/11: Release the Human Feedback data, 8.6k feedback, cover 5 aspects like visual quality and t2v alignment! ๏ผ๐Ÿš€๐Ÿ”ฅ

๐Ÿ”ฅ 2024/08/01: Environment setup using Conda is Supported๏ผ๐Ÿ›ซ๐Ÿค˜

AI-Created Video Gallery

๐Ÿ”† Join our Discord to enjoy free text-to-video generation and more: Discord

๐Ÿ”† Watch our project for more details and findings.

Installation ๐Ÿ’ป

Clone the repository:

git clone https://github.com/evalcrafter/EvalCrafter
cd EvalCrafter

Data Preparation ๐Ÿ“š

Generate videos of your model using the 700 prompts provided in prompt700.txt or ./prompts and organize them in the following structure:

/EvalCrafter/videos
โ”œโ”€โ”€ 0000.mp4
โ”œโ”€โ”€ 0001.mp4
โ”œโ”€โ”€ 0002.mp4
โ”œโ”€โ”€ 0003.mp4
โ”œโ”€โ”€ 0004.mp4
...
โ””โ”€โ”€ 0699.mp4

Pretrained Models ๐Ÿง 

Please download all checkpoints using

cd checkpoints
bash download.sh

Alternatively, you can follow ./checkpoints/README.md to download pretrained models for specific metrics.

Note: Please organize the pretrained models in this structure:

/EvalCrafter/checkpoints/
โ”œโ”€โ”€ bert-base-uncased
โ”œโ”€โ”€ blip2-opt-2.7b
โ”œโ”€โ”€ ckpt
โ”œโ”€โ”€ clip-vit-base-patch32
โ”œโ”€โ”€ Dover
โ”œโ”€โ”€ FlowNet2_checkpoint.pth.tar
โ”œโ”€โ”€ pt_inception-2015-12-05-6726825d.pth
โ”œโ”€โ”€ RAFT
โ”œโ”€โ”€ stable-diffusion-xl-base-1.0
โ”œโ”€โ”€ tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth
โ”œโ”€โ”€ vgg_face_weights.h5
โ””โ”€โ”€ VideoMAE

Setup ๐Ÿ› ๏ธ

Download Docker Image ๐Ÿณ

docker pull bruceliu1/evalcrafter:v1

Or, create a conda environment ๐ŸŽฎ

conda env create -f environment.yml
conda activate EvalCrafter

cd ./metrics/RAFT/networks/resample2d_package
python setup.py install --user

Note that if you encounter this problem RuntimeError: GET was unable to find an engine to execute this computation when running the evaluation code, you may update Cuda to version Cuda 11.7 (Compatibility issue).

Usage ๐Ÿš€

Running the Whole Pipeline

  1. Run with command line:

    docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 \
   bash -c "source /opt/conda/bin/activate EvalCrafter \
      && bash $bash_file $EC_path $EC_path/videos"

    ๐Ÿ” Please replace $EC_path, $bash_file, and $dir_videos with your local path to EvalCrafter, EvalCrafter/start.sh, and EvalCrafter/videos, respectively.

Alternatively, you can:

  1. Enter the Docker container and run:

    docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 bash
cd $EC_path
bash start.sh $EC_path $dir_videos

  2. If you use conda, you can directly run the evaluation code by:

    bash start.sh $EC_path $dir_videos

Running a Single Metric

  1. ๐Ÿ”ง To test a specific metric, pick out the code for the metric in start.sh. For example, to test the Celebrity ID Score:

    docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 bash
cd $EC_path
cd /metrics/deepface
python3 celebrity_id_score.py --dir_videos $dir_videos

  2. If you use conda, you can directly run the evaluation code by:

    cd $EC_path
cd /metrics/deepface
python3 celebrity_id_score.py --dir_videos $dir_videos

EvalCrafter Gallery๐ŸŒŒ

Please ref to EvalBoard to use it.

  1. Clone the repository:

    git clone https://github.com/evalcrafter/EvalBoard

  2. Navigate to the project directory:

    cd EvalBoard

  3. Install the required dependencies:

    pip install -r requirements.txt

  4. Start EvalBoard:

    python app.py

Acknowledgements ๐Ÿ™

This work is based on the following open-source repositories:

Citation

If you find this repository helpful, please consider citing it in your research:

@article{liu2023evalcrafter,
title={Evalcrafter: Benchmarking and evaluating large video generation models},
author={Liu, Yaofang and Cun, Xiaodong and Liu, Xuebo and Wang, Xintao and Zhang, Yong and Chen, Haoxin and Liu, Yang and Zeng, Tieyong and Chan, Raymond and Shan, Ying},
journal={arXiv preprint arXiv:2310.11440},
year={2023}
}

Know More About Video Generation at: