Bora: Biomedical Generalist Video Generation Model

Weixiang Sun^1*, Xiaocao You^2*, Ruizhe Zheng^3*, Zhengqing Yuan⁴, Xiang Li⁵, Lifang He⁶, Quanzheng Li⁵, Lichao Sun⁶

¹Northeastern University, ²Shanghai University of Finance and Economics, ³Fudan University, ⁴University of Notre Dame, ⁵Massachusetts General Hospital and Harvard Medical School, ⁶Lehigh University

📰 News

[2024.6.19] We release Bora, a video generation model specificaly for biomedical domain.

🎥 Some Demos

Endoscopy	Ultrasound	RT-MRI	Cell

Installation
Prepare
Inference
Training
Citation
Acknowledgement
Star History

Installation

# create a virtual env
conda create -n bora python=3.10
# activate virtual environment
conda activate bora
# install torch
# We recommend torch==2.2.2 under CUDA12.1
pip install torch torchvision

# install flash attention
pip install packaging ninja
pip install flash-attn --no-build-isolation

# install apex
# We recommend install from source
git clone https://github.com/NVIDIA/apex.git
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./

# install xformers
pip install -U xformers --index-url https://download.pytorch.org/whl/cu121

# install opensora
pip install -v .

Prepare

Before running, besides Bora's weights, you also need to download the weights for the VAE and Text Encoder. We have provided all the links in the table below:

Bora	Video Encoder	Text Encoder
Bora	VAE	T5

Inference

# on single card
torchrun --standalone --nproc_per_node 1 scripts/inference.py configs/infer.py --ckpt-path Bora_CKPT

# on multi cards
torchrun --standalone --nproc_per_node N scripts/inference.py configs/infer.py --ckpt-path Bora_CKPT

Train

# on four cards
torchrun --nnodes=1 --nproc_per_node=4 scripts/train_origin.py configs/train.py --data-path CSV_PATH --ckpt-path Bora_CKPT

To launch training on multiple nodes, prepare a hostfile according to ColossalAI, and run the following commands.

colossalai run --nproc_per_node 8 --hostfile hostfile scripts/train_origin.py configs/train.py --data-path CSV_PATH --ckpt-path Bora_CKPT

Citation

If you're using Bora in your research or applications, please cite using this BibTeX:

@article{sun2024bora,
  title={Bora: Biomedical Generalist Video Generation Model},
  author={Sun, Weixiang and You, Xiaocao and Zheng, Ruizhe and Yuan, Zhengqing and Li, Xiang and He, Lifang and Li, Quanzheng and Sun, Lichao},
  journal={arXiv preprint arXiv:2407.08944},
  year={2024}
}

Acknowledgement

We are greatful for the following works and generous contribution to open source.

Open-Sora: Democratizing Efficient Video Production for All

LLaVA: Large Language and Vision Assistant

Apex: A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Weixiang-Sun/Bora