bryandlee/Tune-A-Video

Model Isn't Learning

ExponentialML opened this issue · 9 comments

ExponentialML commented

Using Stable Diffusion 1.5 on torch 1.13.1, Cuda 11.6, and the latest version of xformers==0.0.16. I cannot build torch 1.12.1 on my machine.
The model won't learn. It simply looks like the first iteration after every epoch.

(0-500 all look like this)
step_0

Ericxgao commented

Same issue here, I'm using 3.9.12 with Torch @ 1.12.1, Cuda 11.6

JulianJuaner commented

Same issue. Whether using this repo or the official repo.
It seems the model is not updating during training.

  • 👍1
ExponentialML commented

@Ericxgao and @JulianJuaner I found a fix that worked for me. You guys can give it a go and report back.

  1. Install from requirements.txt. Torch version must be 1.12.1 on CUDA 11.3 or 11.6.
    pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
  2. Clone xformers: git clone https://github.com/facebookresearch/xformers
  3. After cloning: cd xformers
  4. Run :git reset --hard 0bad001ddd56c080524d37c84ff58d9cd030ebfd
  5. git submodule update --init --recursive
  6. pip install -e .

After install, try running the script.

  • 🎉5
JulianJuaner commented

@Ericxgao and @JulianJuaner I found a fix that worked for me. You guys can give it a go and report back.

  1. Install from requirements.txt. Torch version must be 1.12.1 on CUDA 11.3.
    pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
  2. Clone xformers: git clone https://github.com/facebookresearch/xformers
  3. After cloning: cd xformers
  4. Run :git reset --hard 0bad001ddd56c080524d37c84ff58d9cd030ebfd
  5. git submodule update --init --recursive
  6. pip install -e .

After install, try running the script.

Thanks! It works for me. It seems the version of xformers is essential.

Ericxgao commented

Hmm I'm still having trouble getting this version of xformers installed. What GPU and python version are you @ExponentialML @JulianJuaner ? I'm using a cloud A100.

ExponentialML commented

@Ericxgao If you're using an A100, you should be able to fit the model in 40GB of vram when training, so xformers shouldn't be needed. Is this not the case?

Ericxgao commented

I still get OOM errors - I disabled Adam 8 bit as that was also failing on my system (bitsandbytes doesn't seem to install properly)

mili-inch commented

facebookresearch/xformers#631
This is probably caused by the same issue as this one, but it has been resolved and the model was successfully trained using xFormers in version 0.0.17.dev441.

ExponentialML commented

Closing with solutions from me & @mili-inch . If there are any other issues, feel free to ask for a re-open to discuss.