Lightning-Universe/lightning-transformers

Integrate DALLE-Pytorch

SeanNaren opened this issue · 3 comments

SeanNaren commented

🚀 Feature

This would be awesome to showcase the power of this library:

https://github.com/openai/CLIP
https://github.com/lucidrains/DALLE-pytorch

SeanNaren commented

Currently the code exists in the temp branch, and needs to be completed after a few clean outs. It currently does not use CLIP to rank because the sampling logic needs to be added in the right way for ranking to happen.

Also might be a good idea to train the VQVAE for even more steps on CIFAR, and DALLE as well. For DALLE, it would also be worth doing some hyper-parameter tuning.

SeanNaren commented

I'm moving this out of the 0.1 release for focusing on getting iGPT in end to end! we can continue working on the https://github.com/PyTorchLightning/lightning-transformers/tree/dalle branch after release.

SeanNaren commented

Keeping this repo HF only for now!