Module support
hlky opened this issue · 5 comments
Intro
Modules have to be compiled for a range of resolutions, batch sizes, windows and linux, and multiple cuda architectures. This process takes a long time, so the goal is to minimize the number of modules that have to be recompiled in case of an update like has just happened with 8px increment support, while maximizing the overall support.
Why are there multiple modules?
The profiling that occurs when compiling AITemplate modules selects the underlying kernels based on resolution, batch size and cuda architecture.
Most importantly is that a module profiled on sm80 is not likely to work on sm75 because sm75 supports different kernels.
The other factor is performance, one kernel may perform better at higher resolution and another at smaller resolution.
The amount of memory allocated by the module is pre-calculated based on the profiling.
What should be supported?
At the moment, there are modules for batch size 1, windows and linux, sm80. Check your cuda architecture here.
sm80 covers everything 80+. sm70 and sm75 however are unique. sm6X and older are unsupported. there needs to be some testing done on whether sm89 benefits from modules profiled for it
unet
Note: v1 control is for use with v1 controlnet modules, controlnet requires an accompanying unet module that accepts the additional inputs
- v1
- v2
- xl
- xlr
- v1 control
unet that could be supported
- v1 inpaint
- v2 inpaint
- v2 control
vae
vae modules are compatible across architectures, however there could be support for fp32 modules, these would work with the vae models that don't work in fp16
controlnet
v1 is supported, v2 could be supported
Resolutions
vae
- 256, 512, 1024, 2048, 4096
unet
- 768 (not for xl), 1024, 1536, 2048, 4096
controlnet
- 768, 1024, 1536, 2048
Vote on what you will use
React to the comments below to vote on what you will use
Anything else?
VRAM usage as estimated by AITemplate is in modules.json, in mb, excluding the size of the weights, this might help narrow the choices of resolution and batch size.
It would also be great to verify these figures against what people are seeing.
Batch sizes
👍 Batch size 2
😄 Batch size 4
🎉 Batch size 8
Models
👍 v1 inpaint
😄 v2 inpaint
🎉 v2 control/v2 controlnet
🚀 fp32 vae
you guys need to start a kofi or a patreon unless it's expressly forbidden by your contracts
can you explain how to use vae and controlnet with sdxl model? (I tried with vae and controlnet with the sdxl model, the results are blank/black using the example workflow) thk