RenShuhuai-Andy/TimeChat

RAM and VRAM requirement

Coronal-Halo opened this issue · 2 comments

Coronal-Halo commented

Is there any way that this program can run with 16 or 24 GB of VRAM?

RenShuhuai-Andy commented

Hi, thanks for your interest.

You can try installing the bitsandbytes library and add model_config.low_resource = True in https://github.com/RenShuhuai-Andy/TimeChat/blob/master/demo.ipynb. This will load the LLAMA model in 8-bit precision, potentially enabling the model to run with 24 GB of VRAM, see https://github.com/RenShuhuai-Andy/TimeChat/blob/master/timechat/models/timechat.py#L139. However, I cannot guarantee the performance of the 8-bit LLAMA model.

onlyonewater commented

I set the low_resource as false, and the memory of the GPU is only 19G, so I think a 24G GPU like 3090 is enough for inference. If you set the low_resource as True, the memory of the GPU is about 15G, which is suitable for a 16G GPU such as 4060Ti. @Coronal-Halo and @RenShuhuai-Andy

  • 👍2