dvmazur/mixtral-offloading

Session crashed on colab

Closed this issue · 4 comments

bitsnaps commented

Hi,

Have you guys managed to make it works on T4 colab?

P.S. It crashes multiple times even with offload_per_layer = 5 as mentioned in the comment.
image

dvmazur commented

Hey! Just tried running the notebook (in the offload_per_layer = 5 setting) and everything works for me. Have you tinkered with the original notebook in any way? If not, try restarting the session and running it again starting with the model initialization cell.

  • 👍1
lavawolfiee commented

Hey!

Have you managed to solve this issue? If no, can you please provide some more information:

  1. On which cell exactly did the notebook crush with offload_per_layer = 4?
  2. How many of RAM and GPU VRAM did you have in Google Colab?

Note that our demo notebook should run normally in Google Colab with offload_per_layer = 4, but will crush sometimes with offload_per_layer = 5. The latter option is made for local run with low VRAM.

  • 👍1
Ellokoakarata commented

hqq_aten package not installed. HQQBackend.ATEN backend will not work unless you install the hqq_aten lib in hqq/kernels.

dvmazur commented

hqq_aten package not installed. HQQBackend.ATEN backend will not work unless you install the hqq_aten lib in hqq/kernels.

Hqq_aten is not required as we have custom triton kernels for GEMV.

  • 👍1