CUDA error: device-side assert triggered

Question

CUDA error: device-side assert triggered

kuprel opened this issue 2 years ago · 4 comments

When running in high load settings on the A100 this error comes up after about 10 minutes. This happens on replicate and on the discord bot. The best fix so far is just to restart the server every 10 minutes. If anyone has a better fix for it, please post it here. Thanks

Answer 1 · 2022-07-07T16:57:30.000Z

Is this happening on your own server or in that buggy Google Colab that likes to give weird errors?

Answer 2 · 2022-07-07T17:54:39.000Z

Hah. I think I found a hacky fix that is working for now. Basically just try-except where except deletes the model and reinitializes it.

Answer 3 · 2022-07-07T18:28:29.000Z

jk, that didn't work

Answer 4 · 2022-07-08T16:48:57.000Z

Fix was to clamp the image and text tokens within their bounds. Either the text or image tokens must sometimes exceed their vocab count, I'm not sure which