context memory pool
Leterax opened this issue · 1 comments
Leterax commented
When running the StarCoder model quantized using "q5_1" with a medium-sized context (3500 tokens) I run into this error:
ggml_new_tensor_impl: not enough space in the context's memory pool (needed 412241472, available 411791504)
Segmentation fault (core dumped)
How can I increase the context memory pool size?
NouamaneTazi commented
Yes, I'm able to reproduce this issue! We're investigating a fix here: ggerganov/ggml#175