ipex-llm version 0510 has regression than 0430, especially for BS=16,32 and 8k input

Question

Closed this issue 17 days ago · 3 comments

Answer 1 · 2024-05-14T02:43:03.000Z

Please enable low memory mode and check if this issue still exists. You could use export IPEX_LLM_LOW_MEM=1 to enable low memory mode.

Answer 2 · 2024-05-14T08:25:09.000Z

Cannot reproduce the performance regression from 0430 to 0510. Maybe check the scripts and env setting with user later.

Answer 3 · 2024-05-23T01:33:48.000Z

No performance regression observed in stable release validation test, close issue for now.