Can't run the provided llama2 example

Question

Can't run the provided llama2 example

KerolosAtef opened this issue a year ago · 6 comments

I got this error message while trying to run the prompt in the README file
File "python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 402, in forward
kv_seq_len += past_key_value.get_usable_length(kv_seq_len, self.layer_idx)
AttributeError: 'list' object has no attribute 'get_usable_length'

KerolosAtef commented a year ago

@urialon

Answer 1 · 2023-12-30T20:13:07.000Z

Hi there-- what version of transformers are you running? This looks like an error inside the transformers library. Are you able to run llama without unlimiformer?

Answer 2 · 2023-12-30T20:26:22.000Z

transformer version is 4.36.0 and also this happenes with the latest version 4.36.2.
yes, it runs normally without unlimiformer

Answer 3 · 2023-12-30T20:48:33.000Z

downgrading to 4.27.0 should (likely) fix this for now; I'll look into the issue with the newer version

Answer 4 · 2023-12-31T10:28:44.000Z

Thanks, it works with version 4.28.0

Answer 5 · 2024-05-15T20:37:18.000Z

Hi, is it possible to have a quick fix of this issue in the new version? Seems like downgrade transformers to a version before 4.35.0 has several rust compiler error. It's probably more convenient to use to most recent one. Thank you.