[Feature] Use FastV with origin llava and transformer.

[tzjtatata]

Thank you for the great work.
I notice that the directory src/ contains codes from transformer and llava. But the changes are not highlighted.

So can I use the original transformer and llava with FastV? If no, it will be too hard to use FastV in the new versions of llava or other models. The feature is actually very important in this day.

[chenllliang]

Hi, thanks for your advice. The main change of source code is in the forward function of LlamaModel from modeling_llama.py of transformers repo.

We are now actively working on integrating FastV to transformers and vLLM framework to support more models. It would be done in 1 or 2 weeks. Stay tuned!

[tzjtatata]

Thanks for your helpful reply！

…

---Original--- From: "Liang ***@***.***> Date: Sun, Mar 17, 2024 19:03 PM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [pkunlp-icler/FastV] [Feature] Use FastV with origin llava andtransformer. (Issue #3) Hi, thanks for your advice. The main change of source code is in the forward function of LlamaModel from modeling_llama.py of transformers repo. We are now actively working on integrating FastV to transformers and vLLM framework to support more models. Stay tuned! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[chenllliang]

We add the reproduction of latency experiment in https://github.com/pkunlp-icler/FastV/blob/main/README.md#latency-experiment-reproduction. Hope it helps.

[tzjtatata]

Thank you!