[Feature] Use FastV with origin llava and transformer.
Closed this issue · 4 comments
Thank you for the great work.
I notice that the directory src/ contains codes from transformer and llava. But the changes are not highlighted.
So can I use the original transformer and llava with FastV? If no, it will be too hard to use FastV in the new versions of llava or other models. The feature is actually very important in this day.
Hi, thanks for your advice. The main change of source code is in the forward function of LlamaModel from modeling_llama.py of transformers repo.
We are now actively working on integrating FastV to transformers and vLLM framework to support more models. It would be done in 1 or 2 weeks. Stay tuned!
We add the reproduction of latency experiment in https://github.com/pkunlp-icler/FastV/blob/main/README.md#latency-experiment-reproduction. Hope it helps.
Thank you!