NetEase-FuXi/EETQ

Does it support Whisper model?

Closed this issue · 2 comments

kadirnar commented
Does it support Whisper model?
SidaZh commented

In theory, eetq supports all models supported by transformers, you can try this:

from transformers import AutoModelForCausalLM, EetqConfig

path = "/path_to_model"
quantization_config = EetqConfig("int8")
model = AutoModelForCausalLM.from_pretrained(path, device_map="auto", quantization_config=quantization_config)
  • ❤️1
kadirnar commented

I tested it and it works. What can I do to optimize further? Have you tested with Torch.compile?