facebookresearch/multimodal

Add support for LLaVA model

youssefadr opened this issue · 7 comments

youssefadr commented

🚀 The feature, motivation and pitch

LLaVA seems to be currently a strong open-source competitor to GPT4-V, it doesn't seem to be supported by the library. Do you plan on adding it? If yes, is there something I could contribute with to help?

Alternatives

No response

Additional context

No response

ebsmothers commented

Hi @youssefadr, thanks for opening this issue. LLaVA is definitely something we're interested in adding and we would be happy to have you contribute. Is there a specific portion of the model you're especially interested in helping out with?

youssefadr commented

Thanks for your answer @ebsmothers, I would like to add the model to torchmultimodal/models first.

  • 👍1
ebsmothers commented

That sounds reasonable to me. We already have CLIP visual encoders in the library here, so feel free to reuse those. Then the bulk of the work for the model should be to add the LLM. A couple pointers to help with that: TransformerDecoderLayer, RMSNorm. We also have an open PR for rotary positional embeddings (#450) that might be useful. Let me know if this makes sense, happy to provide more details as needed.

  • 👍1
youssefadr commented

Nice ! I'll come back to you with more questions later, not sure I'll start working on it this week.

  • ❤️1
theadamsabra commented

@youssefadr have you worked on this to any capacity? i'm interested in picking this up if not

ebsmothers commented

@theadamsabra if not, you are more than welcome to take it up

  • 👍2
theadamsabra commented

@ebsmothers thanks! If I don't get a response by tomorrow I'll just pick it up myself

  • 👍1
  • ❤️1