microsoft/hi-ml

ImageInferenceEngine can only consume a very specific ImageModel class

corcra opened this issue · 1 comments

I want to use the ImageInferenceEngine from hi-ml-multimodal with an image model that can't cleanly be fit into the existing ImageModel class.

I'd like to propose something like:

  • We define a more abstract class for ImageModels that just requires the methods actually used by ImageInferenceEngine / ImageTextInferenceEngine (from what I can see, this is forward, get_patchwise_projected_embeddings, and the training attribute). The inference engine can then just expect something from this class as an input.
  • The existing ImageModel class can subclass this, so no expected change to current behaviour.
  • I can write a custom subclass for my special model and be able to use it in the inference engine.

Sounds good!