ImageInferenceEngine can only consume a very specific ImageModel class
corcra opened this issue · 1 comments
corcra commented
I want to use the ImageInferenceEngine
from hi-ml-multimodal with an image model that can't cleanly be fit into the existing ImageModel
class.
I'd like to propose something like:
- We define a more abstract class for ImageModels that just requires the methods actually used by
ImageInferenceEngine
/ImageTextInferenceEngine
(from what I can see, this isforward
,get_patchwise_projected_embeddings
, and thetraining
attribute). The inference engine can then just expect something from this class as an input. - The existing
ImageModel
class can subclass this, so no expected change to current behaviour. - I can write a custom subclass for my special model and be able to use it in the inference engine.
fepegar commented
Sounds good!