aimagelab/show-control-and-tell

Running on other images

phiresky opened this issue · 2 comments

phiresky commented

It would be great if you could describe if it is possible and how to run inference on new images (without loading all the training data) :)

marcellacornia commented

Hi @phiresky,
thanks for your interest.

For the moment, it is not possible to test our model on new images.

For each input image, our model takes as input a control signal in the form of a sequence or a set of image regions and predicts an output caption following the given control. In our experiments, we use the control signals coming from COCO and Flickr30k captions.

phiresky commented

Ah okay, thanks for the info. So to work on general image, it basically needs a object detection model that predicts bounding boxes and classifies them. That shouldn't be that hard, but I'm not sure if I'll spend the effort :)