Runtime error when running pope evaluation

[tsamoura]

Dear Authors,

First of all, I would like to congratulate you for your great work! I have an issue when trying to reproduce the experiments for the POPE evaluation that I would like to check with you.

Following your instructions from https://github.com/CircleRadon/Osprey/blob/main/osprey/eval/README.md#5-pope, I ran pope_eval.py with input a file coco_pope_adversarial.json including the single following line:

{"question_id": 1, "image": "COCO_val2014_000000127153.jpg", "text": "Is there a person in the image?", "label": "yes"}

and the parameter --img pointing to the images in the validation set of COCO-2014.

However, when I do this, I am getting the following exception:

object of type 'NoneType' has no len()
  File "/home/user/Osprey/osprey/model/layer.py", line 43, in forward
    num_imgs = len(masks)
  File "/home/user/Osprey/osprey/model/osprey_arch.py", line 106, in prepare_inputs_labels_for_multimodal
    mask_feats, pos_feats = self.mask_extractor(image_features_dict, masks)
  File "/home/user/Osprey/osprey/model/language_model/osprey_llama.py", line 79, in forward
    input_ids, attention_mask, past_key_values, inputs_embeds, labels = self.prepare_inputs_labels_for_multimodal(input_ids, masks, attention_mask, past_key_values, labels, images)
  File "/home/user/Osprey/osprey/eval/pope_eval.py", line 89, in eval
    output_ids = self.model.generate(
  File "/home/user/Osprey/osprey/eval/pope_eval.py", line 157, in <module>
    POPE_EVAL.eval(args.img, args.json, args.answer)
TypeError: object of type 'NoneType' has no len()

The problem is caused by the fact that masks=None when calling the MaskExtractor. Perhaps, the format of the input coco_pope_adversarial.json file should be different? Apologies in advance if I misunderstood something in the instructions that you provided for reproducing the POPE experiments.

PS: to run the experiment, I had to change line 60 in pope_eval.py from

img_path = os.path.join(root_path, data['image'].split('_')[-1])

to

img_path = os.path.join(root_path, data['image'])

[LiWentomng]

@tsamoura
Hello, sorry for late reply. We hava updated the codes for this issue. Please try the latest code.
Any further questions are welcomed. Thanks.

[tsamoura]

Thanks a lot for looking into this! I will look at your fix and let you know if there is any issue. Thanks again.