Clarification on Referring Segmentation

[yxchng]

Based on the code:

X-Decoder/datasets/dataset_mappers/refcoco_dataset_mapper.py

Line 158 in 584123d

texts_grd.append([x['raw'].lower() for x in ann['sentences']])

X-Decoder/xdecoder/architectures/xdecoder_model.py

Lines 419 to 424 in 584123d

	t_emb = getattr(self.sem_seg_head.predictor.lang_encoder, "{}_text_embeddings".format('grounding')).t()
	v_emb = caption_pred_result[:-1]
	v_emb = v_emb / (v_emb.norm(dim=-1, keepdim=True) + 1e-7)
	vt_sim = v_emb @ t_emb
	max_id = vt_sim.max(0)[1][0]
	grd_masks += [mask_pred_result[max_id]]

Is it true that referring segmentation in X-Decoder is done by segmentation -> classification (matching mask with highest similarity)?

[MaureenZOU]

Yes, but we are not directly matching the mask embedding, we compute hungarian matching between pred mask and gt mask, pred text embedding and gt text embedding. During evaluation we use text embedding matching score only.