jianzongwu/robust-ref-seg

Question about metric r

Closed this issue · 2 comments

lxa9867 commented

Hi,

Is the metric r the same as the one defined in https://lxa9867.github.io/works/rrvos/index.html?

jianzongwu commented

Hi,

Thanks for your question!

I have just read your paper and checked the definition of r metric.

We both call it robustness, but the definition is totally different.

In R2VOS, $R = 1 - \frac{IoU_{neg}}{IoU_{pos}}$, which is a mask area ratio that encourages the model to predict smaller masks.

In RefSegformer, we generate 10 negative sentences for each reference in RefCOCO/+/g, and count how many masks the model outputs for the object (even 1 pixel is counted as a mask. We expect the model to output 0 pixels). R is the none-mask count divided by the number of negative sentences (10 in our settings).

Moreover, the final metrics to evaluate our proposed R-RIS task is rIoU, which is calculated by $rIoU = \sum_{i=1}(r_i \cdot mIoU_i)$

XLHappy123 commented

Our metric will not encourage the model to predict smaller masks.

Anyway, good luck