SHI-Labs/Matting-Anything

evaluation result is inconsistent with Table1 of the paper on the ppm100

Closed this issue · 1 comments

haoxurt commented

I evaluate using mam_vitl.pth on the ppm100, and the following picture is result.
image
It is inconsistent with Table1 of the paper.
image

chrisjuniorli commented

  1. The result you got is the SAD / MAD / MSE on PPM-100 and Table 1 is a summary table for all benchmarks (e.g. 17.30 is for AM2K, 15.4 is for PM-10K and 4.6 is for PPM-100). The full results for PPM-100 are in Table 3.

  2. All results we reported in all Tables are based on mam_vitb.pth. Using mam_vitl.pth should give you better results.

  3. MSE and MAD should be scaled by 10^3 to match the numbers in the Table, for example, the 2.5 MSE you got with mam_vitl.pth is better than 4.6 MSE in the Table with mam_vitb.pth