/SOD_Evaluation_Metrics

A more complete python version (GPU) of the evaluation for salient object detection (with S-measure, Fbw measure, MAE, max/mean/adaptive F-measure, max/mean/adaptive E-measure, relax-F-Measure, IOU score, PRcurve and F-measure curve)

Primary LanguageJupyter Notebook

SOD_Evaluation_Metrics

A more complete python version (GPU) of the fast evaluation for salient object detection (with S-measure, Fbw measure, MAE, max/mean/adaptive F-measure, max/mean/adaptive E-measure, PRcurve and F-measure curve)

  • A fork from zyjwuyan's repo

Main Contribution

  • adding IOU and Relax F-measure
  • reorganize directory per dataset
  • enable dynamic query and storage of results with sqlite

Data Storage

  • The source files should be orginized as follows:

    --Dataset_1/
      --masks/
          --img1.png
          --img2.png
                  ...
      --method1/
          --img1.png
          --img2.png
                  ...
    --Dataset_2/
      --masks/
          --img1.png
          --img2.png
                  ...
      --method1/
          --img1.png
          --img2.png
      	    ...
      ...
    
  • Evaluate your map by run:

python ./src/main.py \ --pred_root_dir ${dataset_dir} \ --save_dir ${score_dir} \ --methods "method1 method2"\ --datasets "Dataset_1 Dataset_2" \ --cuda False

  • The format of the result file is shown as this.

    #[Dataset_Name] [Method_Name]# [value mae], [value max-fmeasure], [value mean-fmeasure], [value9 adp-fmeasure], [value max-Emeasure], [value mean-Emeasure], [value adp-Emeasure], [value S-measure_alpha05], [value Fbw-measure], [value relax-F-meausre], [value mean_IoU].

  • Create Sqlite DB by running: python store_metrics.py

    Then, the sqlite database will be created and available as metrics.db

  • Draw the PR curve and F-measure Curve by run:

    python draw_curve.py

    Then, the image file will be saved to './score/'. Two virtual curves are shown as follows:

PR curve: Dataset_2 F-measure curve: Dataset_2
pr fm
  • The above metrics are related to the following papers:

    @inproceedings{Fmeasure,
        title={Frequency-tuned salient region detection},
        author={Achanta, Radhakrishna and Hemami, Sheila and Estrada, Francisco and S{\"u}sstrunk, Sabine},
        booktitle=CVPR,
        number={CONF},
        pages={1597--1604},
        year={2009}
    }
    
    @inproceedings{MAE,
        title={Saliency filters: Contrast based filtering for salient region detection},
        author={Perazzi, Federico and Kr{\"a}henb{\"u}hl, Philipp and Pritch, Yael and Hornung, Alexander},
        booktitle=CVPR,
        pages={733--740},
        year={2012}
    }
    
    @inproceedings{Smeasure,
        title={Structure-measure: A new way to eval foreground maps},
        author={Fan, Deng-Ping and Cheng, Ming-Ming and Liu, Yun and Li, Tao and Borji, Ali},
        booktitle=ICCV,
        pages={4548--4557},
        year={2017}
    }
    
    @inproceedings{Emeasure,
        title="Enhanced-alignment Measure for Binary Foreground Map Evaluation",
        author="Deng-Ping {Fan} and Cheng {Gong} and Yang {Cao} and Bo {Ren} and Ming-Ming {Cheng} and Ali {Borji}",
        booktitle=IJCAI,
        pages="698--704",
        year={2018}
    }
    
    @inproceedings{wFmeasure,
      title={How to eval foreground maps?},
      author={Margolin, Ran and Zelnik-Manor, Lihi and Tal, Ayellet},
      booktitle=CVPR,
      pages={248--255},
      year={2014}
    }
    
    @inproceedings{relaxFmeasure,
        title = {BASNet: Boundary-Aware Salient Object Detection},
        author = {Qin, Xuebin and Zhang, Zichen and Huang, Chenyang and Gao, Chao and Dehghan, Masood and Jagersand, Martin},
        booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
        month = {June},
        year = {2019}
    }