Unofficial pytorch implementation of DoctorNet from "Who Said What: Modeling Individual Labelers Improves Classification"
Paper: https://arxiv.org/abs/1703.08774
We used LabelMe for validating implementation. LabelMe is an image classification task that was labeled by 77 annotators in AMT(Amazon Mechanical Turk). Original data is here. However in the real data, 18 of the annotators didn't labeled any image. So total 59 annotators' labels were used for training. You can download the preprocessed version here.
Training is done in two-stage. First, we train annotator classifiers with shared feature extractor - Inception v3. After the classifiers converged, we fix them and train weights used for averaging decisions of annotators.
| Model | DoctorNet(paper)[1] | doctornet-pytorch |
|---|---|---|
| Accuracy | 82.12 | 77.61 |