In this project, we want to answer this question. Can we improve overfitting problem by using regularization approaches on the output? We consider three recent approaches for this problem; Label Smoothing, Knowledge Distillation and Penalizing Output Confidence. In this work, we implement these three methods and evaluate their effect on the test accuracy of two well-known datasets; MNIST dataset and German Traffic Sign Benchmarks (GTSRB) dataset. Our results show that all methods can improve the accuracy on the test data, but using Label Smoothing which is a less complicated method has better accuracy.
Full report of this project is attached here PDF.