I would like to share my results (93.43% accuracy on average) on the fashion-MNIST dataset. You can find further informations about the dataset on Zalando Research and Kaggle
This dataset is a great option instead of using traditional handwritten MNIST.
Thanks to Han and Kashif!
I splitted the training data randomly in train (80%) and validation (20%). The testset of 10k images are used for final evaluation. I created 5 models with the same architecture but with random train/validation data. I only saved the weights of every model with best loss. Finally I used the models to evaluate them on the testset. The average loss/accuracy of the 5 models is the final result.
Furthermore I evaluate the generalization of the network for classifying traditional MNIST at the end.
Actually I'm focussed on a very simple architecture with less than 500,000 parameters which could run on CPU with 4GB memory. If you are interested in further results, you can find them on Zalando benchmark. If it fits in time, I will evaluate architectures e.g. DenseNet.
I'm just using two convolutional layers, batchnorm, dropout and three fully connected layers. For a detailed implementation check the model definition in the jupyter notebook.
This is the summary of the current model:
It's easy to reach an accuracy over 90% but around 92% it stucks very fast. If we take a look on the confusion matrix, we will find out that distinguishing between Shirt and T-Shirt/top is very difficult:
Here's an example where my model fails. I'm not a fashion expert. Maybe thats the reason why I don't see any difference. If you could explain why this is a shirt and not a t-shirt I would appreciate it:
Another two plots which illustrate the accuracy / loss of training and validation over the time:
And a detailed plot of iteration 0:
Scores for training: 0.1066 loss / 95.99% accuracy.
Scores for validation: 0.1245 loss / 95.64% accuracy.
Scores for test: 0.2149 loss / 93.43% accuracy.
I used the same architecture of neural network to train on traditional MNIST.
Scores for training: 0.078 loss / 99.74% accuracy.
Scores for validation: 0.161 loss / 99.61% accuracy.
Scores for test: 0.0248 loss / 99.43% accuracy.
In the plot below you can track training / validation for MNIST:
| Fashion-MNIST test accuracy |
Fashion-MNIST train accuracy |
Fashion-MNIST validation accuracy |
MNIST test accuracy |
Add. Settings | Trainable Params |
|---|---|---|---|---|---|
| 93.43% | 95.99% | 95.64% | 99.43% | BatchSize : 250 Epochs : 80 Data augmentation (2x) |
493,772 |
- TensorFlow 2.x
- TensorFlow-Datasets
If you have any questions or hints contact me through an issue. Thanks!
[1] Fashion-MNIST
[2] Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Christopher Masch