classification service and class names with labels
Opened this issue · 10 comments
I see in your classificationService.py file you are assigning the classes = ['ant', 'bee'].
However, I do not see the linkage between your ./data/train/ants and ./data/train/bees.
Can you please explain where you associate the class names to the directories named ants and bees?
Second, I am interested in expanding the classes themselves, to include for example horses and pigs.
Because I want my service to detect ants, bees, horses, and pigs.
I did notice when I tested your service with a Seal or Whale, that the neural net performance had degraded and it now thinks that they are either an Ant or a Bee, which was incorrect, based on what the Resnet model originally thought.
Can you please explain how I can add class labels and pictures to this neural net example?
Yes that is actually a mistake from my part and I should not have hardcoded those values.
The right way to do it I think, is to use the classes and class_to_idx attributes to fetch the class names from the saved model. Can you verify?
I don't see how there can be a drop in performance when using seal or whale though, if your doing a proper training it should work. Can you share the training logs?
@jjmachan Look at this article that explains the class_to_idx label mapping and jpeg process. (https://discuss.pytorch.org/t/how-to-represent-class-to-idx-map-for-custom-dataset-in-pytorch/37510/4). If you had an example to show me, with cats, dogs, ants, and bees, it would make it clearer. Can you interpret the article, and provide a working example?
@jjmachan See sample here (https://stackoverflow.com/questions/51906144/pytorch-image-label/51906229). It is not clear to me if this will work with your example. Please advise.
@jjmachan I am retraining now, and will upload a GitHub repo with logs, and whales and seals pictures as data. stay tuned...
@jjmachan Take a look at the dataset that includes Seals and Whales. (https://www.dropbox.com/s/4tzs3upaakjjihu/resnet-bentoml.tar.gz?dl=0) See Checksums: 427653750 Aug 13 10:36 resnet-bentoml.tar.gz
@jjmachan See training log as requested.
python3 train.py
/Users/hottelet/Library/Python/3.8/lib/python/site-packages/setuptools/distutils_patch.py:17: UserWarning: Setuptools is replacing distutils
warnings.warn("Setuptools is replacing distutils")
[2020-08-13 09:53:47,550] WARNING - bentoml.handlers.* will be deprecated after BentoML 1.0, use bentoml.adapters.* instead
[2020-08-13 09:53:47,569] WARNING - ImageHandler will be deprecated after BentoML 1.0, use LegacyImageInput instead
Epoch 0/24
train Loss: 0.6281 Acc: 0.6926
val Loss: 0.4317 Acc: 0.8039
Epoch 1/24
train Loss: 0.3828 Acc: 0.8402
val Loss: 0.2063 Acc: 0.9281
Epoch 2/24
train Loss: 0.4210 Acc: 0.8115
val Loss: 0.3512 Acc: 0.8693
Epoch 3/24
train Loss: 0.4522 Acc: 0.8238
val Loss: 0.7040 Acc: 0.7320
Epoch 4/24
train Loss: 0.4790 Acc: 0.7910
val Loss: 0.3442 Acc: 0.8693
Epoch 5/24
train Loss: 0.3671 Acc: 0.8279
val Loss: 0.2244 Acc: 0.9150
Epoch 6/24
train Loss: 0.4931 Acc: 0.8156
val Loss: 0.2152 Acc: 0.8954
Epoch 7/24
train Loss: 0.3986 Acc: 0.8197
val Loss: 0.2025 Acc: 0.9150
Epoch 8/24
train Loss: 0.2944 Acc: 0.8893
val Loss: 0.2000 Acc: 0.9085
Epoch 9/24
train Loss: 0.3064 Acc: 0.8525
val Loss: 0.1949 Acc: 0.9281
Epoch 10/24
train Loss: 0.2143 Acc: 0.9262
val Loss: 0.1955 Acc: 0.9216
Epoch 11/24
train Loss: 0.2836 Acc: 0.8811
val Loss: 0.2172 Acc: 0.9085
Epoch 12/24
train Loss: 0.2737 Acc: 0.8689
val Loss: 0.2224 Acc: 0.9020
Epoch 13/24
train Loss: 0.2586 Acc: 0.8689
val Loss: 0.1914 Acc: 0.9281
Epoch 14/24
train Loss: 0.2657 Acc: 0.8811
val Loss: 0.1948 Acc: 0.9412
Epoch 15/24
train Loss: 0.2530 Acc: 0.8852
val Loss: 0.1936 Acc: 0.9216
Epoch 16/24
train Loss: 0.2689 Acc: 0.8811
val Loss: 0.1962 Acc: 0.9281
Epoch 17/24
train Loss: 0.2730 Acc: 0.8770
val Loss: 0.1917 Acc: 0.9346
Epoch 18/24
train Loss: 0.2515 Acc: 0.9016
val Loss: 0.1975 Acc: 0.9281
Epoch 19/24
train Loss: 0.2534 Acc: 0.8852
val Loss: 0.1878 Acc: 0.9346
Epoch 20/24
train Loss: 0.2875 Acc: 0.8893
val Loss: 0.1895 Acc: 0.9281
Epoch 21/24
train Loss: 0.2165 Acc: 0.9426
val Loss: 0.1799 Acc: 0.9346
Epoch 22/24
train Loss: 0.3299 Acc: 0.8443
val Loss: 0.2110 Acc: 0.9085
Epoch 23/24
train Loss: 0.2156 Acc: 0.9098
val Loss: 0.1950 Acc: 0.9281
Epoch 24/24
train Loss: 0.2368 Acc: 0.9057
val Loss: 0.2114 Acc: 0.9020
Training complete in 30m 11s
Best val Acc: 0.941176
Save model to Bento ML (yes/no): yes
[2020-08-13 10:24:25,805] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via @env(pip_dependencies=['torchvision']) when defining a BentoService
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/storage.py:34: FutureWarning: pickle support for Storage will be removed in 1.5. Use torch.save instead
warnings.warn("pickle support for Storage will be removed in 1.5. Use torch.save instead", FutureWarning)
/Users/hottelet/Library/Python/3.8/lib/python/site-packages/traitlets/config/loader.py:795: SyntaxWarning: "is" with a literal. Did you mean "=="?
if len(key) is 1:
/Users/hottelet/Library/Python/3.8/lib/python/site-packages/traitlets/config/loader.py:804: SyntaxWarning: "is" with a literal. Did you mean "=="?
if len(key) is 1:
[2020-08-13 10:24:54,200] INFO - BentoService bundle 'AntOrBeeClassifier:20200813102426_7E8770' saved to: /Users/hottelet/bentoml/repository/AntOrBeeClassifier/20200813102426_7E8770
Bento Service saved in /Users/hottelet/bentoml/repository/AntOrBeeClassifier/20200813102426_7E8770
Saving model to: ./saved_models
@jjmachan See Seals being detected as Ants or Bees. This is wrong.
curl -X POST "http://127.0.0.1:5000/predict" -F image=@SealBeach.jpg
"ant"
curl -X POST "http://127.0.0.1:5000/predict" -F image=@seal1.jpg
"bee"
curl -X POST "http://127.0.0.1:5000/predict" -F image=@seal2.jpg
"bee"
curl -X POST "http://127.0.0.1:5000/predict" -F image=@seal3.jpg
"ant"
@jjmachan See Whales being detected as Ants. This is wrong.
curl -X POST "http://127.0.0.1:5000/predict" -F image=@BlueWhale.jpg
"ant"
curl -X POST "http://127.0.0.1:5000/predict" -F image=@killerWhale.jpg
"ant"
curl -X POST "http://127.0.0.1:5000/predict" -F image=@humpBackWhale.jpg
"ant"
@jjmachan If you wish to recreate the results, you can run the python3 train.py script, and then use the curl commands in each respective folder. Either ./whales or ./seals
@jjmachan What I am trying to do is have your model detect Seals and Whales, but not have them detected as Ants or Bees. Please advise, because the results are wrong, based on the resnet model performance that should know if it is a Seal or Whale already.