Solutions of L&T EduTech Hackathon by our team - Bharatfly_Coders
Each folder contains a folder explaining the results, metrics, training and observations in detailed.
Throughout all task general observation is that the Vision transformers (ViT) performed very well compared to classical models such as Resnet, VGG16, EfficientNet-Bx.
https://drive.google.com/file/d/1VYsAKcM1DK613JGXaUBbw3NfFrW6ZXLg/view?usp=drivesdk
SNO | Model | Training loss | Training accuracy | Validation loss | Validation accuracy | Test Loss | Test Accuracy | Precision | Recall | F1 Score | Support | Size(mb) | ||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Positive | Negative | Positive | Negative | Positive | Negative | Positive | Negative | |||||||||
1 | vit_tiny_r_s16_p8_224 | 0.000554299960640491 | 100 | 0.0162891943918321 | 98.5576923076923 | 0.0168405814239612 | 100 | 1 | 1 | 1 | 1 | 1 | 1 | 100 | 100 | 24.64 |
1a | vit_tiny_r_s16_p8_224(adam) | 8.65E-06 | 100 | 0.341207414359325 | 96.1538461538461 | 1.32988578424848 | 84.1346153846153 | 0.75757576 | 1 | 1 | 0.68 | 0.86206897 | 0.80952381 | 100 | 100 | 24.64 |
* 2 | vit_tiny_patch16_224 | 0.000391737806766238 | 100 | 0.0066456968404684 | 100 | 0.00862143541542956 | 100 | 1 | 1 | 1 | 1 | 1 | 1 | 100</td> | 100 | 24.64 |
3 | vit_small_patch8_224_dino | 0.000114268117565523 | 100 | 0.000981450813504544 | 100 | 0.00134592478011304 | 100 | 1 | 1 | 1 | 1 | 1 | 1 | 100 | 100 | 86.74 |
4 | vit_small_patch16_224 | 0.00109570801348462 | 100 | 0.0165256364203882 | 99.0384615384615 | 0.0172053847032097 | 100 | 1 | 1 | 1 | 1 | 1 | 1 | 100 | 100 | 86.72 |
5 | vit_base_patch8_224 | 2.34E-05 | 100 | 0.0109473363600591 | 99.5 | 0.0298415567417396 | 99 | 1 | 0.98039216 | 0.98 | 1 | 0.98989899 | 0.99009901 | 100 | 100 | 343.3 |
6 | vit_base_patch16_224 | 2.89E-05 | 100 | 0.000303537664754003 | 100 | 0.000899293540896906 | 100 | 1 | 1 | 1 | 1 | 1 | 1 | 100 | 100 | 343.26 |
7 | vit_large_patch16_224 | 0.000131708909235233 | 100 | 0.00477138461696085 | 100 | 0.0421330400995793 | 99.5 | 1 | 0.99009901 | 0.99 | 1 | 0.99497487 | 0.99502488 | 100 | 100 | 1.21gb |
Model name: vit_tiny_patch16_224
TEST PREC: [1. 1.] RECALL: [1. 1.] F1 SCORE: [1. 1.] SUPPORT: [100 100]
SNO | Model | Training loss | Training accuracy | Validation loss | Validation accuracy | Loss | Accuracy_best | Precision | Recall | F1 Score | Support | Kappa | Size(mb) | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Surge Arrestor | Transformer | Transformer with Surge Arrestor | Surge Arrestor | Transformer | Transformer with Surge Arrestor | Surge Arrestor | Transformer | Transformer with Surge Arrestor | Surge Arrestor | Transformer | Transformer with Surge Arrestor | ||||||||||
1 | vit_tiny_r_s16_p8_224 | 0.0225348259555175 | 99.4791666666666 | 1.20923614501953 | 66.6666666666666 | 0.377547923475503 | 89.0625 | 0.78947368 | 0.9375 | 0.89473684 | 0.83333333 | 0.83333333 | 0.94444444 | 0.81081081 | 0.81081081 | 0.91891892 | 18 | 18 | 18 | 0.805555555555555 | 24.64 |
2 | vit_tiny_patch16_224 | 0.0177038602996617 | 98.9583333333333 | 0.967087268829345 | 75 | 0.401243545114994 | 81.7708333333333 | 0.76470588 | 0.8 | 0.88235294 | 0.72222222 | 0.88888889 | 0.83333333 | 0.74285714 | 0.84210526 | 0.85714286 | 18 | 18 | 18 | 0.722222222222222 | 22.16 |
3 | vit_small_patch8_224_dino | 0.0191240947363742 | 98.9583333333333 | 0.354451566934585 | 91.6666666666666 | 0.413140682503581 | 81.7708333333333 | 0.72222222 | 0.85 | 0.875 | 0.72222222 | 0.94444444 | 0.77777778 | 0.72222222 | 0.89473684 | 0.82352941 | 18 | 18 | 18 | 0.722222222222222 | 86.74 |
4 | vit_small_patch16_224 | 0.0229812290053814 | 98.9583333333333 | 0.567398607730865 | 75 | 0.31965248286724 | 83.3333333333333 | 0.8 | 0.77272727 | 0.94117647 | 0.66666667 | 0.94444444 | 0.88888889 | 0.72727273 | 0.85 | 0.91428571 | 18 | 18 | 18 | 0.75 | 86.72 |
* 5 | vit_base_patch8_224(sgd) | 0.0134351163142127 | 98.8888888888888 | 1.07717802127202 | 58.3333333333333 | 0.220577826648617 | 92.8571428571428 | 0.88888889 | 0.94444444 | 0.94444444 | 0.88888889 | 0.94444444 | 0.94444444 | 0.88888889 | 0.94444444 | 0.94444444 | 18 | 18 | 18 | 0.888888888888888 | 343.3 |
5a | vit_base_patch8_224(adam) | 1.22904528545008 | 50 | 1.18211032946904 | 50 | 1.16420696462903 | 44.6428571428571 | 0.27272727 | 0.40740741 | 0.625 | 0.16666667 | 0.61111111 | 0.55555556 | 0.20689655 | 0.48888889 | 0.58823529 | 18 | 18 | 18 | 0.166666666666666 | 343.3 |
6 | vit_base_patch16_224 | 0.0138681949919878 | 99.4444444444444 | 1.38310711582501 | 75 | 0.315895141800865 | 85.7142857142857 | 0.77777778 | 0.88888889 | 0.88888889 | 0.77777778 | 0.88888889 | 0.88888889 | 0.77777778 | 0.88888889 | 0.88888889 | 18 | 18 | 18 | 0.777777777777777 | 343.26 |
7 | vit_large_patch14_224 | 0.949684994750552 | 53.3333333333333 | 1.11181551218032 | 58.3333333333333 | 1.00124096231801 | 53.5714285714285 | 0.48148148 | 0.53846154 | 0.64285714 | 0.72222222 | 0.38888889 | 0.5 | 0.57777778 | 0.4516129 | 0.5625 | 18 | 18 | 18 | 0.305555555555555 | 1.21gb |
8 | vit_large_patch16_224(batch_size4) | 0.0184153128463852 | 99.4444444444444 | 1.32563134034474 | 66.6666666666666 | 0.415346324234893 | 85.7142857142857 | 0.78947368 | 0.84210526 | 0.9375 | 0.83333333 | 0.88888889 | 0.83333333 | 0.81081081 | 0.86486486 | 0.88235294 | 18 | 18 | 18 | 0.777777777777777 | 1.21gb |
8a | vit_large_patch16_224(batch_size3) | 0.0220416884048366 | 99.4444444444444 | 1.28843541815876 | 66.6666666666666 | 0.453457333567914 | 85.1851851851851 | 0.78947368 | 0.84210526 | 0.9375 | 0.83333333 | 0.88888889 | 0.83333333 | 0.81081081 | 0.86486486 | 0.88235294 | 18 | 18 | 18 | 0.777777777777777 | 1.21gb |
Among multiple models vit_base_patch8_224
stood out in performance with a kappa score of 0.8888888888888888
Model name: vit_base_patch8_224
TEST PREC: [0.88888889 0.94444444 0.94444444] RECALL: [0.88888889 0.94444444 0.94444444] F1 SCORE: [0.88888889 0.94444444 0.94444444] SUPPORT: [18 18 18]
KAPPA: 0.8888888888888888
SNO | Model | Training loss | Training accuracy | Validation loss | Validation accuracy | Precision | Recall | F1 Score | Kappa | Size(mb) | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Cargo | Military | Carrier | Cruise | Tanker | Cargo | Military | Carrier | Cruise | Tanker | Cargo | Military | Carrier | Cruise | Tanker | ||||||||
1 | resnet50 | 0.0155044855898387 | 99.547803617571 | 0.30526700746268 | 92.5 | 0.90243902 | 0.93333333 | 1 | 0.95454545 | 0.82 | 0.82222222 | 0.93333333 | 1 | 0.93333333 | 0.91111111 | 0.86046512 | 0.93333333 | 1 | 0.94382022 | 0.86315789 | 0.903333333333333 | 94.39 |
2 | vit_tiny_r_s16_p8_224 | 0.000518766089389687 | 99.9838501291989 | 0.465771711990237 | 91.25 | 0.84615385 | 0.9375 | 1 | 0.93333333 | 0.8125 | 0.73333333 | 1 | 1 | 0.93333333 | 0.86666667 | 0.78571429 | 0.96774194 | 1 | 0.93333333 | 0.83870968 | 0.883333333333333 | 24.64 |
3 | vit_tiny_patch16_224 | 0.000258662122500031 | 100 | 0.284326805872842 | 93.75 | 1 | 0.9375 | 1 | 1 | 0.78947368 | 0.73333333 | 1 | 1 | 0.93333333 | 1 | 0.84615385 | 0.96774194 | 1 | 0.96551724 | 0.88235294 | 0.916666666666666 | 22.16 |
4 | vit_small_patch8_224_dino | 3.42E-05 | 100 | 0.0970607578463386 | 97.5 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.966666666666666 | 86.74 |
5 | vit_small_patch16_224 | 0.000357729069072556 | 100 | 0.294432922825217 | 96.25 | 0.92857143 | 1 | 1 | 1 | 0.875 | 0.86666667 | 1 | 1 | 1 | 0.93333333 | 0.89655172 | 1 | 1 | 1 | 0.90322581 | 0.95 | 86.73 |
* 6 | vit_base_patch8_224(sgd) | 5.43E-05 | 100 | 0.0298978400795022 | 98.75 | 0.95454545</td> | 1 | 1 | 1 | 0.93478261 | 0.93333333 | 1 | 1 | 1 | 0.95555556 | 0.94382022 | 1 | 1 | 1 | 0.94505495 | 0.97 | 343.3 |
6a | vit_base_patch8_224(adam) | 1.26545023194271 | 46.2855297157622 | 1.36305949687957 | 33.6363636363636 | 0.21568627 | 0.42857143 | 0.42857143 | 0.125 | 0.51111111 | 0.48888889 | 0.6 | 0.06666667 | 0.02222222 | 0.51111111 | 0.29931973 | 0.5 | 0.11538462 | 0.03773585 | 0.51111111 | 0.227777777777777 | 343.3 |
7 | vit_base_patch16_224 | 5.04E-05 | 100 | 0.0669359442894347 | 97.5 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.966666666666666 | 343.27 |
8 | vit_large_patch16_224 | 0.000314225390369969 | 99.9838187702265 | 0.186438377808216 | 96.9298245614035 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.93333333 | 1 | 1 | 1 | 0.93333333 | 0.973099415204678 | 1.21gb |
Among multiple models vit_base_patch8_224(sgd)
stood out in performance with a kappa score of 0.97 on validation set.
Note: The validation set was split evenly from the train set such that all classes were balanced out, and since there was no test set to find the kappa value, we used this validation dataset which was kept unseen from the model.
Model name: vit_base_patch8_224
VALIDATION PREC : [0.95454545 1. 1. 1. 0.93478261]
RECALL : [0.93333333 1. 1. 1. 0.95555556]
F1 : [0.94382022 1. 1. 1. 0.94505495]
ACC : 98.75 KAPPA : 0.97