/Shopee-Product-Matching

This repo deals with a Kaggle competition: Shopee - Price Match Guarantee. The purpose is to determine if two products are the same by their images.

Primary LanguageJupyter NotebookMIT LicenseMIT

Shopee Product Matching

GitHub Version Repo Size License

Introduction

This is the project of pattern recognition, with the problem chosen from a Kaggle competition.

Structure

The directory of this project is as follows.

|- bin/
|- doc/
|- input/
  |- shopee-competition-utils/
  |- shopee-product-matching/
  |- text-model-trained/
|- notebook/
|- notebook-image-text/
|- notebook-text/
|- post-process/
|- refs/
|- test/
|- README.md
|- requirements.txt

File Descriptions

  • bin/
    • This folder contains the executable code files
  • doc/
    • This folder contains the report of this project written with LaTeX
  • input/
    • This folder includes all data and utilities of this project
  • input/shopee-competition-utils/
    • This folder contains utilities of this project
  • input/shopee-product-matching/
    • This folder contains all data files of this project
  • input/text-model-trained/
    • This folder stores trained text model (the best single model)
  • notebook/
    • This folder contains jupyter notebooks for training and inference of image models
  • notebook-image-text/
    • This folder contains jupyter notebooks for ensembles of image and text models
  • notebook-text/
    • This folder contains jupyter notebooks for training and inference of text models
  • post-process/
    • This folder contains post process codes and results
  • refs/
    • This folder contains some reference papers
  • test/
    • This folder includes test jupyter notebooks
  • README.md
    • This file serves as user manual of this project
  • requirements.txt
    • This file contains python packages used in this project

Quick Start

First, please check the following basic requirements. Note that you should have GPU memory >= 10 GiB.

  • System: Ubuntu 18.04.4 LTS
  • Python: 3.8.3
  • GPU: Nvidia GeForce RTX 3080

Then, please install required packages included in requirements.txt via pip.

~$ pip install -r requirements.txt

Second, download Shopee dataset into specific directory via wget and unzip it via unzip.

~$ cd input/shopee-product-matching/
~/input/shopee-product-matching$ wget -O shopee-dataset https://cloud.tsinghua.edu.cn/f/5c7ba8c55e04478d86d9/?dl=1
~/input/shopee-product-matching$ unzip shopee-dataset

Third, download pretrained model into specific directory via wget and unzip it via unzip.

~/input/shopee-product-matching$ cd ..
~/input$ cd text-model-trained/
~/input/text-model-trained$ wget -O pretrained-model https://cloud.tsinghua.edu.cn/f/a6df401f7fd34248a7f9/?dl=1
~/input/text-model-trained$ unzip pretrained-model

Then, you can get inference results predicted by the best performing pretrained model stored in input/text-model-trained/paraphrase-xlm-r-multilingual-v1_epoch8-bs16x1_margin_0.8.pt via following command.

~/input/text-model-trained$ cd ..
~/input$ cd ..
~$ cd bin/
~/bin$ python best_model_inference.py

Expected output is listed as follows.

2021-05-30 15:01:17.692841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Building Model Backbone for sentence-transformers/paraphrase-xlm-r-multilingual-v1 model, margin = 0.8
paraphrase-xlm-r-multilingual-v1_epoch8-bs16x1_margin_0.8.pt
get_bert_embeddings: 100%|████████████████████| 216/216 [00:04<00:00, 53.66it/s]
Searching best threshold...
threshold = 0.01 -> f1 score = 0.5027464921403513, recall = 0.36920819465293997, precision = 0.9980143086581982
threshold = 0.02 -> f1 score = 0.51908882853155, recall = 0.38723567961541383, precision = 0.9974181145666033
threshold = 0.03 -> f1 score = 0.5349359255261192, recall = 0.4057179921329236, precision = 0.996348441552121
threshold = 0.04 -> f1 score = 0.5483479866781269, recall = 0.4214936699090649, precision = 0.9959288460602522
threshold = 0.05 -> f1 score = 0.5615151720270062, recall = 0.4369220353693416, precision = 0.9951892525151396
threshold = 0.06 -> f1 score = 0.572522496832261, recall = 0.4499498851171465, precision = 0.9941684222236128
threshold = 0.07 -> f1 score = 0.5828269001282261, recall = 0.4630646540637196, precision = 0.9928750460616429
threshold = 0.08 -> f1 score = 0.5923589345616298, recall = 0.47527675084010945, precision = 0.9918085017625099
threshold = 0.09 -> f1 score = 0.6028817489215734, recall = 0.48869024336912714, precision = 0.9896568146896665
threshold = 0.1 -> f1 score = 0.6140175003791694, recall = 0.5028970367148061, precision = 0.9877276140416749
threshold = 0.11 -> f1 score = 0.62386027143292, recall = 0.5154268014490383, precision = 0.9860473200814861
threshold = 0.12 -> f1 score = 0.6329263729093964, recall = 0.5272049660782009, precision = 0.984572014915861
threshold = 0.13 -> f1 score = 0.6415742308480433, recall = 0.5385016389206072, precision = 0.9825331933522911
threshold = 0.14 -> f1 score = 0.6509542129922906, recall = 0.5505961898060125, precision = 0.9806906117681413
threshold = 0.15 -> f1 score = 0.6613230553681155, recall = 0.5637775207575203, precision = 0.9796416230384691
threshold = 0.16 -> f1 score = 0.6707598680559608, recall = 0.576318907369572, precision = 0.9780584035292751
threshold = 0.17 -> f1 score = 0.6797794458855402, recall = 0.588864470205251, precision = 0.9756061806981647
threshold = 0.18 -> f1 score = 0.6893698890550152, recall = 0.601842795280555, precision = 0.9738874730661853
threshold = 0.19 -> f1 score = 0.6977106466800822, recall = 0.6135103219186495, precision = 0.9715956493661273
threshold = 0.2 -> f1 score = 0.705557075500399, recall = 0.6244402770499733, precision = 0.9684070249116241
threshold = 0.21 -> f1 score = 0.7148727538141246, recall = 0.6371927778462941, precision = 0.9658399046484898
threshold = 0.22 -> f1 score = 0.7230899823376746, recall = 0.6487996833863309, precision = 0.9637884795619662
threshold = 0.23 -> f1 score = 0.7312534197718673, recall = 0.660648264078936, precision = 0.9613948045324894
threshold = 0.24 -> f1 score = 0.7396437624371135, recall = 0.6724451072826095, precision = 0.9592029407622928
threshold = 0.25 -> f1 score = 0.747392900496289, recall = 0.6841417369370686, precision = 0.9562429090468643
threshold = 0.26 -> f1 score = 0.7538771362762666, recall = 0.6945884160419811, precision = 0.9531772548970213
threshold = 0.27 -> f1 score = 0.759814282983688, recall = 0.7047343504781323, precision = 0.9493948631602904
threshold = 0.28 -> f1 score = 0.7665991383537749, recall = 0.7153999383729984, precision = 0.9469576309760154
threshold = 0.29 -> f1 score = 0.7736130938560478, recall = 0.7272729195702812, precision = 0.9432776941237178
threshold = 0.3 -> f1 score = 0.779379946678424, recall = 0.7372108343822307, precision = 0.9397291852517998
threshold = 0.31 -> f1 score = 0.785689762548553, recall = 0.7495067019249689, precision = 0.9346678389263784
threshold = 0.32 -> f1 score = 0.7901121078804931, recall = 0.7591831656237933, precision = 0.9294421088474973
threshold = 0.33 -> f1 score = 0.7937219879211048, recall = 0.7685087876110084, precision = 0.9231231060422324
threshold = 0.34 -> f1 score = 0.7981994636517731, recall = 0.7784750110025295, precision = 0.9181930097375393
threshold = 0.35 -> f1 score = 0.8021934727794962, recall = 0.788138692028146, precision = 0.913231398437369
threshold = 0.36 -> f1 score = 0.8043672987939667, recall = 0.7966856405948681, precision = 0.9060587264079084
threshold = 0.37 -> f1 score = 0.8072324527609893, recall = 0.8049980232592455, precision = 0.9001233290473561
threshold = 0.38 -> f1 score = 0.8082144423775688, recall = 0.8127577760142956, precision = 0.8925303891940227
threshold = 0.39 -> f1 score = 0.809614292997088, recall = 0.8219766435514532, precision = 0.8837434600443329
threshold = 0.4 -> f1 score = 0.8099172083192242, recall = 0.8304567849311008, precision = 0.8735989203634735
threshold = 0.41 -> f1 score = 0.8108409376275685, recall = 0.8388584092063919, precision = 0.8659611695732178
threshold = 0.42 -> f1 score = 0.8095973922481051, recall = 0.8462106247388016, precision = 0.8552502716038131
Best threshold = 0.41
Best f1 score = 0.8108409376275685
________________________________
Searching best min2 threshold...
min2 threshold = 0.41 -> f1 score = 0.8108409376275685, recall = 0.8388584092063919, precision = 0.8659611695732178
min2 threshold = 0.415 -> f1 score = 0.8116496526062353, recall = 0.8403184763694813, precision = 0.8650851292753641
min2 threshold = 0.42 -> f1 score = 0.8119074227720214, recall = 0.8410038967877096, precision = 0.8639170755448925
min2 threshold = 0.425 -> f1 score = 0.8123004735811915, recall = 0.8417610459022831, precision = 0.8630410352470388
min2 threshold = 0.43 -> f1 score = 0.8130044523622618, recall = 0.8427455483322522, precision = 0.8625300117399575
min2 threshold = 0.435 -> f1 score = 0.813370627936497, recall = 0.8435850869510287, precision = 0.8616539714421039
min2 threshold = 0.44 -> f1 score = 0.8137780562020066, recall = 0.8444051580076306, precision = 0.860923937860559
min2 threshold = 0.445 -> f1 score = 0.8141418339201425, recall = 0.8450152575007785, precision = 0.8604129143534777
min2 threshold = 0.45 -> f1 score = 0.814340224942011, recall = 0.8454739619345158, precision = 0.8597558841300874
min2 threshold = 0.455 -> f1 score = 0.8147394991728866, recall = 0.846241621896734, precision = 0.8590258505485426
min2 threshold = 0.46 -> f1 score = 0.8152679169592982, recall = 0.8471638976547522, precision = 0.8581498102506889
min2 threshold = 0.465 -> f1 score = 0.8158717663147901, recall = 0.8481178082013042, precision = 0.8575657833854531
min2 threshold = 0.47 -> f1 score = 0.8160661996454475, recall = 0.8486886249350645, precision = 0.8566897430875993
min2 threshold = 0.475 -> f1 score = 0.8162040916361648, recall = 0.8492118156685049, precision = 0.8555946927152823
min2 threshold = 0.48 -> f1 score = 0.8164556529915369, recall = 0.8498128766506434, precision = 0.8548646591337374
min2 threshold = 0.485 -> f1 score = 0.8168963205149902, recall = 0.8506159135903425, precision = 0.8543536356266561
min2 threshold = 0.49 -> f1 score = 0.8172026120389777, recall = 0.8511919448496853, precision = 0.8536966054032658
min2 threshold = 0.495 -> f1 score = 0.8170725965344552, recall = 0.8516591663418739, precision = 0.8520175281657126
Best min2 threshold = 0.49
Best f1 score after min2 = 0.8172026120389777
get_bert_embeddings: 100%|████████████████████| 216/216 [00:03<00:00, 61.97it/s]
Test f1 score = 0.8211137129836418, recall = 0.8483214531160035, precision = 0.8724217517245711
Test f1 score after min2 = 0.8285356189862213, recall = 0.8627090001536798, precision = 0.8590660372303366
Searching best threshold...
threshold = 0.01 -> f1 score = 0.7487150548288662, recall = 0.6982848267193497, precision = 0.94961581998194
threshold = 0.02 -> f1 score = 0.7729656472412717, recall = 0.7452999015407432, precision = 0.9285892501433977
threshold = 0.03 -> f1 score = 0.7883627503081633, recall = 0.7729596755071844, precision = 0.9176470919190673
threshold = 0.04 -> f1 score = 0.799624631051321, recall = 0.7941936289620009, precision = 0.9081190304415624
threshold = 0.05 -> f1 score = 0.807271975602742, recall = 0.8103105651668139, precision = 0.8999584102541189
threshold = 0.06 -> f1 score = 0.8119752656268617, recall = 0.8213284726087994, precision = 0.8922592984801269
threshold = 0.07 -> f1 score = 0.815824492694392, recall = 0.8300783159368174, precision = 0.8874437182905293
threshold = 0.08 -> f1 score = 0.8174777546207815, recall = 0.8363261240829811, precision = 0.882693432678336
threshold = 0.09 -> f1 score = 0.8191989132279784, recall = 0.8413581814356539, precision = 0.8793886151244932
threshold = 0.1 -> f1 score = 0.8200570524655117, recall = 0.8456916825158566, precision = 0.8755977688010829
threshold = 0.11 -> f1 score = 0.820444773296711, recall = 0.8500903856971065, precision = 0.8716004677210466
threshold = 0.12 -> f1 score = 0.8207133839305376, recall = 0.8527055624654046, precision = 0.8690345416491833
threshold = 0.13 -> f1 score = 0.8207064158101671, recall = 0.8556725213105225, precision = 0.8660346294793263
Best threshold = 0.12
Best f1 score = 0.8207133839305376
________________________________
Searching best min2 threshold...
min2 threshold = 0.12 -> f1 score = 0.8207133839305376, recall = 0.8527055624654046, precision = 0.8690345416491833
min2 threshold = 0.125 -> f1 score = 0.8207949622863164, recall = 0.8529671578321247, precision = 0.8686695248584108
min2 threshold = 0.13 -> f1 score = 0.8208703604636274, recall = 0.8530701112859325, precision = 0.8685965215002562
min2 threshold = 0.135 -> f1 score = 0.820943363821782, recall = 0.8532161180022413, precision = 0.8685235181421018
min2 threshold = 0.14 -> f1 score = 0.8211591305239287, recall = 0.8535100969098862, precision = 0.8682315047094838
min2 threshold = 0.145 -> f1 score = 0.8212838102762912, recall = 0.8536840882468212, precision = 0.867939491276866
min2 threshold = 0.15 -> f1 score = 0.8212804034529105, recall = 0.8537147708176397, precision = 0.8677204812024025
Best min2 threshold = 0.145
Best f1 score after min2 = 0.8212838102762912
CFG.BEST_THRESHOLD after INB is 0.12
CFG.BEST_THRESHOLD_MIN2 after INB is 0.145
Test f1 score after INB = 0.834544833312457, recall = 0.8656430033625991, precision = 0.8770051228664779

Among the long output, the following three lines are of the most importance, which give inference results on test dataset.

Test f1 score = 0.8211137129836418, recall = 0.8483214531160035, precision = 0.8724217517245711
Test f1 score after min2 = 0.8285356189862213, recall = 0.8627090001536798, precision = 0.8590660372303366
Test f1 score after INB = 0.834544833312457, recall = 0.8656430033625991, precision = 0.8770051228664779

Fourth, you can get the best image model trained via following command. Note that once you run this command, the stored model input/text-model-trained/paraphrase-xlm-r-multilingual-v1_epoch8-bs16x1_margin_0.8.pt will be replaced by newly trained one.

~/bin$ python best_model_train.py

Fifth, you can get inference results predicted by this newly trained model via following command.

~/bin$ python best_model_inference.py

What's more, you can also use this trained model to make inference on test file input/shopee-product-matching/test.csv via following command. Note that test.csv should be replaced with a file like train.csv. Specifically, test.csv should possess label_group property and include more than 50 samples, otherwise the following python script can not compute f1 score and other criteria, and thus you will get an error.

~/bin$ python best_model_inference_using_test_csv.py

Finally, if you have any question in running the codes in this project, please do not hesitate to email me: yangjx20@mails.tsinghua.edu.cn.