Predict Tags by Learning Recipe Representation
This is capstone project repository collaborated with Plated. Our goal is to help Plated build a deep learning model to auto-generate recipe tags using both cooking instruction, dish images, and other data sources. We successfully build models on top of instruction data with high AUC, where we applied single-task to make prediction of 9 cuisine tags individually and a multi-task module to obtain comprehensive recipe representations then use the it to predict multiple tags all together.
Data
- Recipe1M
- Cooking Insturctions
- Dish Images
- Tags: includes cuisine, cooking difficulty levels, more
Model
- Recipe1M-instruction data: A Skip-Gram model to learn recipe language embeddings (Domain-Edmbd)
- Cooking instruction data: A Two-Stage LSTM/GRU to obtain intruction representation using self-pretrained recipe language/GloVe embeddings in both single-tasking and multi-tasking manner
- Dish Image: A deep neural network (Resnet 50) to learn recipe image representation
Results
Instruction Model -- K-fold (mean) Validation AUC on Cuisine Tag Prediction under Variance Settings
| Cuisine Category | Tags Percentage | GRU | GRU + Aug | LSTM + Aug | GRU + Domain-Edmbd + Aug | Multi-task |
|---|---|---|---|---|---|---|
| American | 27.35% | 0.80612 | 0.81249 | 0.79381 | 0.69369 | 0.74103 |
| Italian | 23.33% | 0.88027 | 0.91504 | 0.89848 | 0.80436 | 0.85489 |
| Asian | 18.22% | 0.97855 | 0.97919 | 0.97982 | 0.88072 | 0.94860 |
| Latin-Ame | 9.49% | 0.90628 | 0.94837 | 0.85706 | 0.92311 | 0.93433 |
| French | 7.74% | 0.74977 | 0.80471 | 0.77272 | 0.85640 | 0.79605 |
| Mediterranean | 7.66% | 0.73317 | 0.75837 | 0.72589 | 0.75442 | 0.79292 |
| Mid-east | 4.63% | 0.81138 | 0.81850 | 0.78675 | 0.77870 | 0.87369 |
| Indian | 2.35% | 0.78643 | 0.87356 | 0.73456 | 0.87249 | 0.88438 |
| Mexican | 1.36% | 0.67503 | 0.70365 | 0.73999 | 0.74288 | 0.90554 |
Image Model -- K-fold (mean) Validation AUC on Cuisine Tag Prediction
| Cuisine Category | American | Italian | Asian | Latin-Ame | French |
|---|---|---|---|---|---|
| AUC | 0.7719 | 0.8810 | 0.7411 | 0.7235 | 0.7188 |
Recipe Representation Visualization
Related Project
Learning Cross-modal Embeddings for Cooking Recipes and Food Images
Team Member (DataZoo): Tingyan Xiang, Hetian Bai, Jieyu Wang, Cong Liu
Industral Metor: Ph.D Andrew Marchese
