/recipe_generation_from_an_image_sequence.pytorch

Recipe generation models from an image sequence (Pytorch implementation).

Primary LanguagePython

recipe_generation_from_an_image_sequence.pytorch

This repo is implementations of neural recipe generators using PyTorch.
Now we implemented the following 5 models:

Note We could not implement the SSiD and SSiL perfectly due to lack of details of a finite state machine (FSM).

Requirements

  1. Python 3.7
  2. CUDA 10.2 and cuDNN v7.6
  3. PyTorch 1.5.0
  4. install other required modules
pip install -r requirements.txt

Data Preparation

The dataset used in this repo is the story boarding dataset (https://www.aclweb.org/anthology/P19-1606.pdf). As mentioned here, the original scripts did not save the train/val/test splits. Thus, this scripts lead you to download the data from instructable.com and snapguide.com and split them into train/val/test datasets.

1. Downloading the story boarding dataset

Follow this repo.
Then, Copy instructables.json and snapguide.json to data/ directory.

2. Preprocessing the dataset

The following scripts lead you to split the dataset with train/val/test.

./make_directory.sh
cd preprocess
python build_dataset.py -d ./data/ --dl
python convert_pickles_into_trainable_format.py -d ./data -o ./data/features/

Training and Validation

Exsisting 5 recipe generation models are divided into two types: scratch models (Images2seq, GLAC Net) and pretraining-based models (SSiD, SSiL, RetAttn).
Note Training scripts are under construction. Now I only implemented the GLAC Net.

Scratch models

Scratch models learn to generate a recipe from random weights. You can train these models as:

python train.py -m {glacnet/images2seq}

Pretraining-based models

[WIP]

Testing

[WIP]

Citation

@misc{taichi19recipe,
    author = {Taichi Nishimura},
    title = {recipe_generation_from_an_image_sequence.pytorch},
    howpublished = {https://github.com/misogil0116/recipe_generation_from_an_image_sequence.pytorch},
    year = {2020}
}