/VNLQAC

Visual Natural Language Query Auto-Completion for Estimating Instance Probabilities

Primary LanguagePython

Visual Natural Language Query Auto-Completion for Estimating Instance Probabilities

Samuel Sharpe, Jin Yan, Fan Wu, Iddo Drori

Requirements

  • Python 2.7
  • tensorflow==1.11.0

Downloading Data

To download Visual Genome Data:

data/visual/download_visual_dataset.sh

To download version of ReferIt data from Hu et al.:

data/referit/download_referit_dataset.sh

Downloading VGG

code/util/vgg/download_vgg_params.sh

Build Data

Run the following to set up data for training.

python data/build_referit_data.py
python data/build_visual_data.py

Training

Query Completion:

Set params: code/query_completion/default_params.json

Train: python code/query_completion/trainer.py /path/to/new/experimentdir --data /path/to/training.txt --valdata /path/to/validation.txt

Instance Selection:

Set params: code/instance_selection/default_params.json

Train: python code/instance_selection/train.py /path/to/new/experimentdir --data /path/to/fulldataset.txt --traindata /path/to/training.txt --valdata /path/to/validation.txt --testdata /path/to/testing.txt

Demo

Code to produce some example query completions and instance selections in code/demo.ipynb

Code to produce images with selected instances adapted from Learning to Segment Everything and located here: https://github.com/jinnick/DL_project_NLQAC_Instance_Selection

Example

Aspects of code adapted from: