GAN-based Object Finding for Robotics
Framework:
Dependencies
python 2.7
[Optional] Torch is needed, if use the pre-trained char-CNN-RNN text encoder.
[Optional] skip-thought is needed, if use the skip-thought text encoder.
In addition, please add the project folder to PYTHONPATH and pip install
the following packages:
prettytensor
progressbar
python-dateutil
easydict
pandas
torchfile
caffe
Data Follow the "Data" process of StackGAN
- Download our preprocessed char-CNN-RNN text embeddings for birds and flowers and save them to
Data/
.
- [Optional] Follow the instructions reedscot/icml2016 to download the pretrained char-CNN-RNN text encoders and extract text embeddings.
- Download the birds and flowers image data. Extract them to
Data/birds/
andData/flowers/
, respectively.
Pretrained Model
- StackGAN for birds trained from char-CNN-RNN text embeddings. Download and save it to
models/
. - StackGAN for flowers trained from char-CNN-RNN text embeddings. Download and save it to
models/
. - StackGAN for birds trained from skip-thought text embeddings. Download and save it to
models/
(Just used the same setting as the char-CNN-RNN. We assume better results can be achieved by playing with the hyper-parameters). - Image Matching model for Cifar100. Download and save it to
models/
.
Run Demos for Bird
- Run
sh demo/birds_demo.sh
to generate bird samples from sentences. The results will be saved toData/birds/example_captions/
.(Need to download the char-CNN-RNN text encoder for birds tomodels/text_encoder/
. Note: this text encoder is provided by reedscot/icml2016). - Run
sh example_birds.sh
in folder "Region-to-Image_Matching/example". The result images will be stored in folder "match_images"
Examples for Region-to-Image Matching for birds conditioned by "A small yellow bird with black wings and a short black pointed beak":
Examples on a Parrot Robot in real environment:
References