GAN-based Object Finding for Robotics
Framework:
Dependencies
python 2.7
[Optional] Torch is needed, if use the pre-trained char-CNN-RNN text encoder.
[Optional] skip-thought is needed, if use the skip-thought text encoder.
In addition, please add the project folder to PYTHONPATH and pip install the following packages:
prettytensorprogressbarpython-dateutileasydictpandastorchfilecaffe
Data Follow the "Data" process of StackGAN
- Download our preprocessed char-CNN-RNN text embeddings for birds and flowers and save them to
Data/.
- [Optional] Follow the instructions reedscot/icml2016 to download the pretrained char-CNN-RNN text encoders and extract text embeddings.
- Download the birds and flowers image data. Extract them to
Data/birds/andData/flowers/, respectively.
Pretrained Model
- StackGAN for birds trained from char-CNN-RNN text embeddings. Download and save it to
models/. - StackGAN for flowers trained from char-CNN-RNN text embeddings. Download and save it to
models/. - StackGAN for birds trained from skip-thought text embeddings. Download and save it to
models/(Just used the same setting as the char-CNN-RNN. We assume better results can be achieved by playing with the hyper-parameters). - Image Matching model for Cifar100. Download and save it to
models/.
Run Demos for Bird
- Run
sh demo/birds_demo.shto generate bird samples from sentences. The results will be saved toData/birds/example_captions/.(Need to download the char-CNN-RNN text encoder for birds tomodels/text_encoder/. Note: this text encoder is provided by reedscot/icml2016). - Run
sh example_birds.shin folder "Region-to-Image_Matching/example". The result images will be stored in folder "match_images"
Examples for Region-to-Image Matching for birds conditioned by "A small yellow bird with black wings and a short black pointed beak":
Examples on a Parrot Robot in real environment:
References