Tweak Story is an open-source project towards generating controllable Instagram captions, powered by PyTorch, Transformers and Streamlit. We apply attention-based condition LSTM network to generate engaging captions that can be tweaked by our users. With different attributes we offer, user can easily mix and match different combination of attributes for their own flavor. In current stage, we offer sentence length and emoji flag as available attributes.
This project has 4 major parts :
app.py- Main script for launching our Streamlit appsrc- This folder contains all utility scripts for powering the appckpts- This folder contains model checkpoint, word map and configuration file for our modeldemo- This folder contains example images you can try out in our app
Download the following files and place them into ckpts folder
Clone the repo and navigate to the repo:
git clone https://github.com/namanphy/Controllable-Image-Captioning-App.git
cd stylised-controllable-image-captioning-StreamlitApp
Build the image and run the docker container locally:
docker build -t ctrl-img-cap-streamlit:latest .
docker run -d -p 8501:8501 ctrl-img-cap-streamlit:latest
Navigate to http://localhost:8501 for the app. (Streamlit runs on port 8501 by default)
Note : When you'll run the container for the first time it will download a resnet101 model. The Streamlit app is run in CPU.
To stop the container :
docker ps
docker stop <containerId>
Clone the repo and navigate to the repo directory:
git clone https://github.com/namanphy/Controllable-Image-Captioning-App.git
cd stylised-controllable-image-captioning-StreamlitApp
Make a python3.6 conda environment and install requirements:
conda create -n app-env python=3.6
conda activate app-env
pip install -r requirements.txt
Run below command to launch the Streamlit app:
streamlit run app.py
Navigate to http://localhost:8501 for the app.
pytest -s tests
This project is a part of Data Science Incubator (Summer 2020) organized by Made With ML, jointly developed by Alex Lau and Naman Bhardwaj. We constantly look for better generation quality and deployment strategy. We welcome your contributions and please contact us if you do!
Our work is mainly based upon the following published research:
@article{Xu2015show,
title={Show, Attend and Tell: Neural Image Caption Generation with Visual Attention},
author={Xu, Kelvin and Ba, Jimmy and Kiros, Ryan and Cho, Kyunghyun and Courville, Aaron and Salakhutdinov, Ruslan and Zemel, Richard and Bengio, Yoshua},
journal={arXiv preprint arXiv:1502.03044},
year={2015}
}
@article{Ficler2017show,
title={Controlling Linguistic Style Aspects in Neural Language Generation},
author={Jessica Ficler and Yoav Goldberg},
journal={arXiv preprint arXiv:1707.02633},
year={2017}
}