/paradeller

Paradeller is a robo-poet that scours Twitter in search of fodder for paradelle poems

Primary LanguageJupyter Notebook

✒️ Paradeller

Paradeller is a robo-poet that scours Twitter in search of fodder for paradelle poems.

Setup

1) Install Git Large File Storage

# MacOS
brew install git-lfs

# Linux
sudo apt-get install git-lfs

2) Clone Repo

git clone https://github.com/dustinmichels/paradeller.git
cd paradeller

# Initialize large file storage
git lfs install

3) Install Python Dependencies

# dev
pipenv install --dev
pipenv shell
jupyter labextension install @jupyter-widgets/jupyterlab-manager

# prod
pip -r requirements.txt

4) [Optional] Twitter API Keys

If you will be scraping tweets:

  • Create Twitter API credentials
  • Copy keys template: cp paradeller/keys.template.py paradeller/keys.py
  • Populate keys.py with API credentials

Usage

Scraping Tweets

Scrape tweets using Tweepy.

python -m paradeller.scrape
python -m paradeller.scrape 100
  • Optional CLI argument is the number of iterations of get_tweets to perform.
  • Each iteration will scrape about 100 tweets, filter that collection down, then save to data/archive.json.
  • Scraper will automatically pause when rate limits are hit and resume when possible.

Can easily run with default of 10,000 tweets using:

./scrape.sh

Analyzing Tweets

To search for paradelles in the saved tweets, use paradeller/run.py.

python -m paradeller.run
python -m paradeller.run 1000
  • Optional CLI argument is number of ids to pair off as initial pairs

Can easily run with default of 1,000 ids using:

./search.sh

Dev Notes

To generate requirements.txt from Pipefile:

pipenv lock -r > requirements.txt