sourcecatcher.com

A reverse image search tool for InSomnia

See the Reddit release thread for more information about Sourcecatcher

Setup

Sourcecatcher has been tested on Arch Linux and Ubuntu 19.04. It should also work on many other Linux distros.

Install requirements

Required system packages

nitter-scraper

Python modules

Sourcecatcher has been tested on python 3.7, but should work on other recent versions of python 3 also

python -m venv sourcecatcher_venv           # create python virtual environment
source ./sourcecatcher_venv/bin/activate
pip install -r requirements.txt

Create `config.yaml`

config.yaml contains runtime information needed by Sourcecatcher. You will need to apply for the Twitter api and create an app.

media_dir: "path/to/directory/to/store/images"

users:
  - "twitter user to scrape 1"
  - "user 2"
  - "user 3"

Create and update the database

The bash scripts are used for creating and updating Sourceactcher's internal database

./initial.sh      # create/recreate the database. scrapes all users, may take a few hours
./update.sh       # fetch the latest tweets and update the database, also backups the current databse
./backup.sh       # make a backup of the current database now

Set up web server

I use nginx + gunicorn, with systemd to manage it. You could use anything else if you want though. Note the systemd service paths may be different for your distribution.

`/etc/systemd/system/sourcecatcher.service`

This service runs Sourcecatcher

[Unit]
Description=Gunicorn instance to serve sourcecatcher
After=network.target

[Service]
User=YOUR_USER
Group=www-data
WorkingDirectory=/PATH/TO/sourcecatcher
Environment="PATH=/PATH/TO/sourcecatcher/sourcecatcher_venv/bin:/usr/bin"
ExecStart=/PATH/TO/sourcecatcher/sourcecatcher_venv/bin/gunicorn -c gunicorn.config.py -w 9 -b unix:sourcecatcher.sock -m 007 wsgi:app

[Install]
WantedBy=multi-user.target

`/etc/systemd/system/sourcecatcher_update.service`

This service updates Sourcecatcher with new tweets

[Unit]
Description=Update sourcecatcher
After=network.target

[Service]
User=YOUR_USER
Group=www-data
WorkingDirectory=/PATH/TO/sourcecatcher
Environment="PATH=/PATH/TO/sourcecatcher/sourcecatcher_venv/bin:/home/YOUR_USER/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
ExecStart=/bin/bash /PATH/TO/sourcecatcher/update.sh

[Install]
WantedBy=sourcecatcher_update.timer

`/etc/systemd/system/sourcecatcher_update.timer`

Periodically update Sourcecatcher

[Unit]
Description=update sourcecatcher

[Timer]
OnBootSec=15min
OnUnitActiveSec=2hr

[Install]
WantedBy=timers.target

Start systemd services

sudo systemctl enable --now sourcecatcher.service
sudo systemctl enable --now sourcecatcher_update.timer

evanc577/sourcecatcher