Kijiji Search
Overview
This will run a search against Kijiji's websetup regularly and notify you by email if there are any new listings. This is useful of items that are hard to find and that don't last long. It does this by using pyppeteer (which uses a real browser) and doing the search the same way that a user would.
You need to generate the query yourself in a real browser and copy the results to data/queries.json
. The file can have multiple queries in it and all of them will be run each cycle.
Setup
make sure you have Pipenv install and then create a new environment for the project
pip install pipenv
pipenv install
pipenv shell
# pipenv --venv
# add above to python.pythonPath in .vscode/settings.json
To be able to send email, the code needs a MailJet api key and secret. You can set them as environment variables or add them to the config file (see config-sample.json
)
export MJ_API_KEY="--api-key--"
export MJ_API_SECRET="--api-secret--"
Build
docker build -t anuaimi/kijiji-search .
If you want to deploy the container to the cloud, you need to push the image to a registry
docker push anuaimi/kijiji-search
Deploying
The simplest way to run this code is to get a small VM at public cloud provider and run it there. The example below uses DigitalOcean. They have VMs that have Docker pre-installed. Note, replace --fingerprint--
with your actual SSH key fingerprint.
doctl compute droplet create --image docker-20-04 --size s-1vcpu-1gb --region tor1 --ssh-keys --fingerprint-- kijiji-search
ssh root@serverIP
Running
You need to define the kijiji search and put the details in the data/queries.json
file.
with secrets
export MJ_API_KEY=""
export MJ_API_SECRET=""
docker run -it --name kijiji-search -e MJ_API_KEY=$MJ_API_KEY -e MJ_API_SECRET=$MJ_API_SECRET -v $PWD/data:/data anuaimi/kijiji-search
You can find a pre-built docker container at https://hub.docker.com/repository/docker/anuaimi/kijiji-search
If you want to run the queries on a regular interval, you can use CRON.
crontab -e
0,10,20,30,40,50 * * * * docker start kijiji-search
Debugging
You can debug docker issues by starting a python container and running the various command in the docker file. Also you can use the following to debug pyppeteer issues
from pyppeteer.launcher import Launcher
' '.join(Launcher().cmd)