domain-scraper 🔍
Configure domain-scraper by setting variables to make send-summary work
- DOMAINS_SUBSCRIBERS - comma separated vaild email addresses
- GMAIL_APP_USERNAME - gmail username
- GMAIL_APP_PASSWORD - gmail password
Usage
See how to use this tool with domain-scraper -h
Using docker image
Remember to add variables GMAIL_APP_USERNAME & GMAIL_APP_PASSWORD to docker run command or use env.list file with --env-file flag, e.g.
mkdir -p emails/input emails/archive
image_name="domain-scraper"
app_path="/domain-scraper"
docker run --rm \
--mount type=bind,source=${PWD}/emails,target=${app_path}/emails \
--volume db:${app_path}/db:rw \
-e GMAIL_APP_USERNAME -e GMAIL_APP_PASSWORD -e DOMAINS_SUBSCRIBERS \
"${image_name}" domain-scraper -ps
minikube deployment
eval $(minikube docker-env) # to add image to minikube docker repo
minikube mount $HOME:/hosthome # run in separate terminal
kubectl apply -f secrets.yaml # prepare secrets.yaml file with GMAIL_APP_USERNAME, GMAIL_APP_PASSWORD and DOMAINS_SUBSCRIBERS defined
kubectl create -f cronjob.yaml
TODO:
- move scanned emails to separate dir to avoid duplication
- add requirements.txt file
- add html alternative message template + use it
- add possibility to read INPUT_DIR, ARCHIEVE_DIR, DB_FILE from env
- add option for send-summary to send all emails instead of only new emails
- add setup.cfg, entrypoint and test building
- add Dockerfile, build and test the image
- add variables to deployment with secrets.yaml (email-secrets)
- configure persistent volume to store db
- add kubernetes yaml, verify the deployment
- do some refactoring, especially variable naming (msg, messages_to_send)
- add docstrings
- add types declarations
- add logging and remove all prints
- add option to parse only one email
- create helm chart from the repo
- add coverage measurement
- write unit tests
- evaluate domain parsing - separate ipv4/ipv6 parsing from domains
- add exception handling when file in INPUT_DIR is not email file or does not contain Received header
- add validation for input files, to check if they are actually emails
- add validation for DOMAINS_SUBSCRIBERS