fundgrube-crawler [WIP]

Quick and dirty Crawler to find deals in Fundgrube by MediaMarkt and Saturn because their shop systems are so horrible.

Implemented in Golang and using MongoDB as persistence since I usually don't use these. :)

Use Case

Cross compiled and deployed on a raspberry pi.
MongoDB Atlas free tier used for persistence.
Is currently started as a script that's configured with env vars. Will fetch all Postings in a Category and search for matching Deals.
Alerts via email when new Deals are found.

Environmental Variables

name	desc	default
`MONGODB_URI`	-	`mongodb://localhost:27017`
`MONGODB_USERNAME`	-	`root`
`MONGODB_PASSWORD`	-	`example`
`MONGODB_DB`	-	`fundgrube`
`MONGODB_COLLECTION_POSTINGS`	-	`postings`
`MONGODB_COLLECTION_OPERATIONS`	-	`operations`
`FIND_ALL`	ignore last run and search in all postings	`false`
`LIMIT_OUTLETS`	only fetch 5 first outlets (for development)	`false`
`LOG_TO_FILE`	log to /tmp/fundgrube.txt instead of stdout	`false`
`MOCKED_POSTINGS`	mock response from api	`false`
`SKIP_CRAWLING`	skip fetching postings from api	`false`
`FAST_CRAWLING`	stop crawling api when no new postings on current page	`false`
`LOG_LEVEL`	levels: trace, debug, info, warn, error, fatal, panic	`info`

API peculiarities

There is only a /api/postings endpoint known to me, but it also returns a list of outlets and brands in the response.
Requests with a limit >= 100 always return the first page.
Requests with an offset > 990 return 422 Unprocessable Entity, so you need to iterate over brands or outlets to see all postings.
I assume that postings are sorted by descending creation date in the API. So it's possible to implement a "fast refresh" by cancelling the update when a page contains no new postings.

Shell script

#1 inspired me to quickly hack my initial idea that solves the same use case with a bash script. Simply run SEARCH_REGEX="sony walkman" ./_sh/fundgrube-crawler.sh to search for matching postings. The script creates a TSV file with previous results in /tmp and will therefore only discover new postings.

MongoDB migrations

cmd/fundgrube-migrate/main.go contains poor man's db migrations and a tooling to clean up obsolete db entries.

Provide a filterString and updateString to perform a migration on the postings collection with MIGRATE=true make migrate.
Provide a filterString to delete entries from the postings collection with CLEANUP=true make migrate.

If the env var is not provided a dry run with the filterString is performed in both cases.