This project contains an assortment of API scrapers for various services that I use personally and professionally. The results of the scrapes are stored in Postgres, where tools such as Metabase, Google Data Studio, and others can be used to perform analysis.
Though I've structured and built this project around my own needs, please feel free to use and contribute back if you find it to be useful!
There may be large, breaking changes at any point in time.
- A Postgres DB. 10.x or higher.
- Go 1.14+ or higher.
In absence of tagged releases, the best bet for installing Scrapenstein is go get:
go get -u github.com/gtaylor/scrapensteinYou can then run the scrapeinstein command.
The included CLI is designed with exploration in mind. Run the scrapeinstein command and take a look at the sub-commands within.
These sources are offered as-is with no official support provided.
Scrapenstein is licensed under the MIT License.