MagicTheGathering Scrapy Project

This project is configured to be hosted on Scrapy Cloud.

It uses Zyte Smart Proxy Manager as proxy service.

This project uses Spidermon for monitoring purposes. It is also set up to send messages via a telegram bot. The dataset can be found here.
A Jupyter Notebook with some EDA on that data can be found here.

scryfall

The spider can be ran with

python3 -m scrapy crawl scryfall -a pages=5 -O scryfall.csv

Arguments

With -a you can specify arguments for the spider.

argument	type	description
pages	int	number of pages to scrape

Setting up locally

When setting up this project locally you must create a .env file with the following data:

setting	description
ZYTE_SMARTPROXY_APIKEY	your smart proxy manager api key
SPIDERMON_TELEGRAM_SENDER_TOKEN	authorization token for your telegram bot
SPIDERMON_TELEGRAM_RECIPIENTS	@channelname, chat it or group id

Deploy to Scrapy Cloud

There's a shortcut in the Makefile, just running make deploy will deploy the project to Scrapy Cloud (given that you provided the project ID in scrapinghub.yml).Don't forget to add the following settings in your cloud project's settings:

setting	description
ZYTE_SMARTPROXY_APIKEY	your smart proxy manager api key
SPIDERMON_TELEGRAM_SENDER_TOKEN	authorization token for your telegram bot
SPIDERMON_TELEGRAM_RECIPIENTS	@channelname, chat it or group id

Also,