A minimal region scraper for spaceweatherlive. It will scrap regions from START_REGION (atm hardcoded to 8124) to END_REGION (atm hardcoded to 13425) and output the region info and the top 10 info to two separate csv files.
A recent (18.17 as of this writing) version of NodeJS
- Clone this repository
- Install packages (
npm -ci) - Run the program (
npm run start)
If you want to observe the program progress as it runs run it with the --verbose option (npm run start -- --verbose)
The program outputs three files:
sunspot-regions.csv: the main output data file for each regiontop10.csv: the secondary output data file for the top 10 sunspots in said region. Important: Not all regions contain top 10 informationerrors.txt: the URLs that couldn't be scraped due to format misalignment, non existing information, etc.
Many thanks to Giorgos Karanikas for his technical contribution on achieving a proper scraping result.