The project is a proof of concept for collecting data using a spider and generation of JSON files that are easily consumable using common visualization libraries. The repository has two part:
- The spider in crawler folder
- The visualization in root project
Primary data source is W3Schools browser statistics.
Install dependencies for Linux with:
$ cd crawler
$ pip install -r requirements_lin.txt
Install dependencies for Windows with:
$ cd crawler
$ pip install -r requirements_win.txt
From the root of the project go to crawler folder:
$ cd crawler
If you are using Linux OS excute Launch_crawler_lin.sh.
If you are using Windows OS execute Launch_crawler_win.bat
Crawled data will be saved in browser-stats.json
C3.js based on D3.js is used for data visualization.
All the displayed data comes from browser-stats.json generated by the spider automatically.