/COVID-19-NL

Scrapes sources daily for new information for the current COVID-19 status of the Netherlands.

Primary LanguageTypeScript

COVID-19 Scraper for NL

The goal is simple get as many historical data for the COVID-19 virus spread in the Netherlands. Currently only the "very sick" are being tested in the Netherlands. Not the ones with "minor symptoms".

If you have an idea please submit an issue!

⚠️ Right now the historical information (before 03-20-2020) come from an independent source. We have no way of actually making sure this data is correct (on municipality level). Please if you have a more reputable source submit an issue.

⚠️ Everything after 03-20-2020 come from the RIVD.nl website directly.

Table of Contents

  1. Projects
  2. Scraper
    1. International CSV
    2. CSV
    3. JSON
  3. History
    1. Country
    2. Municipalities
  4. API
  5. GeoJSON
  6. Contributing
  7. Sources

Projects

Projects currently using this API/data:

Project Description URL Repository
coronamap-nl Map created for tracking the progress of the corona virus by codefor.nl URL codefornl/coronamap-nl

Scraper

This scraper will contact the RIVM website every 2 hours. But also every day at 14:30.

Currently it outputs in 3 formats.

  • international-csv
  • csv
  • json

International CSV

The RIVM csv is in has ; delimiter. We convert this CSV into a "international" format. Which is with the , delimiter. You can find the ; delimimter below.

Examples:

CSV

The RIVM csv is in has ; delimiter. This delimiter works with most tools but not all. Since in America they use a different delimiter: ,

Examples:

JSON

The RIVM csv converted to a JSON format.

Examples:

History

Since we now have historical data for all of the municipalities we generate the progress of COVID-19 in the Netherlands.

Country

Overview of the history of the COVID-19 numbers in the Netherlands as a whole. Every day at around 14:30 we generate a new file containing the latest information. You can view it live here:

Format Link Direct URL
JSON View total-nl-latest.json on Github API URL
CSV View total-nl-latest.csv on Github API URL

Municipalities

Overview of the history per municipality in the netherlands. Every day at around 14:30 we generate a new file containing the latest information. You can view it live here:

Format Link Direct URL
JSON View municipality-history-latest.json on Github API URL
CSV View municipality-history-latest.csv on Github API URL

In depth municipality data

As you can see over here we have lots of folders with a number. This number represents the municipality code.

Files in folder:

File Description
{municipalityCode}/info.json Information about the municipality
{municipalityCode}/{municipalitySlug}.csv CSV file containing history for the municipality
{municipalityCode}/{municipalitySlug}.json JSON file containing history for the municipality

But how do you know what files or currently generated? We generate a files.json "index" file containing all of the municipalities with their respective information and location of the file.

Check it out here: /History/municipalities/files.json

Example contents:

[
    {
        "municipality": "Appingedam",
        "population": 11721,
        "municipalityNr": 3,
        "csv": "3/appingedam.csv",
        "csvLines": 26,
        "json": "3/appingedam.json",
        "jsonLength": 25
    }
]

Example usage:

# Info about all of the available files
curl https://kapulara.github.io/COVID-19-NL/History/municipalities/files.json

# Information about the municipality
curl https://kapulara.github.io/COVID-19-NL/History/municipalities/3/info.json

# If we would like to get the JSON data
curl https://kapulara.github.io/COVID-19-NL/History/municipalities/3/appingedam.json

# If we would like to get the CSV data
curl https://kapulara.github.io/COVID-19-NL/History/municipalities/3/appingedam.csv

API

Currently everything is hosted on Github Pages.

The base url is: https://kapulara.github.io/COVID-19-NL/

Now if you would like to get the latest municipality information you just add /Municipalities/latest.json to that url.

All RAW url's in this document currently make use of that endpoint.

GeoJSON formats

Keep in mind all of these are in format EPSG::28992 tried converting it with https://github.com/perliedman/reproject but no luck. The GeoJSON ended up in Norway.

Contributing

I'm currently using Typescript, yarn and Node.js with NVM.

git clone https://github.com/Kapulara/COVID-19-NL.git
cd COVID-19-NL/
yarn
cp example-dev.env .env
yarn build:watch

After that open the folder with your favorite editor.

To start scraper yarn start or if you would like to regenerate the historical data yarn start:historical

Sources