This is the data fetcher-python repository for the OxCOVID19 Database, a large, single-centre, multimodal database consisting of information relating to COVID-19 pandemic.
OxCOVID19 Project https://covid19.eng.ox.ac.uk/ aims to increase our understanding of the Covid-19 pandemic and elaborate possible strategies to reduce the impact on the society through the combined power of Statistical and Mathematical Modelling, and Machine Learning techniques. OxCOVID19 data source fetchers written in Python3.
Cite as: Adam Mahdi, Piotr Błaszczyk, Paweł Dłotko, Dario Salvi, Tak-Shing Chan, John Harvey, Davide Gurnari, Yue Wu, Ahmad Farhat, Niklas Hellmer, Alexander Zarebski, Lionel Tarassenko, Oxford COVID-19 Database: multimodal data repository for understanding the global impact of COVID-19. University of Oxford, 2020 medRxiv doi: 10.1101/2020.08.18.20177147.
| Name | Country | Data source | Terms of Use |
|---|---|---|---|
| GOOGLE_MOBILITY | World | Google COVID-19 Community Mobility Reports | Attribution required |
| APPLE_MOBILITY | World | COVID‑19 Mobility Trends Reports - Apple | |
| GOVTRACK | World | Oxford COVID-19 Government Response Tracker | |
| WEATHER | World | MET Informatics Lab | |
| AUS_C1A | Australia | covid-19-au.com | Strictly for educational and academic research purposes |
| BEL_LE | Belgium | Laurent Eschenauer | CC0 1.0 Universal (CC0 1.0) Public Domain Dedication |
| BEL_SCI | Belgium | Epistat | |
| BRA_MSHM | Brazil | Ministério da Saúde (Brasil) | CC0 1.0 Universal |
| CAN_GOV | Canada | Government of Canada | Attribution required, non-commercial use |
| CHE_OPGOV | Switzerland | Swiss Cantons and the Principality of Liechtenstein | CC 4.0 |
| CHN_ICL | Mainland China | MRC Centre for Global Infectious Disease Analysis | CC BY NC ND 4.0 |
| DEU_JPGG | Germany | Jan-Philip Gehrcke | MIT |
| ESP_ISCIII | Spain | Instituto de Salud Carlos III | |
| ESP_MS | Spain | Ministerio de Sanidad, Consumo y Bienestar Social | "Desnaturalización" prohibited, citation required |
| ESP_MSVP | Spain | Ministerio de Sanidad, Consumo y Bienestar Social | Apache License 2.0 |
| EU_ZH | Austria | Covid19-eu-zh | MIT |
| EU_ZH | Czech republic | Covid19-eu-zh | MIT |
| EU_ZH | Hungary | Covid19-eu-zh | MIT |
| EU_ZH | Ireland | Covid19-eu-zh | MIT |
| EU_ZH | Germany | Covid19-eu-zh | MIT |
| EU_ZH | Norway | Covid19-eu-zh | MIT |
| EU_ZH | Poland | Covid19-eu-zh | MIT |
| EU_ZH | Sweden | Covid19-eu-zh | MIT |
| EU_ZH | Slovenia | Covid19-eu-zh | MIT |
| FRA_SPF | France | Santé publique France | License Ouverte/Open License 2.0 |
| FRA_SPFCG | France | Cédric Guadalupe | GPL 3.0 |
| GBR_NIDH | UK - Northern Ireland | Department of Health (Northern Ireland) | |
| GBR_PHE | UK - England | Public Health England | Open Government Licence v3.0 |
| GBR_PHS | UK - Scotland | Scottish Government | GPL 3.0 |
| GBR_PHTW | UK - | Tom White | The Unlicense |
| GBR_PHW | UK - Wales | Public Health Wales | Open Government Licence v3.0 |
| IDN_GTPPC | Indonesia | Satuan Tugas Penanganan COVID-19 (Indonesia) | Standard "all rights reserved" notice. No licensing information. |
| IND_COVIND | India | COVID-19 India Org Data Operations Group | GPL 3.0 |
| IRL_HSPC | Ireland | Health Protection Surveillance Centre | No license specified' - T&C state not for commercial use. T&C very focussed on OSI information, which is mapping information that we do not use |
| ITA_PC | Italy | Protezione Civile | CC BY 4.0 |
| ITA_PCDM | Italy | Protezione Civile | CC0 1.0 Universal |
| JPN_C1J | Japan | Shane Reustle | |
| JPN_C1JACD | Japan | COVID-19 Japan Anti-Coronavirus Dashboard | CC BY |
| KOR_DS4C | South Korea | Jihoo Kim | CC BY-NC-SA 4.0 |
| LAT_DSRP | Argentina | Data Science Research Peru | CC BY-NC-SA 4.0 |
| LAT_DSRP | Brazil | Data Science Research Peru | CC BY-NC-SA 4.0 |
| LAT_DSRP | Chile | Data Science Research Peru | CC BY-NC-SA 4.0 |
| LAT_DSRP | Colombia | Data Science Research Peru | CC BY-NC-SA 4.0 |
| LAT_DSRP | Dominican Republic | Data Science Research Peru | CC BY-NC-SA 4.0 |
| LAT_DSRP | Ecuador | Data Science Research Peru | CC BY-NC-SA 4.0 |
| LAT_DSRP | Mexico | Data Science Research Peru | CC BY-NC-SA 4.0 |
| LAT_DSRP | Peru | Data Science Research Peru | CC BY-NC-SA 4.0 |
| MYS_MHYS | Malaysia | Young Shung | Public Domain Dedication and License v1.0 |
| NGA_CDC | Nigeria | Nigeria Centre for Disease Control | |
| NGA_HERA | Nigeria | Humanitarian Emergency Response Africa | CC BY |
| NGA_SO | Nigeria | Nigeria Centre for Disease Control | No licensing information. |
| NLD_CW | Netherlands | Jonathan de Bruin | CC0 |
| PAK_GOV | Pakistan | National Information Technology Board, Government of Pakistan | |
| POL_WIKI | Poland | Wikipedia | CC BY-SA |
| PRT_MSDS | Portugal | Data Science for Social Good Portugal | MIT |
| RUS_GOV | Russia | Government of Russia | |
| SWE_GM | Sweden | Elin Lütz | MIT |
| SWE_SIR | Sweden | Svenska Intensivvårdsregistret | Public data may be used, but the source must be reported: Svenska Intensivvårdsregistret https://portal.icuregswe.org/siri/report/corona.inrapp (2020) |
| THA_STAT | Thailand | Open Government Data of Thailand | DGA Open Government License |
| TUR_MHOE | Turkey | Ministry of Health (Turkey) | MIT |
| USA_CTP | United States | COVID Tracking Project | CC BY-NC-4.0 |
| USA_NYT | USA, county | The New York Times | Attribution required, non-commercial use |
| WRD_ECDC | World | European Centre for Disease Prevention and Control | Attribution required |
| WRD_WHO | World | World Health Organization | |
| WRD_WHOJHU | World | Center for Systems Science and Engineering, Johns Hopkins University | CC BY 4.0 |
| ZAF_DSFSI | South Africa | Data Science for Social Impact Research Group, University of Pretoria | MIT |
| LBN_GOV | Lebanon | Ministry of Information (Lebanon) | |
| SAU_GOV | Saudi Arabia | Ministry of Health (Saudi Arabia) | |
| IRQ_GOV | Iraq | World Health Organization |
You need:
- Python3
- (optional) Running instance of a PostgreSQL database
- (optional) Docker
- Add the
DB_ADDRESS,DB_PORT,DB_NAME,DB_USERNAMEandDB_PASSWORDenvironment variables - Install requirements
pip install -r requirements.txt - Run fetcher
python3 ./main.py
- Add the
STAGE=test,DB_ADDRESS,DB_PORT,DB_NAME,DB_USERNAMEandDB_PASSWORDenvironment variables - Run
docker-compose up
| Variable name | Default value | Description |
|---|---|---|
| DB_USERNAME | Postgres database adapter user name | |
| DB_PASSWORD | Postgres database adapter password | |
| DB_ADDRESS | Postgres database adapter address | |
| DB_NAME | Postgres database adapter name | |
| DB_PORT | 5432 | Postgres database adapter port |
| SQLITE | SQLITE adapter file path | |
| CSV | CSV adapter file path | |
| VALIDATE_INPUT_DATA | False | Validate input data |
| SLIDING_WINDOW_DAYS | Sliding window, number of days in the past to process | |
| RUN_ONLY_PLUGINS | ALL | Run selected plugins from given list, run all plugins if empty |
| LOGLEVEL | DEBUG | Log level |
| SYS_EMAIL | Notifications SMTP username | |
| SYS_EMAIL_PASS | Notifications SMTP password |
We need fetchers!
Create a fetcher for a country that is not listed yet and send us a pull request. Use only official sources, or sources derived from official sources.
You can find example code for fetcher in /src/plugins/_EXAMPLE/example_fetcher.py