owid-grapher

This is the project we use at the University of Oxford to create embeddable visualizations for Our World in Data. It's not currently designed for immediate reuse as a full library, but you are very welcome to adapt any of our code or to send pull requests.

An example of what this can make (click for interactive):

The owid-grapher visualization frontend code can run isomorphically under node to render data directly to an SVG string, which is how the above image works!

Initial development setup

macOS

Install Homebrew first, follow the instructions here: https://brew.sh/
Install Homebrew services:
```
brew tap homebrew/services
```
Install MySQL 5.7 and Node 12.13.1+:
```
brew install mysql@5.7 node
```
Start the MySQL service:
```
brew services start mysql@5.7
```
Install yarn:
```
npm install -g yarn
```
Inside the repo folder, install all dependencies by running:
```
yarn
```

Other platforms

You will need: MySQL 5.7, Node 12.13.1+ and Yarn. Running yarn in the repo root will grab the remaining dependencies.

Database setup

Remove the password

Remove the password for root by opening the MySQL shell with mysql and running:

ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY ''

We do this for convenience so we can run mysql commands without providing a password each time. You can also set a password, just make sure you include it in the .env file later.

Import the latest data extract

Daily exports from the live OWID database are published here and can be used for testing:

File	Description	Size (compressed)
owid_metadata.sql.gz	Table structure and metadata, everything except `data_values`	~15 MB
owid_chartdata.sql.gz	All data values used by published visualizations	>200MB

This script will create a database, then download and import all OWID charts and their data (might take a while!):

./db/downloadAndCreateDatabase.sh

Since the full data_values table (including everything we haven't visualized yet) is really big (>10GB uncompressed), we don't currently have an export for it. If you'd like a copy please contact us.

Inspecting the database

On macOS, we recommend using Sequel Pro (it's free).

We also have a rough sketch of the schema as it was on November 2019 (there may be slight changes).

Development server

cp .env.example .env and populate .env with your database details.

Finally, run yarn dev and head to localhost:3030/admin. If everything is going to plan, you should see a login screen! The default user account is "admin@example.com" with a password of "admin".

This development server will rebuild and live-reload the site upon changes, so you can just make changes to the code, save the file and see the result in the browser right away.

Migrations

If you need to make changes to the MySQL database structure, these are specified by typeorm migration files. Use yarn typeorm migration:create -n MigrationName and then populate the file with the SQL statements to alter the tables, using past migration files for reference if needed. Then run migrations with yarn migrate.

Architecture notes

owid-grapher is based around reactive programming using React and Mobx, allowing it to do client-side data processing efficiently. New code should be written in TypeScript. Visual Studio Code is recommended for the autocompletion and other awesome editor analysis features enabled by static typing.

The OWID tech stack has evolved over time as we've found different ways to solve our problems. We're happy with the combination of React + Mobx + TypeScript + node and expect to be using these core tools for the foreseeable future. The MySQL database and data structure however is much older and we're interested in exploring alternatives that might allow us to work with large amounts of data more quickly and with more flexibility.

Cross-browser testing provided by BrowserStack

marcelgerber/owid-grapher