popgetter

Popgetter is a convenience tool for downloading census data from a number of different jurisdictions and coercing the data into common formats. The aim is that city or region scale analysis can be easily replicated for different geographies, using the most detailed, locally available data.

What popgetter does and doesn't do

Popgetter DOES:

For each of the implemented countries:

Download the most detailed geometries, for which census data is available.
Download the most detailed census available for selected variables (currently focused on population and car ownership).
Ensures that the geometries and census data join correctly.

Popgetter WILL

present some standard metadata to allow the user to see which variables are available and any possible trade-off between geographic and demographic disaggregation.
publish the data in a set of common file types (eg CloudGeoBuff, Parquet, PMtiles).

Popgetter DOES NOT:

It does not attempt to ensure that census variables are comparable between different jurisdictions. Nor does it attempt to ensure that the results of any analysis can be directly compared across multiple countries.

Getting started

At present, this is still a development project, so the first step is to clone the repo and then install using pip, with the --editable option:

Create a virtual environment and activate it (you should be able to use your own choice of environment manager, such as conda or venv, but so far pyenv is the most tested with popgetter). eg:

python  -m venv popgetter_venv    # create a virtual environment called `popgetter_venv`
source popgetter_venv/bin/activate  # activate the virtual environment

Clone the repo and then do an 'editable' install:

git clone https://github.com/Urban-Analytics-Technology-Platform/popgetter.git
cd popgetter
pip install -e ".[dev]"

Then, start the Dagster UI web server:

dagster dev

Open http://localhost:3000 with your browser to see the project.

Development

You can start writing assets in popgetter/assets/ directory. New assets and jobs will need to be added to the popgetter/__init__.py file.

Adding new Python dependencies

You can specify new Python dependencies in pyproject.toml.

Unit testing

Tests are in the popgetter_tests directory and you can run tests using pytest:

pytest

Repo structure

This is a Dagster project. The repo layout was initially created with the dagster project scaffold command. It has been subsequently updated using the copier command and the Scientific Python template.

There is code, which predates the migration to Dagster in the previous_code directory. In due course, this will be removed as the remaining countries are migrated to Dagster. There are usage instructions for this old code in previous_code/previous_code_usage.md.