Gov Transparency Hub is a data pipeline project built using Dagster to collect information from transparency portals of multiple cities. It focuses on gathering data like revenue and expenses, and aims to create a consolidated database that can be accessed via an API by third-party applications.
To set up the project, make sure you have Poetry and Docker installed.
-
Install dependencies using Poetry
poetry install
-
Adjust service environment variables in .env. It is possible to launch services locally using Docker Compose present in the project
-
Launch Dagster
dagster dev