ETL Data Warehouse Testing and Data Quality Management using Postgres
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
- Docker (required*)
- JetBrains DataGrip IDE for working with databases (optional)
Step by step instructions on how to get a working version of the project on your local machine:
$ git clone <repo>
$ cd ./<repo>
docker-compose up
This will fetch the latest postgres docker image. On start-up it will copy and run the init scripts to setup a DB called postgres with a public schema. The data folder is copied to the container and the scripts will insert the data into the postgres DB.
The DB connection: jdbc:postgresql://localhost:5432/postgres
An explanation on how to run any automated tests that relate to the project.
remove containers & volumes
docker-compose down -v
remove any unused containers
docker system prune -a
remove any unused volumes
docker volume prune
Commands to help debug issues
Exec into docker:
docker exec -it dataetl-data-warehouse_db_1 bin/bash
docker exec -it dataetl-data-warehouse_db_1 psql -U postgres postgres
Exit container:
\q