Simple data infrastructure environment built with docker
and docker-compose
.
The environment is composed by:
Just run make build
and the project will take care of building the required images.
make
will build everything necessary and use docker-compose
to deploy the environment.
-
Start the environment
-
The postgres database is initialized with two tables that represent a soccer tournament, one for Teams and another for Matches. See setupdb.sh
-
During deployment, one of the hadoop datanodes will use
sqoop
to import the Matches table into hive. See the hadoop-datanode startup script -
Run the
sbaldrich/presto-consumer-example
image making sure to attach it to the network that was created by compose. Notice that the url of the presto coordinator is given as argument to the execution of the image.
docker run -it --network data-environment_default sbaldrich/presto-consumer-example jdbc:presto://coordinator:8080
- The
sbaldrich/presto-consumer-example
image contains a simple java application that will run a SQL script that consumes and joins data from different data sources (postgres and hive) and joins it to produce a result.
Thanks to Lewuathe, the owner of the docker-presto-cluster code on which this project is heavily based.