This is the mono repo for the visilake application. We deploy the AWS infra, the web-app docker container, and the rust processing jobs here.
The easiest way to deploy the visilake application is by using the nix
package manager along with our deploy script.
flake.nix provides the all the project’s dependencies, builds the nodejs application, packages the nodejs application in a docker container, pulls the Localstack-pro docker image, and sets the necessary environment variables to deploy the test environment.
We provide comprehensive instructions for a windows deployment through WSL2
in our devenv repository.
./infra/deploy-tf.sh simplifies multiple aspects of the deploy process
From the project root on a linux machine:
chmod +x ./infra/deploy-tf.sh
./infra/deploy-tf.sh
If you’d like to deploy the project with our supplied mockdata:
./infra/deploy-tf.sh --init
This will setup the mockdata for you. For all options see the usage below:
Usage: ./infra/deploy-tf.sh [options]
Options:
none Full Localstack and Terraform Deployment
-i, --init Initiate the git submodule for the datalake and mock data
-v, --verbose Enable debug for Terraform and Localstack
-r, --redeploy Re-deploy Terraform config to Localstack
-d, --docker Re-deploy the visilake docker image to Localstack
-t, --teardown [mode] Perform teardown.
Modes: 'h' or 'hard' for hard teardown, no mode for standard teardown
hard teardown will remove all docker images on your system
-h, --help Display this help message
Note:
-r -d Localstack must already be running by running './infra/deploy-tf.sh' first
The web app will be deployed to http://localhost:45139
For an outline of the current AWS services used see the infrastructure readme.
To stop the localstack docker containers and any additional containers it has spawned we have a stop script:
./infra/deploy-tf.sh --teardown
It is wise to prune any unneeded images afterwards. If you do not have other docker containers you’d like to maintain you can simply run the following commands:
./infra/deploy-tf.sh --teardown hard
To run the rust processing jobs first cd into the visilake-proc directory
cd visilake-proc
Next just run the app with cargo
cargo run
After the application is done compiling and building, you can run commands we’ve created for both debugging and running the processing
Usage: <COMMAND>
Commands:
list-topics List topics from AWS SNS
list-queues List queues from AWS SQS
list-s3 List buckets and objects from AWS s3
list-messages List messages from AWS SQS
process-queued-jobs Process queued jobs
test-parquet Injest parquet and display it
delete-queues Deletes old update topic queues
exit Exits the REPL
help Print this message or the help of the given subcommand(s)
This section is only for non-=nix= based deployments.
Note that we not longer provide a Dockerfile
or docker-compose.yml
for this project, but one can be found on an older branch. We do not currently support this method.
- Docker
- Docker Compose
- Localstack
- awscli v1 (do not use v2)
- awscli-local