/visilake

Self-Service Data Lake Exploration and Analysis on AWS

Primary LanguageTypeScript

Visilake

This is the mono repo for the visilake application. We deploy the AWS infra, the web-app docker container, and the rust processing jobs here.

The easiest way to deploy the visilake application is by using the nix package manager along with our deploy script.

flake.nix provides the all the project’s dependencies, builds the nodejs application, packages the nodejs application in a docker container, pulls the Localstack-pro docker image, and sets the necessary environment variables to deploy the test environment.

We provide comprehensive instructions for a windows deployment through WSL2 in our devenv repository.

Deployment

./infra/deploy-tf.sh simplifies multiple aspects of the deploy process

From the project root on a linux machine:

chmod +x ./infra/deploy-tf.sh
./infra/deploy-tf.sh

If you’d like to deploy the project with our supplied mockdata:

./infra/deploy-tf.sh --init

This will setup the mockdata for you. For all options see the usage below:

Usage: ./infra/deploy-tf.sh [options]

Options:
  none                   Full Localstack and Terraform Deployment
  -i, --init             Initiate the git submodule for the datalake and mock data
  -v, --verbose          Enable debug for Terraform and Localstack
  -r, --redeploy         Re-deploy Terraform config to Localstack
  -d, --docker           Re-deploy the visilake docker image to Localstack
  -t, --teardown [mode]  Perform teardown.
                         Modes: 'h' or 'hard' for hard teardown, no mode for standard teardown
                                 hard teardown will remove all docker images on your system
  -h, --help             Display this help message

Note:
  -r -d Localstack must already be running by running './infra/deploy-tf.sh' first

The web app will be deployed to http://localhost:45139

For an outline of the current AWS services used see the infrastructure readme.

Stopping the deployed services

To stop the localstack docker containers and any additional containers it has spawned we have a stop script:

./infra/deploy-tf.sh --teardown

It is wise to prune any unneeded images afterwards. If you do not have other docker containers you’d like to maintain you can simply run the following commands:

./infra/deploy-tf.sh --teardown hard

Visilake proc

To run the rust processing jobs first cd into the visilake-proc directory

cd visilake-proc

Next just run the app with cargo

cargo run

After the application is done compiling and building, you can run commands we’ve created for both debugging and running the processing

Usage: <COMMAND>

Commands:
  list-topics          List topics from AWS SNS
  list-queues          List queues from AWS SQS
  list-s3              List buckets and objects from AWS s3
  list-messages        List messages from AWS SQS
  process-queued-jobs  Process queued jobs
  test-parquet         Injest parquet and display it
  delete-queues        Deletes old update topic queues
  exit                 Exits the REPL
  help                 Print this message or the help of the given subcommand(s)

Prerequisites

This section is only for non-=nix= based deployments.

Note that we not longer provide a Dockerfile or docker-compose.yml for this project, but one can be found on an older branch. We do not currently support this method.