plantinformatics/pretzel

Standardisation of Pretzel Deployment

kennyAgricultureVic opened this issue · 5 comments

kennyAgricultureVic commented

Pretzel is designed to be deployed in many different contexts :

  • in a public cloud such as AWS
  • in an internal data centre, associated with a supercomputing cluster
  • on a personal desktop or laptop, supporting the major PC operating systems : MS Windows, Linux and iOS / MacOS

The configuration requirements for each of these contexts is progressively being integrated together into a common deployment method which provides configuration to handle the variations.

This process streamlines system administration and reduces the effort of supporting multiple platforms. The focus is on making the deployment of Pretzel quick and easy to manage for staff administering the system, and enabling non-specialists to install Pretzel on their PC with minimum effort. This includes access to the connected databases such as VCF Genotypes, Blast databases, and other external systems.

The focus of this github issue is this process of standardising the deployment of Pretzel.

kennyAgricultureVic commented
  • run blast micro-service on test server to enable blast within container
Don-Isdale commented

Deployment

  • pulses VCF config / error
    • check pretzel-hosting version and deploy.sh link on other hosts
    • commit additions to deploy script repo pretzel-hosting
      da0fc6b
      add functions prepareLogDir and doCommandOnServers
  • added --volume blast to container run configuration on test server
  • documented process for building and running Pretzel server containers on test server 14cfef6
Don-Isdale commented

Simplify the installation process for self-hosting on a PC by adding a docker-compose yaml based on an existing docker-compose.yaml, with added shared volumes for VCF and blast databases, and including the python flask blastServer.py,
Targetting MS Windows PC - define the data directories relative to standard directories e.g. My Documents\

  • create a Dockerfile for the blastServer.py

  • add that Dockerfile to a docker-compose.yaml based on existing docker-compose.yaml with mongoDb + Pretzel server

    • change file path to reflect blastServer running within a container
      1e0f721 include the Python Flask blastServer.py in a docker-compose yaml

  • add volume : resultCache

Don-Isdale commented

Adjust configuration for internal test server

To generalise the docker compose yaml to additional contexts, it will be used to run the internal test server.
By ensuring that the docker compose yaml can be configured to support all the context which we expect users will want to install the Pretzel application server on, we streamline the usability and uptake of the application.

This context is different in several ways to the web-facing AWS servers, e.g. database configuration. Also the test server is the second server on the host it is running on, so the ports for mongodb and blastserver should not confict, and ROOT_URL is used to position the server URL in a separate URL sub-directory. The servers on AWS use sub-domains instead to separate URL spaces of multiple servers on a single host, which is easy to configure in nginx and requires no special configuration in the application server.

Changes Required

I had to copy some extra binaries into the blastserver container : /usr/bin/parallel and vmtouch
I got them from the ncbi/blast image.
I'll add those to the blastserver image via blastServer.Dockerfile
I got a successful test with IWGSC_RefSeq_v1.0, WAPO1

There were a bunch of other configurations to add / change :

  • db.createUser( ) in pretzel2 database
  • add DB_USER, DB_PASS to the .env file
  • add volumes :
    • /group directories for data
    • /local-ssd/data/resultsCache:/app/node_modules/flat-cache/.cache

Results

/pretzelUpdate is now running from docker-compose, and blast works.

Don-Isdale commented

Complete and used in v3.0.0 Release

This work is complete and is used in the v3.0.0 release #414 .