plantinformatics/pretzel

Include blastServer in docker-compose yaml to enable single-command installation and startup

Don-Isdale opened this issue · 10 comments

Don-Isdale commented

Significance

The completion of this issue means that Pretzel can now be deployed using a single command.
This means all the processes which make up the Pretzel server, including the DNA Sequence Search server (blastServer.py) which was not part of the previous single-command startup.

The method of use will be described in the documentation.

part of #377

Simplify the installation process for self-hosting on a PC by adding a docker-compose yaml based on an existing docker-compose.yaml, with added shared volumes for VCF and blast databases, and including the python flask blastServer.py,
Targeting MS Windows PC - define the data directories relative to standard directories e.g. My Documents\

  • create a Dockerfile for the blastServer.py

  • add that Dockerfile to a docker-compose.yaml based on existing docker-compose.yaml with mongoDb + Pretzel server

    • change file path to reflect blastServer running within a container
      1e0f721 include the Python Flask blastServer.py in a docker-compose yaml

  • add volume : resultCache

The next stage of this process will be to customise the docker-compose yaml for installation on a MS Windows PC, which is tracked in this sub-issue : #400

Don-Isdale commented

Testing

Tested by running Pretzel on a development server, using existing database directory and blast databases.

This is running OK and has been tested by other team members.

Don-Isdale commented

Documentation

Documentation explaining the method of using this added docker-compose yaml has been added to the User Documentation : https://docs.plantinformatics.io/Local-Setup/setup/

  • [main e669f21] add Combined setup of Pretzel with supporting Database server and Blast server
Don-Isdale commented

Introduction

The goal of this change is to enable users to set up Pretzel with arbitrary data directory structure the external databases : Blast for Sequence Search, VCF Genotype and Fasta index for DNA Lookup.

This change adds mntData to the docker compose yaml and env, and enables the scripts to use mntData to for default values of vcfDir and blastDir, and also enables users to configure those individually to handle different directory structures, and databases on separate volumes.

Add configuration of directories for VCF and Blast databases

  • ensure that vcfDir is configured, via mntDatab
    [feature/composeDirs 15fa1a5] include the Python Flask blastServer.py in a docker-compose yaml (from git cherry-pick -x 1e0f721)
    [feature/composeDirs b65ce69] use mntData for blastDir and vcfDir
Don-Isdale commented

Introduction

Update the script to match with the new path configuration.

Change

In scripts/dnaSequenceSearch.bash the path of the Blast database is now simply dbName="$datasetId".dir/$(cat "$datasetId".dbName),
removing $blastDir/GENOME_REFERENCES/ in the case where .dir is a soft-link.

  • this is implemented and tested on the internal test server
Don-Isdale commented
Additional configuration

Some additional path configuration was necessary to handle the blast database path configuration on the development server. To confirm these changes they were set up as patches added to the current image version; these worked, and now equivalent changes have been made in the source control.

  • 8c69758 adjust paths configuration for blastserver
Don-Isdale commented

Reported Issue : blastserver container exiting after error

It is reported that the blastserver container on agg was exiting after errors.

To investigate this the blastserver image was built on local and run using docker run, and interrogated via the Pretzel GUI and server.
Various path errors were hit while setting up this configuration - they did not cause the blastserver to exit.

The next test will be running blastserver along with Pretzel API server and database via docker compose instead of docker run.
It may be possible to do this on the server on which the error was reported.

Don-Isdale commented

Enable multiple docker-compose pretzel servers to run on an instance

The blastserver included in docker-compose was exporting its service port outside of the docker-compose network bridge, which is not required, and prevents multiple docker-compose pretzel servers being run on one instance.

This has been addressed in these commits :

  • ee38373 don't expose the blastserver port 4000
  • .env define hostIp=blastserver for use in docker compose
Don-Isdale commented

Environment configuration of docker compose in version control

config/dev.bash aab9b7a don't set API_HOST because it would override env file setting

Don-Isdale commented

Releases

The recent Docker images for blastserver and pretzel are shown here :

The blastserver image is available as:

  • docker pull plantinformaticscollaboration/blastserver:v1.0.2
Don-Isdale commented

Further iteration

Testing of the above release AWS instance showed some configuration issues. These were debugged and patched on the instance, and corresponding changes propagated back to the source repository.

Source repository changes

The explanation of the changes is documented in the commit message 8e9f394.

8e9f394 copy blastn from ncbi/blast image into blastserver image
87ff0e0 increment BLASTSERVER_VERSION to 1.0.5