/unite-environment

Unite environment installation files and pipelines.

Primary LanguageShellGNU General Public License v3.0GPL-3.0

UNITE Environment

UNITE platform installation scripts and configuration files

Prerequisites

Folder Structure

  • applications
    • unite - web portal
      • docker-compose.yml - installation configuration
      • docker-compose.build.yml - build and installation configuration (from repository source code)
      • deploy.sh - installation script
      • build.local.sh - local installation script (from repository source code - configurable local source)
      • build.remote.sh - remote installation script (from repository source code - remote source)
    • unite-identity - identity web API
    • unite-composer - composer web API
    • unite-analysis - analysis web API
    • unite-analysis-deseq2 - analysis DESeq2 web API
    • unite-donors-feed - donors data feed web API
    • unite-images-feed - imaging data feed web API
    • unite-specimens-feed - specimens data feed web API
    • unite-genome-feed - genome data feed web API
    • unite-data-migrations - domain data migrations service
    • unite-identity-data-migrations - identity data migration service
    • unite-ensembl-data - ensembl data Web API
    • unite-ensembl-vep - ensembl Varian Effect Predictor (VEP) web API
  • programs
    • postgresql - data storage
      • docker-compose.yml - installation configuration
      • docker-compose.mgmt.yml - management tools installation configuration
      • deploy.sh - installation script
      • deploy.mgmt.sh - management tools installation script
    • elasticsearch - search engine and indices storage
    • mongodb - cache and technical data storage
    • mysql - Ensembl data storage
  • scripts
    • configure.sh - general environment configuration sub-script (used by dedicated environemnt scripts)
    • configure.linux.sh - Linux environment configuration script
    • configure.macos.sh - MacOS environment configuration script
    • generate-ssl.sh - SSL certificate generation script
    • deploy.sh - installation script
    • build.sh - build and install script
    • uninstall.hard.sh - hard uninstallation script (removes data and volumes)
    • uninstall.soft.sh - soft uninstallation script (keeps data and volumes)
    • deploy.mgmt.sh - install management tools
    • deploy.apps.sh - install applications script (for easy updates)
    • build.apps.sh - build and install applications script (for easy updates)
  • secrets_template.json - JSON configuration template file with all required credentials

Secrets

All sensitive information is stored in secrets.json file. Template of this file is stored in unite-environment repository on github. File has the following structure:

{
    "api": {
        "key": "Defautl32BitApiKeyHasToBeChanged"
    },
    "admin": {
        "user": "admin@unite.net",
        "password": "Long-pa55w0rd"
    },
    "github":{
        "user": "root",
        "token": "Long-t0ken"
    },
    "sql": {
        "user": "root",
        "password": "Long-pa55w0rd"
    },
    "mongodb": {
        "user": "root",
        "password": "Long-pa55w0rd"
    },
    "elasticsearch": {
        "user": "elastic",
        "password": "Long-pa55w0rd"
    }
}

Each section represents credentials for specific part of the platform services:

  • api - API key, used for user and service authentication
  • admin - root user and password for portal administration
  • github - user name and personal access token with full repository and packages permissions for accessing source code and packages from Github (follow github instructions to generate it)
  • sql - user name and password for SQL server
  • mongodb - user name and password for MongoDb server
  • elasticsearch - user name and password for Elasticsearch server

To generate passwords one of command line tools can be used:

  • openssl rand -base64 22 - to generate 32 bit Base64 string

Warning

ALWAYS CHANGE ALL CREDENTIALS FOR PRODUCTION USE!

Warning

NEVER COMMIT THIS FILE TO THE REPOSITORY!

Note

For development it is best to keep most of the default credentials.

Installation

  1. Download unite-environment source files from git this repository
  2. Change credentials in secrets_template.json and rename the file to secrets.json
  3. Open unite-environment/scripts folder in terminal
  4. Configure environment
    • For Linux environment: sh configure.linux.sh
    • For MacOS environments: sh configure.macos.sh
  5. Install SSL certificate
    • If valid certificate is available
      • Copy certificate and it's key to unite-environment/ssl directory (create directory if missing)
      • Name certificate and it's key unite-cert.pem and unite-key.pem
      • Change certificate and it's key permissions to 600 using chmod command
    • If valid certificate is not available, generate self signed development certificate
      • For localhost: sh generate-ssl.sh
      • For network: sh generate-ssl.sh <IP address or domain name>
  6. Install programs and applications
    • sh deploy.sh
  7. Install management tools (optional)
    • sh deploy.mgmt.sh
  8. Install Ensembl Data service
    • Open unite-environment/programs/mysql folder in terminal
    • Download cache sh download-ensembl-cache.sh (This may take some time, if download process breaks, run the script again to continue)
    • Extract cache sh extract-ensembl-cache.sh
    • Restore database from cache sh install-ensembl-cache.sh (This may take several minutes)
  9. Install Ensembl VEP service
    • Open unite-environment/applications/unite-ensembl-vep folder in terminal
    • Download cache sh download-cache.sh (This may take several hours, if download process breaks, run the script again to continue)
    • Extract cache sh extract-cache.sh

For custom installation run installation scripts for dedicated programs or applications.
Cache downloaded and extracted for Ensembl Data and Ensembl VEP services will persist on the disk unless you remove it manually, so no need to run download and extraction scripts again for recurrent installation of the platform.

Uninstallation

To uninstall UNITE platform, run corresponding command from unite-environment/scripts folder:

  • To remove all services with all their data: sh uninstall.hard.sh
  • To remove only services, but keep all their data: sh uninstall.soft.sh

!Note that uninstallation scripts will remove all containers/images from host machine. If you have other applications runing in docker, clear unite containers and images manually.

Services

Application is running in docker and has the following components:

Service Container Alias Container Port Host Port
Elasticsearch es es.unite.net 9200 9200
Mongodb mongo mongo.unite.net 2701 2701
Postgresql pgsq pgsql.unite.net 5432 5432
Mysql mysql mysql.unite.net 3306 3306
Web Portal unite.portal portal.unite.net 80;443 80;443
Identity Service unite.identity identity.unite.net 80 5000
Composer Service unite.composer composer.unite.net 80 5002
Analysis Service unite.analysis analysis.unite.net 80 5004
Analysis DESeq2 unite.analysis.deseq2 deseq2.analysis.unite.net 80 5300
Analysis Single Cell unite.analysis.scell scell.analysis.unite.net 80 5302
Analysis Kaplan Meier unite.analysis.kmeier kmeier.analysis.unite.net 80 5304
Donors Feed Service unite.donors.feed feed.donors.unite.net 80 5100
Images Feed Service unite.images.feed feed.images.unite.net 80 5102
Specimens Feed Service unite.specimens.feed feed.specimens.unite.net 80 5104
Genome Feed Service unite.genome.feed feed.genome.unite.net 80 5106
Ensembl Data Service unite.ensembl.data data.ensembl.unite.net 80 5200
Ensembl VEP Service unite.ensembl.vep vep.ensembl.unite.net 80 5202

Note

Only Web portal ports are publicly exposed (0.0.0.0). Ports of other components are exposed to host machine local network only (127.0.0.1).