T2DREAM Metadata Database

Setting Up Your Environment

These are the primary software versions used in production, and you should be able to use them locally:

Python 3.4.3
Node 6
Elasticsearch 1.7
Java VM 1.8
Ubuntu 14.04

0. Xcode for Mac OS build tools

Install Xcode from the website or Mac App store because the local build will use some of Xcode's compilation tools.
Install the Xcode commandline tools (the commandline tools may need to be reinstalled after OS and Xcode updates)
- xcode-select --install

⭐ Note: You will need to open Xcode to accept the end-user agreement from the application, or from the commandline run:_

sudo xcodebuild -license accept

1. Homebrew for Mac OS package management

Verify that Homebrew is installed and working properly:
- brew doctor

2. (Mac) Install or update other dependencies:

brew install libevent libmagic libxml2 libxslt openssl graphviz nginx
brew install freetype libjpeg libtiff littlecms webp chromedriver
brew tap petere/postgresql
brew install postgresql@9.3
brew link --force postgresql@9.3
brew install --force node@6
brew link node@6 --force
brew cask install java8
brew install elasticsearch@5.6
pip3 install typing

⭐ Note: Elasticsearch 1.7 does not work with Java 9 :star: Note: To unlink elasticsearch 1.7 and install elasticsearch 5.6

brew search elasticsearch

brew install elasticsearch@5.6

brew unlink elasticsearch@1.7 && brew link elasticsearch@5.6 --force

⭐ Note: Brew cannot find java8

brew tap caskroom/versions # lookup more versions

brew cask search java # java8 now in list

brew cask install java8

⭐ Note: This additional step is required for new macOS Sierra installations

brew cask install Caskroom/cask/xquartz

⭐ Note: Node version mangement with nvm: If you need to easily switch between node versions you may wish to use nvm instead (not required for most users)

npm install -g nvm

nvm install 6

nvm use 6

⭐ Note: Node version mangement with homebrew: Switching node versions with homebrew (which is not package manager but not a typical version manager) is possible but is not as flexbible for this purpose, e.g.:

brew install node@7

brew unlink node@6 && brew link --overwrite --force node@7

node --version

brew unlink node@7 && brew link --overwrite --force node@6

node --version

⚠️ Note: If you need to update Python dependencies (do not do this randomly as you may lose important brew versions of packages you need):

rm -rf encoded/eggs (then re-run buildout below) and possibly

brew update

brew upgrade

3. Python

Encoded requires a UNIX based system (Mac or Linux) and Python 3.4.3 (but works with 3.5.x):

For local development on a Mac, follow the steps below. For Linux use apt-get or yum as your Linux flavor demands. You can consult cloud-config.yml for other steps.
Note: Production is currently using the versions above thus your primary development should always work on that version, and you should test that your code works on versions that will be used in production.
Linux: apt-get install python3.4-dev or equivalent

Mac OSX Python install instructions

The Python version management tool pyenv is very useful.

⚠️ Note: If you have previously installed python3 from homebrew, you may possibly wish to uninstall it (not required):

brew uninstall --force python3

Install pyenv and set the default versions:

brew install pyenv
pyenv install 3.4.3
pyenv install 2.7.13
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
echo 'eval "pyenv shell 2.7.13 3.4.3"' >> ~/.bash_profile
source ~/.bash_profile

⭐ Note: Migrating pyenv Python packages

If you have previously installed a Python version from pyenv, and want to quickly migrate all your pypi packages to a new version (Python 2 to 2, and Python 3 to 3 only):

brew install pyenv-pip-migrate

Example if you previously installed 2.7 which really is 2.7.0:

pyenv install 2.7.13

pyenv migrate 2.7 2.7.13

⭐ _Note: pyenv install fails with "ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib?" for MAC OS High Sierra

Uninstall and re-install openssl using the following command when you install pyenv

brew uninstall --ignore-dependencies openssl && brew install openssl && CFLAGS="-I$(brew --prefix openssl)/include" LDFLAGS="-L$(brew --prefix openssl)/lib" pyenv install <VERSION>

Set the correct Python for the current directory:

pyenv local 3.4.3

4. Run buildout:

python3 bootstrap.py --buildout-version 2.9.5 --setuptools-version 18.5
bin/buildout

⭐ Note: If you have issues with postgres or the python interface to it (psycogpg2) you probably need to install postgresql via homebrew (as above)

⭐ Note: If you have issues with Pillow you may need to install new xcode command line tools: Update or Install Xcode from the Mac AppStore (reboot may be required) and re-run:

xcode-select --install

⭐ Note: Clean ALL the Things! If you wish to completely rebuild the application or cleanly reload dependencies (:warning: long re-build time!):

make clean && buildout bootstrap && bin/buildout

5. Start the application locally

Terminal window 1:
In one terminal window startup the database servers and nginx proxy with:
- bin/dev-servers development.ini --app-name app --clear --init --load
This will first clear any existing data in /tmp/encoded. Then postgres and elasticsearch servers will be initiated within /tmp/encoded. An nginx proxy running on port 8000 will be started. The servers are started, and finally the test set will be loaded.
Terminal window 2:
In a second terminal, run the app with:
- bin/pserve development.ini

Indexing will then proceed in a background thread similar to the production setup.

⭐ Note: If you run into a java stack trace when you run the app, it is worth checking if /usr/local/etc/elasticsearch/elasticsearch.yml might have the line: ‘path.plugins: /usr/local/var/lib/elasticsearch/plugins’. If it does, it needs to be commented out.

6. 🎉 Check out the app! 🎉

Browse to the interface at http://localhost:8000/.

Running tests

To run specific tests locally:

bin/test -k test_name
To run with a debugger:

bin/test --pdb
Specific tests to run locally for schema changes:

bin/test -k test_load_workbook
bin/test -k test_load_schema
Run the Pyramid tests with:

bin/test -m "not bdd"
Run the Browser tests with:

bin/test -m bdd -v --splinter-webdriver chrome
Run the Javascript tests with:

npm test
Or if you need to supply command line arguments:

./node_modules/.bin/jest
Test ALL the things!

bin/test -v -v --splinter-webdriver chrome && npm test

Building Javascript and CSS

Our Javascript is written using ES6 and JSX, so needs to be compiled using babel and webpack. Our CSS is written in the SCSS variant of Sass and also needs compilation using webpack.

To re-build production-ready bundles, do:

npm run build

(This is also done as part of running buildout.)
To build development bundles and continue updating them as you edit source files, run:

npm run dev

The development bundles are not minified, to speed up building. The above command runs continually in your terminal window and watches for changes in Javascript and SCSS files, rebuilding the bundles as you make changes.

Creating a demo machine

After buildout you (if you have the correct permissions) can run for a single-node "cluster":

bin/deploy
The script above will spin up a server in the AWS cloud with the current branch, and with a computed nameserver alias based on the branch and your username. Note that this retrieves a Postgres database from the current backup, so "as is" applies specifically to the T2DREAM Portal (if you have forked the repo you will not have permission to retrieve the db). There are options to use a different branch and/or different instance name and also if you want to use AWS spot instances...and you can specify which AWS profile you want to use.
Deploy script help (how to specify name, instance size, etc):

bin/deploy --help

For all options, including setting up ES clusters (needed for full production). After indexing (currently 8+hrs) the machine can be downsized at AWS to an m4.2xlarge, unless you are planning to submit significant data to it.

Linting your code within your code editor

To set up linting with Sublime Text 3 or Visual Studio Code, first install the linters:

pip3 install flake8
npm install -g eslint
npm install -g eslint-plugin-react

⚠️ Note: You don't have to use Sublime Text 3 but you must insure that linting in your editor behaves as it does in Sublime Text 3.

Sublime Text 3

After first setting up Package Control (follow install and usage instructions on site), use it to install the following linting packages in Sublime Text 3:

sublimelinter
sublimelinter-flake8
SublimeLinter-contrib-eslint (Sublime linter eslint instructions)
babel (Babel instructions)

Sublime Linting with pyenv
To get Sublime to lint Python code using pyenv you must add the python version and paths and python_paths to your Sublime Linter Preferences. In Sublime Text, navigate to the user linter preferences:

Sublime Preferences -> Package Settings -> Sublime Linter -> Settings-User
Add the following (modify existing preference settings file or add this entire JSON object below if the file is blank):

{
    "user": {
        "@python": 3.4,
        
        "paths": {
            "linux": [],
            "osx": [
                "/Users/YOURUSERNAME/.pyenv/versions/3.4.3/bin/",
                "/Users/YOURUSERNAME/.pyenv/versions/2.7/bin/"
            ],
            "windows": []
        },
        "python_paths": {
            "linux": [],
            "osx": [
                "/Users/YOURUSERNAME/.pyenv/versions/3.4.3/bin/python3",
                "/Users/YOURUSERNAME/.pyenv/versions/2.7/bin/python"
            ],
            "windows": []
        }
    }
}

Restart Sublime

Visual Studio Code

Go to the Visual Studio Code marketplace and install these extensions:

ESLint
Python
Sass

Check versions and linting

Versions

python3 --version returns Python 3.4.3 (or variant like 3.4.x)
node --version returns v6.10.3 (or variant like v6.x.y)
elasticsearch -v returns Version: 1.7.6 (or variant like Version: 1.7.x)
postgres --version returns postgres (PostgreSQL) 9.3 (or variant like 9.3.x)

Linting check

Python: Open Sublime, make a change to a Python file, make an intentional syntax error (no : at the end an if evaluation). Warnings and errors should show on the left side of the line.
JavaScript: Make a syntax error, or style error, in a JS file. Warnings and errors should show on the left side of the line.

jensreeder/t2dream-portal