Sequencescape

Sequencescape is a cloud based and highly extensible LIMS system for use in labs with large numbers of samples.

Work order tracking
Sample and study management
Capacity management for pipelines
Accounting
Accessioning for samples and studies at the EBI ENA/EGA
Dynamically defined workflows for labs with support for custom processes
Labware and freezer tracking
API support for 3rd party applications

Current installation supports over 5 million samples and 1.8 million pieces of labware and is used in a organisation of 900 people.

Documentation
Requirements
Getting started
Testing
Linting and formatting
Rake tasks
Supporting applications
Miscellaneous

Documentation

In addition to the externally hosted YARD docs, you can also run a local server:

yard server -r --gems -m sequencescape .

You can then access the Sequencescape documentation through: http://localhost:8808/docs/sequencescape

Yard will also try and document the installed gems: http://localhost:8808/docs

Requirements

The following tools are required for development:

ruby (version defined in the .ruby-version)
yarn
node (version defined in the .nvmrc)
mysql client libraries - if you do not want to install mysql server on your machine, consider using mysql-client: brew install mysql-client. Alternatively, to install the MySQL required by Sequencescape (currently 5.7) use this helpful link.

Getting started (using Docker)

To set up a local development environment in Docker, you have to build a new Docker image for Sequencescape. start a stack of services that include a mysql database, and reset this database contents. You can do all together by running the command:

RESET_DATABASE=true docker-compose up

Optionally, if this is not the first time you start the app, you may not want to reset the database, and you can run this command instead:

docker-compose up

With this we should have started Sequencescape server and all required services. You should be able to access Sequencescape by going to http://localhost:3000 and log in with username and password admin/admin.

ABOUT LOCAL DEVELOPMENT SETUP You may want to start only the required services for Sequencescape (server and jobs worker) and use your local version of Mysql instead of the Docker version, in that case you can start this setup with the command:

docker-compose -f docker-compose-dev.yml up

ABOUT RECREATE DOCKER IMAGE If you ever need to recreate the image built on first start (because you made modifications to the Dockerfile file) you can run a building process with:

docker-compose build

Getting started (using native installation)

This section only applies if you don't have Docker installed or if you prefer a native installation of Sequencescape.

Installing ruby

It is strongly recommended that you use a ruby version manager such as RVM or rbenv to manage the Ruby version you are using. The ruby version required should be found in .ruby-version.

rbenv

If you have the rbenv ruby-build plugin it is as simple as:

rbenv install

It will pick up the version from the .ruby-version file automatically

Automatic Sequencescape setup

To automatically install the required gems, set-up default configuration files, and set up your database run:

bin/setup

Manual Sequencescape setup

In the event you have trouble with the automatic process, you may wish to step through the various steps manually.

Installing gems

Bundler is used to install the required gems:

gem install bundler
bundle install

Adjusting config

The config/database.yml file saves the list of databases.

Default setup

Create the database tables
```
bundle exec rake db:setup
```
Install webpacker and the required JS libraries
```
yarn
```

Starting rails

bundle exec rails s

Once setup, the default user/password is admin/admin.

Delayed job

For background processing Sequencescape uses delayed_job to ensure that the server is running. It is strongly recommended to start one for Sequencescape to behave as expected.

bundle exec rake jobs:work

bundle exec ./script/delayed_job start

Message broker

Sequencescape has its own message broker and consumer. To develop this or run it locally, you must have RabbitMQ installed. It may be easiest to use the docker image https://hub.docker.com/_/rabbitmq.

docker run -d --hostname my-rabbit --name some-rabbit -p 8080:15672 -p 5672:5672 rabbitmq:3-management

It can be useful to follow the rabbitmq logs, to look for broken connections or other problems. To do this using the docker image, get the container id using docker ps, and then:

docker logs -f <container id>

To start the consumer off listening for messages:

bundle exec warren consumer start

The consumer will run in the foreground, logging to the console. You can stop it with Ctrl-C.

For more warren actions, either use bundle exec warren help or see the warren documentation

You will also have to change the config in config/warren.yml from type: log to type: broadcast to get it to actually send messages in development mode.

Testing

Testing is done in one of three ways; using rspec, via rails tests or with cucumber.

To run the rspec tests (found in rspec/ dir.):

bundle exec rspec --fail-fast [<path_to_spec>]

To run the rails tests (found in tests/ dir.):

bundle exec rake test -f

For a single file:

bundle exec ruby -Itest test/lib/label_printer/print_job_test.rb

To run cucumber tests (found in features/ dir.) first ensure you have a sequencescape_test_cuke database configured by running:
```
RAILS_ENV=cucumber bundle exec rake db:setup
```
then run cucumber itself:
```
bundle exec cucumber
```

For a single file:

bundle exec cucumber features/create_plates.feature

Linting and formatting

Rubocop is used for linting.

bundle exec rubocop

Prettier is used for formatting.

yarn prettier --check .
yarn prettier --write .

Prettier rules are configured in .prettierrc.json
Whole files can be ignored in .prettierignore
Sections of files can be disabled using #prettier-ignore

Rake tasks

Rake tasks are available for specialised tasks as well as support tasks. Support tasks allow ease of running standalone scripts multiple times.

A breakdown of the the available tasks and how to run them can be found here

Supporting applications

There are a number of services that are needed in certain parts of Sequencescape these are listed below.

Barcode printing

Barcode printing is carried out by a separate REST service, PrintMyBarcode. The source for this is also available on GitHub sanger/print_my_barcode

Plate barcode service

Due to DNA plate barcode series being stored in a legacy system in Sanger you are required to use a webservice for supplying numbers for plates with a simple service.

Data warehousing

There is a client application for building a data warehouse based on the information in Sequencescape. This is driven asynchronously via RabbitMQ.

See our various clients on GitHub:

sanger/unified_warehouse

sanger/event_warehouse

Miscellaneous

Lefthook

Lefthook is a git-hook manager that will ensure staged files are linted before committing.

You can install it either via homebrew brew install Arkweid/lefthook/lefthook or rubygems gem install lefthook

You'll then need to initialize it for each repository you wish to track lefthook install

Hooks will run automatically on commit, but you can test them with: lefthook run pre-commit

In addition you can also run lefthook run fix to run the auto-fixers on staged files only. Note that after doing this you will still need to stage the fixes before committing. I'd love to be able to automate this, but haven't discovered a solution that maintains the ability to partially stage a file, and doesn't involve running the linters directly on files in the .git folder.

Ruby warnings and rake 11

Rake 11 enables ruby warnings by default when running the test suite. These can be disabled with RUBYOPT='-W0', (eg. RUBYOPT='-W0' bundle exec rake test).

Currently these warnings are excessive, covering both our own code and external dependencies. As it stands it makes the output of the test suite unusable in travis, as it fills the buffer. These warnings will need to be fixed, especially in our own code.

NPG - Illumina tracking software

For tracking illumina instruments you need the NPG systems. NPG is linked to Sequencescape via a cluster formation batch which represents a flowcell.

NPG Software

Troubleshooting

MySQL errors when installing

If you are using homebrew with rbenv and run into errors relating to SSL, have a look here

Updating the table of contents

To update the table of contents after adding things to this README you can use the markdown-toc node module. To install it, make sure you have install the dev dependencies from yarn. To update the table of contents, run:

npx markdown-toc -i README.md --bullets "*"

CI

The GH actions builds use the Knapsack-pro gem to reduce build time by parallelizing the RSpec and Cucumber tests. There is no need to regenerate the knapsack_rspec_report.json file, Knapsack Pro will dynamically allocate tests to ensure tests finish as close together as possible.

c2cn/sequencescape