secondary-analysis
This repo is the gateway of the Secondary Analysis Service which is part of the Human Cell Atlas Data Coordination Platform, containing the testing suites, automations and utility scripts of the Secondary Analysis Service. This repo also serves as a issue tracker and hosting all of the tickets of the Secondary Analysis Service.
Other Secondary Analysis Service repos:
-
Adapter Pipelines: Contains Data Coordination Platform adapter pipelines
-
Cromwell Tools: A collection of Python clients and accessory scripts for interacting with the Cromwell workflow execution engine - a scientific workflow engine designed for simplicity and scalability
-
Falcon: Queueing system that (after launching) throttles and inititates workflows
-
Lira: Listens to storage service notifications and launches workflows
-
Pipeline Tools: Contains Data Coordination Platform adapter pipelines and associated tools
-
scTools: Tools for single cell data processing
-
Secondary Analysis Deploy: Contains the deployment configuration and scripts for the Pipeline Execution Service
-
Skylab Analysis: Analysis and benchmarking reports for standardized HCA pipelines
-
Skylab: Standardized HCA data processing pipelines
Development
File Structure Layout
.
├── operations
│ ├── big_red_button # The "BIG RED BUTTON" scripts of Secondary Analysis
│ ├── dashboard # The scripts to create log-based metrics on Google Cloud
│ ├── data_cleanup # The scripts to cleanup data from the Cromwell execution buckets
│ ├── failure_analysis # The scripts to perform failure analysis on failed workflows
│ ├── gcp_quota # The scripts to setup quotas monitors and fetch results
│ ├── run_analysis # The scripts to trigger analysis manually in HCA DCP through Lira
│ └── tls_cert # The scripts to renew TLS certs
└── tests
├── integration # The integration test suite
├── meteoroid # The "Next-Gen Data Driven" test suite which is under construction
└── scaling # The scaling test suite
Code Style
The Secondary Analysis code base is complying with the PEP-8 and using Black to format our code, in order to avoid "nitpicky" comments during the code review process so we spend more time discussing about the logic, not code styles.
In order to enable the auto-formatting in the development process, you have to spend a few seconds setting up the pre-commit
the first time you clone the repo. It's highly recommended that you install the packages within a virtualenv
.
- Install
pre-commit
by running:pip install pre-commit
(or simply runpip install -r requirements.txt
). - Run
pre-commit install
to install the git hook.
Once you successfully install the pre-commit
hook to this repo, the Black linter/formatter will be automatically triggered and run on this repo. Please make sure you followed the above steps, otherwise your commits might fail at the linting test!
If you really want to manually trigger the linters and formatters on your code, make sure Black
and flake8
are installed in your Python environment and run flake8 DIR1 DIR2
and black DIR1 DIR2 --skip-string-normalization
respectively.