This is a custom plugin for Torrent SuiteTM that supports the upload of Ion Torrent data (BAM files) to the onco-PMNet platform (or any other Amazon S3-compatible object store).
It is important to have this code in a folder named oncopmnetUploader
and this will be the case if you clone this repository without specifying a different local directory, e.g. you should just do:
git clone git@gitlab.repository.oncopmnet.gr:sgsfak/oncopmnetUploader.git
After that you need to package the newly created directory into a Zip file as described here. In a Unix-like operating system you could issue the following command (assuming you are not inside the oncopmnetUploader
directory):
zip --exclude \*.git\* --exclude \*img\* -r oncopmnetUploader oncopmnetUploader
The produced oncopmnetUploader.zip
file can then be uploaded in the Torrent SuiteTM server at the configure plugins GUI, by clicking the "Install or Upgrade plugin" button.
The current implementation of the plugin uses the MinIO Client command line tool so you need to download the Linux executable and copy it into the Torrent Server. The plugin assumes that the minio client executable mc
is available as /usr/local/bin/mc
so you need to copy it there (or change the mc_command
variable in the oncopmnetUploader.py
source code).
After the upload of the plugin you need to configure it globally (i.e. it will be done once and the configured information will then be available in all future runs). The configuration can be done from the "Configure" option available in the "cog" button of the plugin's row.
The following screenshot shows the options available. You need to set the S3 server address, the "access key", and "secret key" for contacting the backend object storage. Also you need to specify the bucket and folder where the BAM files will be uploaded to.
After installation and configuration the plugin should be available to be selected to run in a specific "Run". The plugin should take automatically the run information and upload all the associated BAM files in the server configured.
The following image depicts the information shown after running the plugin. The plugin creates the subfolders in the configured directory based on Run's date, Project, and Sample name.
The plugin also attaches the following metadata to the uploaded files:
aligned
: True or False, depending on whether it's an aligned or unaligned BAM fileref
: The genome reference used ifaligned
is Truerun_date
: The datetime of the runrun_name
: The (local) run nameread_count
: The number of reads in the BAM filesystem_type
: The type of sequencer (e.g. 'pgm') used in the runchip_type
: The chip which was used to do the run (e.g. '540')run_flows
: The number of flows used in the runplatform
: This is always 'IonTorrent'
You can use the stat
command of the
Minio client
to check these Metadata values after the upload, e.g.
> mc stat 'precmed/workflow-inputs/.../2019_05_15_08_11_52/Q4DP-2019.bam'
Name : Q4DP-2019.bam
Date : 2019-11-06 00:17:51 EET
Size : 555 MiB
ETag : aadeec657b97f056ea0b1a91eea851c9-5
Type : file
Metadata :
Content-Type : application/gzip
X-Amz-Meta-Aligned : True
X-Amz-Meta-Platform : IonTorrent
X-Amz-Meta-Run-Date : 2019_05_15_08_11_52
X-Amz-Meta-Chip-Type : 520
X-Amz-Meta-System-Type: S5XL
X-Amz-Meta-Run-Flows : 400
X-Amz-Meta-Read-Count : 1423248
X-Amz-Meta-Ref : hg19
X-Amz-Meta-Run-Name : R_2019_05_15_11_09_26_user_ARIADNE-23-ColonLungV2