gdc-client: A Shell repository from sbamin

gdc-client docker image

22-Apr-2018
v1.3.0.b1

Source: https://github.com/sbamin/gdc-client
Docker: https://hub.docker.com/r/sbamin/gdc-client
Based on gdc-client Ubuntu x64 binary at https://gdc.cancer.gov/access-data/gdc-data-transfer-tool

To download controlled access data

In the host machine (where you've installed docker): Create or preferably mount an external disk to store sequence data, e.g., /mnt/myscratch/.

To avoid potential issues with file permissions on the host machine, avoid mounting entire home directory or path to critical data as a docker shared volume. I usually prefer creating a new directory for shared volume.

Download and keep GDC manifest file in tsv/txt format for open access files in the shared directory.
Save GDC token file in the same director and do chmod 600 gdc_token.key
IMPORTANT: Make sure to pass valid (identical to host machine) user and group environment variables in docker run command else stored files may inherit root or strange user ownership.

## path where data will be stored on the host machine
export USERMOUNT="/mnt/myscratch"

cd "${USERMOUNT}"

## MAKE SURE TO GIVE PROPER USER AND GROUP IDs, matching to those of host machine
docker run -d -e HOSTUSER=$USER -e HOSTGROUP=$(id -gn $USER) -e HOSTUSERID=$UID -e HOSTGROUPID=$(id -g $USER) -v "${USERMOUNT}":/scratch sbamin/gdc-client "gdc-client download --log-file=download.log -n 4 -t gdc_token.key -m controlled_manifest.tsv"

Here, we run docker in daemon mode, mount /mnt/myscratch (supply full path and not relative) directory on the host machine to /scratch location within docker container. Then we start, gdc-client download with 4 threads and fetch controlled access data from the downloaded manifest using download key. For logging, -v does not seem to work, so using --log-file=download.log to save file in in mounted host volume.

At the -m flag, only specify name of the manifest file, e.g., controlled_manifest.tsv and not the whole path. docker container will start with container work directory, /scratch which is mapped to /mnt/myscratch on the host machine. So, docker container would look for controlled_manifest.tsv in the mounted /scratch directory, i.e., /mnt/myscratch/controlled_manifest.tsv location on the host machine!

Notes:

For controlled access data, avoid using daemon mode with -d flag before testing that API token, gdc_token.key is working else you may end up requesting too many login requests and get blocked of further data access.
In case of authorization failure, kill docker container using docker kill <container ID> command. Check container ID using command to check running docker processes: docker ps

To download open access data:

Remove -t gdc_token.key and replace -m controlled_manifest.tsv with -m open_manifest.tsv

## path where data will be stored on the host machine
export USERMOUNT="/fastscratch/foo/dump/gdc/test"

cd "${USERMOUNT}"

## MAKE SURE TO GIVE PROPER USER AND GROUP IDs, matching to those of host machine
docker run -d -e HOSTUSER=$USER -e HOSTGROUP=$(id -gn $USER) -e HOSTUSERID=$UID -e HOSTGROUPID=$(id -g $USER) -v "${USERMOUNT}":/scratch sbamin/gdc-client "gdc-client download --log-file=download.log -n 4 -m open_manifest.tsv"

To track download progress:

docker ps
docker logs <container NAME or ID>

To debug in case of download failure, add --debug flag. Not recommended if gdc-client download is working properly else this will increase write operations a lot!

docker run -d -e HOSTUSER=$USER -e HOSTGROUP=$(id -gn $USER) -e HOSTUSERID=$UID -e HOSTGROUPID=$(id -g $USER) -v "${USERMOUNT}":/scratch sbamin/gdc-client "gdc-client download --debug --log-file=download.log -n 4 -t gdc_token.key -m controlled_manifest.tsv"

To parallelize downloads

Instead of supplying download manifest, you can supply analysis UUID, i.e., first column value of the manifest, and run multiple (prefer not to run more than 2-3 on one compute node) docker instances using one the above two command for open or controlled access data, respectively.

gdc-docker-dn-x64-el6 is a bash wrapper to start per-sample (analysis UUID) docker download instance. It's not ready for release yet, but it formats following docker run command by taking a few user-supplied arguments.

docker run -d --name 1da7105a-f0ff-479d-9f82-6c1d94456c91 -e HOSTUSER=foo -e HOSTGROUP=staff -e HOSTUSERID=1000 -e HOSTGROUPID=1001 -v /fastscratch/foo/gdc:/scratch sbamin/gdc-client:1.3.0.b1 "gdc-client download --log-file=/scratch/docker_logs/docker_1da7105a-f0ff-479d-9f82-6c1d94456c91_22Apr18_124819EDT.log -n 8 -t gdc_token.key 1da7105a-f0ff-479d-9f82-6c1d94456c91"

manpage

PS: Valid volume mounts are required (see below) before executing gdc-client

docker run -e HOSTUSER=$USER -e HOSTGROUP=$(id -gn $USER) -e HOSTUSERID=$UID -e HOSTGROUPID=$(id -g $USER) -v "${USERMOUNT}":/scratch sbamin/gdc-client "gdc-client download --help"

docker run -e HOSTUSER=$USER -e HOSTGROUP=$(id -gn $USER) -e HOSTUSERID=$UID -e HOSTGROUPID=$(id -g $USER) -v "${USERMOUNT}":/scratch sbamin/gdc-client "gdc-client upload --help"

END