Developing workflow to transfer large genome feature files from Andromeda to OSF.io via Globus Connect Personal

[daniellembecker]

Objective:

To develop a streamlined way to transfer large genome feature files (e.g., BAMs, bisulfite-converted genomes, etc) from Andromeda (URI HPC server) to OSF.io through the Globus Connect Personal online database.

I will be following the general methods and protocol for this approach by the Roberts Lab. Their HPC server is called Mox and they have used this to transfer large files in the past.

Notebook post:

I will be continuously updating this notebook post with a helpful workflow at each step I make.

[daniellembecker]

Through this notebook post you can learn how to make your own Globus Connect Personal (GCP) account and add GCP to Andromeda to transfer files between those two endpoints. However, you personally or URI needs a subscription (quoted at ~$3,000) to set up for the URI HPC server to be able to transfer to other accounts.

To bypass this, Kevin Bryan, URI HPC manager, was able to help make the Andromeda access public for specific directories through the use of a S3 client, like rclone.