Some tools for local fetching ABCD raw (not preprocedded, yet...) data for local processing
This shell script takes a S3 path, fetches from S3, and prepares the data for local use in a BIDS-like representation
Usage: fetch_a_ABCD.sh Study_Directory Subject_ID image_suffix type S3_path
For example: ./fetch_a_ABCD.sh test sub1 anat T1w s3://NDAR_Central_1/submission_12844/fast-track/mssm/NDARINV007W6H7B_baselineYear1Arm1_ABCD-T1_20170224175304.tgz
This command will:
- Create, if needed, a 'Study_Directory' in the local directory from where the command is run [Note: This should be updated to requiring a full path...]
- Create, if needed, a 'Subject_ID' subdirectory in the Study_Directory
- Create, if needed, a directory called 'type' ('anat' in this example). [Note, it is currently up to the user to name this in BIDS compliant fashion]
- Temporarily creates a 'tmp' directory within this 'type' subdirectory
- Copies the named S3 file into this temporary directory
- unzip's, and dcm2nii's the downloaded S3 file
- Stores the nii.gz version in the 'type' directory, named 'Subject_ID'_'image_suffix'.nii.gz. In this example case, in the 'anat' directory, we have a file called sub1_T1w.nii.gz. [Note, it is currently up to the user to name this in BIDS compliant fashion]
- The dicom's are stored in the 'type' directory in a subdirectory called 'Subject_ID'_'image_suffix_dicom, and the json file is paired with the image file in the 'type' directory
- Finally, the 'tmp' directory is cleaned out
This is a bash script. The following are assumed to be present:
- AWS CLI (Amazon webservices command line interface)
- Chris Rorden's dcm2nii (currently hard coded to /Applications/MRIcron/dcm2nii, would be nice to generalize!)
- NDA Access permission to ABCD, and an 'NDA' profile in your AWS configuration (more about this elsewhere)
- My workflow for NDA access includes use of the NDA downloadmanager commandline tool
I'm not really a programmer, and this is a virtually complete hack; would welcome any cleaning and improvements... There may be various MAC OS-specific things going on here that I've not generalized...
The helper-script 'prep_creds.sh' looks at your AWS credentials, sees if you have an NDA 'personna', creates one for you if you don't, checks if you current one is active, and re-activates it if it isn't. This script should streamline the management of making sure your AWS credentials are in order for the above fetch_a_ABCD.sh script
Expected usage: prep_creds NDA_UserName
This is a bash script. The following are assumed to be present: [TODO]
I'm not really a programmer, and this is a virtually complete hack; would welcome any cleaning and improvements... There may be various MAC OS-specific things going on here that I've not generalized...