Clone repo and sample data inside the AWS/HPC machine

[morteza]

Clone repo and sample data inside the AWS/HPC machine

[morteza]

here is the code I used to run codes and download OpenNeuro on HPC:

# I will use spack here, but in addition I also tried module, EasyBuild and pre-built tools:
git clone -c feature.manyFiles=true https://github.com/spack/spack.git
cd spack/bin
spack compiler find    # see ~/.spack/linux/compilers.yml and make sure GCC is 12.2.0 (for Graviton3)
./spack install micromamba
./spack load micromamba
# OR curl -Ls https://micro.mamba.pm/api/micromamba/linux-aarch64/latest | tar -xvj bin/micromamba

# install AWS CLI and download a sample dataset
./spack install awscli
./spack load awscli
aws s3 sync --no-sign-request s3://openneuro.org/ds002843 /shared/projects/TPDLMMMRSEF/data/openneuro/ds002843/