Utils to pre-process data for Vitessce.
Sample datasets come from:
- Codeluppi et al.: Spatial organization of the somatosensory cortex revealed by cyclic smFISH
- Dries et al.: Giotto, a pipeline for integrative analysis and visualization of single-cell spatial transcriptomic data
- Wang et al.: Multiplexed imaging of high density libraries of RNAs with MERFISH and expansion microscopy
- Cao et al.: The single-cell transcriptional landscape of mammalian organogenesis
JSON is our target format right now because it is easily read by Javascript, and not so inefficient as to cause problems with storage or processing. For example: The mRNA HDF5 is 30M, but as JSON it is still only 37M.
Set up the vitessce-data environment using conda:
conda env create -f environment.ymlUsers may also install the dependencies with pip:
pip install -r requirements.txtconda activate vitessce-data
# To update with new packages:
conda env update --file environment.yml --prunetest.shexercises all the scripts, using the fixtures infake-files/, and errors if the output is not what is expected.process.shdownloads full data from the internet, caches these input files inbig-files/input, processes them, caches the output inbig-files/output, and pushes to S3.
process.sh only performs the work necessary. To regenerate just a portion of the data,
delete the files in big-files/output that need to be replaced.
Install aws CLI and add to your PATH (reference).
Install gcloud and gsutil and add to your PATH (reference).
Configure the AWS CLI by setting AWS environment variables (reference) or running aws configure (reference).
Configure the Google Cloud CLI by running gcloud auth login (reference).
Update the contents of cloud_target.txt to bump the version number. Then update the version where it is referenced in test fixtures in the fake-files/ directory.