Deposit single cell profiles in S3

Question

Deposit single cell profiles in S3

Opened this issue 3 years ago · 5 comments

Answer 1 · 2022-02-25T20:20:23.000Z

First, move data to s3://cellpainting-gallery

aws s3 sync \
  --profile  jump-cp-role  \
  --acl bucket-owner-full-control \
  s3://cellpainting-datasets/Rosetta-GE-CP/preprocessed_data \
  s3://cellpainting-gallery/rosetta/broad/workspace/preprocessed_data

Answer 2 · 2022-02-25T20:33:54.000Z

@MarziehHaghighi I have successfully copied the files over from s3://cellpainting-datasets/Rosetta-GE-CP/preprocessed_data. You can now delete the files at s3://cellpainting-datasets/Rosetta-GE-CP/preprocessed_data.

Note that the topic of this issue is about single cell data, but I have not gotten to that yet. This is only about copying the existing data to the s3://cellpainting-gallery/

Answer 3 · 2022-02-25T20:44:11.000Z

@shntnu How can I have access to s3://cellpainting-gallery for adding DVC and also any other modifications?

Answer 4 · 2022-02-25T21:03:33.000Z

@shntnu How can I have access to s3://cellpainting-gallery for adding DVC and also any other modifications?

I should have clarified

We won't use DVC after all; just store directly on S3
For now, the best approach to update any data on s3://cellpainting-gallery is to have you upload to s3://cellpainting-datasets/ and then ask me (in this issue) to sync it. It needs more work on my front to make it possible for you to upload to s3://cellpainting-gallery/rosetta

Answer 5 · 2022-04-22T20:38:37.000Z

I did this

aws s3 sync \
  --profile  jump-cp-role  \
  --acl bucket-owner-full-control \
  s3://cellpainting-gallery/rosetta/ \
  s3://cellpainting-gallery/cpg0003-rosetta/


aws s3 rm \
  --profile  jump-cp-role  \
  --recursive \
  s3://cellpainting-gallery/rosetta/