Test data migration
Closed this issue · 5 comments
We would like to migrate large test data out of the repository. There're two things to decide:
- where to put the data
- how we can delete existing large data from the repo (or do we need to?)
where to put the data
we decided to create a Zenodo dataset and retrieve by DOI.
deleting existing large data from repo
BFG cleaner https://rtyley.github.io/bfg-repo-cleaner/
https://docs.github.com/en/enterprise-server@3.0/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository
Will rewrite the history, but may be safe at this point.
Whenever we get a chance, we should register for a zenodo account. It should probably be @tying84 's
Thanks @tdmorello for your comment! My Zenodo account is yfukai just in case.
Zenodo dataset is already created: https://doi.org/10.5281/zenodo.6334810
The remaining task is to clean up the data in the current repo.