Bulk load tsv files
Parul-Kudtarkar opened this issue · 9 comments
Hi,
Is it possible to bulk load tsv,csv files on the portal or do they have to be in JSON format?
Thank you so much!
We have a crude "import" code that can take an .xls file and create the objects -- but its not trivial or very stable, since the schema's are updated frequently. If you are with an ENCODE Producing lab - it's best to work with your DCC wrangler and see what option(s) are the best for your particular situation.
That code is here: https://github.com/ENCODE-DCC/pyencoded-tools/blob/master/ENCODE_import_data.py if you'd like to play with it on the Test server.
@carrieadavis thanks for pointing out the script. I will work with this script and try to adapt for our use case.
@carrieadavis what would it take to post .xls file (i.e. --type target etc.) to a test server running on cloud?
Thank you so much!
You need to be affiliated with an ENCODE lab or other lab that has been approved to submit their data to the DCC - then you can get a key to programmatically access the site and credentials and a wrangler can step you through that process. Is that your situation?
Actually, we have setup our own AWS instance and the code is based on encode codebase. I have user account on this portal and have generated the access Key Id and Secret Key Id. Is it possible to connect to this ec2 instance.
so looks like the --key isn't the unique user access Key Id and Secret Key Id
Okay, so you've now exceeded my expertise here.. I've circled in the more computational types at the DCC into this thread who are better equipped to answer you (stay tuned). If you develop the code and process to take the xls/tsv files and do bulk submission on them, please do let us know!
Thanks @carrieadavis! I look forward to more updates from other folks and will keep you posted.
@carrieadavis figured this out,
In keypair.json
{
"myec2":
{"server":"EC2 publicDNA", "key":"user Access Key ID", "secret":"user secret key"},
}
Thank you so much for your help!