Downloading files/folders from synapse with additional jamboree credentials fails
s-hoyt opened this issue · 4 comments
Bug Report
Operating system
MacOS Catalina version 10.15.7
Client version
Output of:
import synapseclient
synapseclient.__version__
'2.2.0'
Description of the problem
Downloading files/folders from synapse with additional jamboree credentials fails.
I am trying to download a folder from synapse, where if I were to do it manually I would click to download each file and then supply my jamboree access key & secret key. I was hoping to do this with the python client because there are a lot of files, but the python client never prompts me for the jamboree keys. Instead each file download silently fails, resulting in an empty list of files.
import synapseclient
import synapseutils
syn = synapseclient.Synapse()
syn.login('synapse_username','password')
files = synapseutils.syncFromSynapse(syn, 'synID')
After running this I don't get any errors, but files is empty
Expected behavior
I expected the files in the folder associated with the synapse ID to be downloaded
Actual behavior
No error, but also no successful downloads.
>>> files
[]
Hello @s-hoyt
Could you provide the synapse id of the folder you are downloading ("synID" in the script above).
Are you able to download these files individually outside of a sync, e.g.
syn.get(<syn_file_id)
If so that would help determine whether the issue is with sync or with the client being able to access these files at all.
Also, could you run the following against one of the files and let me know what the result is?
entity = syn.get(<syn_file_id>, downloadFile=False)
print(entity['_file_handle']['concreteType'])
Thank you.
Hi @jkiang13
The synID of the folder = syn22349509
When I try to download an individual file:
syn22350959 = syn.get(entity='syn22350959')
I get the error:
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found
I am not sure if this is an error in how I set up the client, or an error with this file.
When I run:
entity = syn.get(entity='syn22350959', downloadFile = False)
print(entity['_file_handle']['concreteType'])
I get:
org.sagebionetworks.repo.model.file.ExternalObjectStoreFileHandle
Thank you
Hi @s-hoyt ,
Okay this confirms that these files are accessed through external AWS credentials (I thought that's what you were referring to with the jamboree credentials but I wasn't sure). Unlike typical files stored in Synapse, the Synapse backend does not have access to files stored in this way and only serves as a directory, the clients themselves only can access these files and they need your AWS credentials to do so.
The Synapse Python client is not able to find any default AWS profile with your credentials (i.e. the credentials that you have been supplying on the web for individual downloads).
Can you create a file at path ~/.aws/credentials with the following content (filling in your values on the right side). You can also do this using the aws configure command if you have the awscli installed.
[default]
aws_access_key_id = <aws_access_key_id>
aws_secret_access_key = <aws_secret_access_key>
After creating this file then try again to retrieve an individual file and the sync as you have above.