Problems with credentialing
Closed this issue · 5 comments
See #49 as the genesis of this issue, i.e. why is ResponseNotReady exception being thrown?
One hypothesis was that our GDAC ops python virtual environment was flawed (e.g. by using old or mismatched oauth2 and/or httplib, etc). But that does not appear to be the case because the same ResponseNotReady exception was thrown even when I built a new Python 2.7.12 venv from scratch (which installed latest oauth2 and httplib packages), using the same underlying python binaries but nothing else from our GDAC operational python venv.
Another interesting twist is that it behaves differently upon different runs (see Attempts 1 and 2 below, on cga3, my first 2 runs of the day today).
Only way I saw to reliably mitigate the ResponseNotReady exception was to set
export NO_GCE_CHECK=True
in the environment prior to running the fiss command.
That brings us to the question of why the exception is occurring, and is why in the issue I asked you to explain your strategy about credentialing.
Because, when the code is explicitly checking for application credentials ... and one THEN DELETES those application credentials ... why would one expect it to work rather than complain and/or throw an exception?
Either way, this is a new issue, about credentialing, namely that our code does not exhibit a complete understanding of the edge cases in how it works. Finally, I went back one release to 0.16.3 (before you added caching) and see the same ResponseNotReady problem, so I don't think it's caching, per se, that is the issue.
Attempt 1: first run on cga3 this morning ... note that it raises ApplicationDefaultCredentials exception
% cd /broad/hptmp/mnoble/fiss_debug
% venv/bin/python ~mnoble/src/fissfc/firecloud/fiss.py -V monitor -p nci-mnoble-bi-org -w dev
File "/home/unix/mnoble/src/fissfc/firecloud/fiss.py", line 2244, in main_as_cli
result = main(argv)
File "/home/unix/mnoble/src/fissfc/firecloud/fiss.py", line 2234, in main
result = args.func(args)
File "/home/unix/mnoble/src/fissfc/firecloud/fiss.py", line 1123, in monitor
r = fapi.list_submissions(args.project, args.workspace)
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/firecloud/api.py", line 915, in list_submissions
return __get(uri)
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/firecloud/api.py", line 57, in __get
headers = _fiss_access_headers()
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/firecloud/api.py", line 45, in _fiss_access_headers
__CREDENTIALS = GoogleCredentials.get_application_default()
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/oauth2client/client.py", line 1271, in get_application_default
return GoogleCredentials._get_implicit_credentials()
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/oauth2client/client.py", line 1261, in _get_implicit_credentials
raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Attempt 2: same test, run immediately after the first, now raises our not-so-friendly ResponseNotReady exception:
% venv/bin/python ~mnoble/src/fissfc/firecloud/fiss.py -V monitor -p nci-mnoble-bi-org -w dev
venv/bin/python ~/src/fissfc/firecloud/fiss.py -V monitor -p nci-mnoble-bi-org -w dev
File "/home/unix/mnoble/src/fissfc/firecloud/fiss.py", line 2244, in main_as_cli
result = main(argv)
File "/home/unix/mnoble/src/fissfc/firecloud/fiss.py", line 2234, in main
result = args.func(args)
File "/home/unix/mnoble/src/fissfc/firecloud/fiss.py", line 1123, in monitor
r = fapi.list_submissions(args.project, args.workspace)
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/firecloud/api.py", line 915, in list_submissions
return __get(uri)
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/firecloud/api.py", line 57, in __get
headers = _fiss_access_headers()
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/firecloud/api.py", line 45, in _fiss_access_headers
__CREDENTIALS = GoogleCredentials.get_application_default()
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/oauth2client/client.py", line 1271, in get_application_default
return GoogleCredentials._get_implicit_credentials()
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/oauth2client/client.py", line 1256, in _get_implicit_credentials
credentials = checker()
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/oauth2client/client.py", line 1187, in _implicit_credentials_from_gce
if not _in_gce_environment():
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/oauth2client/client.py", line 1042, in _in_gce_environment
if NO_GCE_CHECK != 'True' and _detect_gce_environment():
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/oauth2client/client.py", line 999, in _detect_gce_environment
http, _GCE_METADATA_URI, headers=_GCE_HEADERS)
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/oauth2client/transport.py", line 282, in request
connection_type=connection_type)
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1659, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1399, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/broad/hptmp/mnoble/fiss_debug/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1355, in _conn_request
response = conn.getresponse()
File "/xchip/tcga/Tools/python_builds/python__2.7.12/lib/python2.7/httplib.py", line 1123, in getresponse
raise ResponseNotReady()
ResponseNotReady:
Note that this report
requests/requests-oauthlib#207
describes an eerily similar situation, where it works first time then returns ResponseNotReady.
test3.py.txt
This behavior reproduces with a very short test program - if NO_GCE_CHECK is set to 'True', you get a helpful error message if the credential file is gone; if it is not set, you get a ResponseNotReady exception. Having an exception rather than a nice error appears to be a bug in Google's oauth2client code.
Moving forward I recommend:
-
moving off the oath2client package, which was deprecated a few months ago, and onto the google-auth package. They said oath2client was getting too messy.
-
Check three things, in order:
a) if a json credential file is passed in on the fiss commandline, use that. This supports DSDE use cases.
b) try to load the json credential file set via 'gcloud init'. This is the same as what gsutil uses, so it won't require an additional oauth dance to set the application-default credentials as well. Unfortunately this appears to require using undocumented functions and information, and so may be fragile.
c) try to load the application-default credentials. This appears to be the path Google is pushing people towards, and so is unlikely to break.
The function in the attached file implements these items, though it would be good to discuss what would be appropriate error handling and status messages before incorporating it into the code.
Thank you, Gordon, this is just the kind of research and summary I was looking for. Let's talk in Fri GDAC meeting how to splice this into the codebase. I know that Sam also recently gave a branch of code to DSDE for testing, but I don't remember what functional area it encompassed--possibly this one, too.
Looking at the documentation, this has the potential to be a much cleaner implementation, with the recommendation being something Mike mentioned a while ago - maintaining a persistent authenticated Requests session:
https://google-auth.readthedocs.io/en/latest/user-guide.html#making-authenticated-requests
Implementing like this should also simplify abstraction if forms of authentication beyond Google are ever added by simply switching out an authenticated requests session with another one using a different authentication scheme.