googlearchive/git-rv

Rietveld OAuth 2.0 redirect fails with uncaught 500 sometimes

Closed this issue · 4 comments

fredsa commented

EDIT: Issue title was "git rv submit sometimes hangs trying to update rietveld"

  1. Type git rv submit
  2. Command line output reads:
Checking out origin/review-9780046 at d0e512fda912a85fa6b10faf223f60325c1bda73.
Adding reviewed commits.
Adding commit:
add missing index.yaml to part5-staticfiles branch

Reviewed in https://codereview.appspot.com/9780046/
Replacing review branch 'part5-templates' with newly committed content.
Your browser has been opened to visit:

    https://codereview.appspot.com/get-access-token?port=8001

If your browser is on a different machine then exit and re-run
upload.py with the command-line parameter

  --no_oauth2_webbrowser

Created new window in existing browser session.
  1. After waiting 10 seconds, hit CTRL-C for traceback
^CTraceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code

    exec code in run_globals
  File "__main__.py", line 28, in <module>
  File "__main__.py", line 23, in main
  File "submit.py", line 123, in callback
  File "submit.py", line 99, in __init__
  File "submit.py", line 411, in advance
  File "submit.py", line 140, in verify_approval
  File "submit.py", line 413, in advance
  File "submit.py", line 159, in update_from_metadata
  File "submit.py", line 415, in advance
  File "submit.py", line 174, in create_branch
  File "submit.py", line 417, in advance
  File "submit.py", line 205, in commit
  File "submit.py", line 419, in advance
  File "submit.py", line 227, in push_commit
  File "submit.py", line 423, in advance
  File "submit.py", line 288, in clean_up_local
  File "submit.py", line 425, in advance
  File "submit.py", line 385, in clean_up_review
  File "submit.py", line 303, in __get_xsrf_server
  File "upload.py", line 467, in _Authenticate
  File "upload.py", line 844, in __call__
  File "upload.py", line 779, in GetAccessToken
  File "upload.py", line 756, in WaitForAccessToken
  File "/usr/lib/python2.7/SocketServer.py", line 265, in handle_request
    fd_sets = select.select([self], [], [], timeout)
KeyboardInterrupt
dhermes commented

What was happening in the browser? The OAuth 2.0 pieces of Rietveld block while waiting to handle a single request:

def WaitForAccessToken(port=DEFAULT_OAUTH2_PORT)
  httpd = ClientRedirectServer((LOCALHOST_IP, port), ClientRedirectHandler)
  # Wait to serve just one request before deferring control back
  # to the caller of wait_for_refresh_token
  httpd.handle_request()

So this would cause the entire thread to block (but it should, since it can't do a think without the token).

Any idea how we could reliably reproduce?

fredsa commented

It's happened to me a couple of times but I don't know (yet) how to reproduce.

The browser just sits there on a blank page (I believe the visible URL is https://codereview.appspot.com/get-access-token?port=8001 -- I'll check next time it happens). Refreshing the browser page does nothing to release the client.

Feel like a race condition.

dhermes commented

After going through the logs and discussing with @fredsa it's clear this is caused by a 500 response instead of a 302 redirect with a Location header set when

/get-access-token?port=8001

is requested.

This was due to a token refresh failing and the resulting error not being caught:

Traceback (most recent call last):
  File "/python27_runtime/python27_lib/versions/third_party/django-1.3/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/s~codereview-hr/1071-81e6672dc680.367625627044972957/codereview/views.py", line 743, in login_wrapper
    return func(request, *args, **kwds)
  File "/base/data/home/apps/s~codereview-hr/1071-81e6672dc680.367625627044972957/codereview/views.py", line 4366, in get_access_token
    credentials.refresh(httplib2.Http())
  File "/base/data/home/apps/s~codereview-hr/1071-81e6672dc680.367625627044972957/third_party/oauth2client/client.py", line 516, in refresh
    self._refresh(http.request)
  File "/base/data/home/apps/s~codereview-hr/1071-81e6672dc680.367625627044972957/third_party/oauth2client/client.py", line 663, in _refresh
    self._do_refresh_request(http_request)
  File "/base/data/home/apps/s~codereview-hr/1071-81e6672dc680.367625627044972957/third_party/oauth2client/client.py", line 682, in _do_refresh_request
    self.token_uri, method='POST', body=body, headers=headers)
  File "/base/data/home/apps/s~codereview-hr/1071-81e6672dc680.367625627044972957/third_party/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/base/data/home/apps/s~codereview-hr/1071-81e6672dc680.367625627044972957/third_party/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/base/data/home/apps/s~codereview-hr/1071-81e6672dc680.367625627044972957/third_party/httplib2/__init__.py", line 1286, in _conn_request
    response = conn.getresponse()
  File "/python27_runtime/python27_dist/lib/python2.7/httplib.py", line 507, in getresponse
    'An error occured while connecting to the server: %s' % e)
error: An error occured while connecting to the server: Unable to fetch URL: https://accounts.google.com/o/oauth2/token

The required fix will have to come in Rietveld and I will work with those maintainers to get this in.