REST calls may hang forever if something happens to the connection

[arpost]

A long running script using the API every 1 minute hung after several days. Hitting Control-C produced the following stack trace:

^CTraceback (most recent call last):
  File "initial_load.py", line 188, in <module>
    submitted_job = jobs_api.get(job_id)
  File "build/bdist.linux-x86_64/egg/eurekaclinical/eureka/client.py", line 72, in get
  File "build/bdist.linux-x86_64/egg/eurekaclinical/eureka/client.py", line 79, in _get
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/sessions.py", line 483, in get
    return self.request('GET', url, **kwargs)
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/sessions.py", line 471, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/sessions.py", line 581, in send
    r = adapter.send(request, **kwargs)
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/adapters.py", line 417, in send
    timeout=timeout
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 595, in urlopen
    chunked=chunked)
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 352, in _make_request
    self._validate_conn(conn)
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 831, in _validate_conn
    conn.connect()
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 289, in connect
    ssl_version=resolved_ssl_version)
  File "/home/i2b2updatesched/i2b2-etl-python/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 308, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python2.7/ssl.py", line 353, in wrap_socket
    _context=self)
  File "/usr/local/lib/python2.7/ssl.py", line 601, in __init__
    self.do_handshake()
  File "/usr/local/lib/python2.7/ssl.py", line 830, in do_handshake
    self._sslobj.do_handshake()
KeyboardInterrupt

It appears we need to be passing the timeout parameter into our requests with timeout=. See http://stackoverflow.com/questions/17782142/why-doesnt-requests-get-return-what-is-the-default-timeout-that-requests-get.

[arpost]

There should also be a retry mechanism so we don't just fail in the event of a timeout.

[arpost]

Retry is implemented.