dakrone/clj-http

Hanging request

Closed this issue · 3 comments

Bost commented

I execute two parallel requests from Heroku using (get url {:accept :json :debug true}). Sometimes, one of them - the one against http://covid-tracker-us.herokuapp.com/all never ends. I get neither an exception nor any timeout nor anything else. The other one - against https://covid.ourworldindata.org/data/owid-covid-data.json runs always fine. Can you tell me please, am I doing anything wrong? Thanks

If you need to see more, here my get-json function

Jan 30 07:11:06 corona-cases-bot app/web.1 2021-01-30T07:11:05.663+01 INF [corona.common:243] Requesting data from http://covid-tracker-us.herokuapp.com/all ...
Jan 30 07:11:06 corona-cases-bot app/web.1 2021-01-30T07:11:05.663+01 INF [corona.common:243] Requesting data from https://covid.ourworldindata.org/data/owid-covid-data.json ...
Jan 30 07:11:06 corona-cases-bot app/web.1 Request: nil
Jan 30 07:11:06 corona-cases-bot app/web.1 {:user-info nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :use-header-maps-in-response? true,
Jan 30 07:11:06 corona-cases-bot app/web.1  :body-type nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :debug true,
Jan 30 07:11:06 corona-cases-bot app/web.1  :headers
Jan 30 07:11:06 corona-cases-bot app/web.1  {"accept" "application/json", "accept-encoding" "gzip, deflate"},
Jan 30 07:11:06 corona-cases-bot app/web.1  :server-port nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :url "https://covid.ourworldindata.org/data/owid-covid-data.json",
Jan 30 07:11:06 corona-cases-bot app/web.1  :flatten-nested-keys (:query-params),
Jan 30 07:11:06 corona-cases-bot app/web.1  :uri "/data/owid-covid-data.json",
Jan 30 07:11:06 corona-cases-bot app/web.1  :server-name "covid.ourworldindata.org",
Jan 30 07:11:06 corona-cases-bot app/web.1  :query-string nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :body nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :scheme :https,
Jan 30 07:11:06 corona-cases-bot app/web.1  :request-method :get}
Jan 30 07:11:06 corona-cases-bot app/web.1 HttpRequest:
Jan 30 07:11:06 corona-cases-bot app/web.1 {:config nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :method "GET",
Jan 30 07:11:06 corona-cases-bot app/web.1  :requestLine
Jan 30 07:11:06 corona-cases-bot app/web.1  #object[org.apache.http.message.BasicRequestLine 0x3fdb4890 "GET https://covid.ourworldindata.org/data/owid-covid-data.json HTTP/1.1"],
Jan 30 07:11:06 corona-cases-bot app/web.1  :aborted false,
Jan 30 07:11:06 corona-cases-bot app/web.1  :params
Jan 30 07:11:06 corona-cases-bot app/web.1  #object[org.apache.http.params.BasicHttpParams 0x6b7122d1 "[parameters={}]"],
Jan 30 07:11:06 corona-cases-bot app/web.1  :protocolVersion
Jan 30 07:11:06 corona-cases-bot app/web.1  #object[org.apache.http.HttpVersion 0x3f845199 "HTTP/1.1"],
Jan 30 07:11:06 corona-cases-bot app/web.1  :URI
Jan 30 07:11:06 corona-cases-bot app/web.1  #object[java.net.URI 0x76e19e3b "https://covid.ourworldindata.org/data/owid-covid-data.json"],
Jan 30 07:11:06 corona-cases-bot app/web.1  :class org.apache.http.client.methods.HttpGet,
Jan 30 07:11:06 corona-cases-bot app/web.1  :allHeaders
Jan 30 07:11:06 corona-cases-bot app/web.1  [#object[org.apache.http.message.BasicHeader 0x29057b3a "Connection: close"],
Jan 30 07:11:06 corona-cases-bot app/web.1   #object[org.apache.http.message.BasicHeader 0x389ab9fd "accept: application/json"],
Jan 30 07:11:06 corona-cases-bot app/web.1   #object[org.apache.http.message.BasicHeader 0x7381f0df "accept-encoding: gzip, deflate"]]}
Jan 30 07:11:06 corona-cases-bot app/web.1 Request: nil
Jan 30 07:11:06 corona-cases-bot app/web.1 {:user-info nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :use-header-maps-in-response? true,
Jan 30 07:11:06 corona-cases-bot app/web.1  :body-type nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :debug true,
Jan 30 07:11:06 corona-cases-bot app/web.1  :headers
Jan 30 07:11:06 corona-cases-bot app/web.1  {"accept" "application/json", "accept-encoding" "gzip, deflate"},
Jan 30 07:11:06 corona-cases-bot app/web.1  :server-port nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :url "http://covid-tracker-us.herokuapp.com/all",
Jan 30 07:11:06 corona-cases-bot app/web.1  :flatten-nested-keys (:query-params),
Jan 30 07:11:06 corona-cases-bot app/web.1  :uri "/all",
Jan 30 07:11:06 corona-cases-bot app/web.1  :server-name "covid-tracker-us.herokuapp.com",
Jan 30 07:11:06 corona-cases-bot app/web.1  :query-string nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :body nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :scheme :http,
Jan 30 07:11:06 corona-cases-bot app/web.1  :request-method :get}
Jan 30 07:11:06 corona-cases-bot app/web.1 HttpRequest:
Jan 30 07:11:06 corona-cases-bot app/web.1 {:config nil,
Jan 30 07:11:06 corona-cases-bot app/web.1  :method "GET",
Jan 30 07:11:06 corona-cases-bot app/web.1  :requestLine
Jan 30 07:11:06 corona-cases-bot app/web.1  #object[org.apache.http.message.BasicRequestLine 0x31175e7 "GET http://covid-tracker-us.herokuapp.com/all HTTP/1.1"],
Jan 30 07:11:06 corona-cases-bot app/web.1  :aborted false,
Jan 30 07:11:06 corona-cases-bot app/web.1  :params
Jan 30 07:11:06 corona-cases-bot app/web.1  #object[org.apache.http.params.BasicHttpParams 0x269f8da2 "[parameters={}]"],
Jan 30 07:11:06 corona-cases-bot app/web.1  :protocolVersion
Jan 30 07:11:06 corona-cases-bot app/web.1  #object[org.apache.http.HttpVersion 0x3f845199 "HTTP/1.1"],
Jan 30 07:11:06 corona-cases-bot app/web.1  :URI
Jan 30 07:11:06 corona-cases-bot app/web.1  #object[java.net.URI 0x2076a5d8 "http://covid-tracker-us.herokuapp.com/all"],
Jan 30 07:11:06 corona-cases-bot app/web.1  :class org.apache.http.client.methods.HttpGet,
Jan 30 07:11:06 corona-cases-bot app/web.1  :allHeaders
Jan 30 07:11:06 corona-cases-bot app/web.1  [#object[org.apache.http.message.BasicHeader 0x17eb31f "Connection: close"],
Jan 30 07:11:06 corona-cases-bot app/web.1   #object[org.apache.http.message.BasicHeader 0x7214bb7a "accept: application/json"],
Jan 30 07:11:06 corona-cases-bot app/web.1   #object[org.apache.http.message.BasicHeader 0x71d08036 "accept-encoding: gzip, deflate"]]}
Jan 30 07:11:27 corona-cases-bot heroku/web.1 Process running mem=700M(136.8%)
Jan 30 07:11:50 corona-cases-bot heroku/web.1 Process running mem=766M(149.6%)
Jan 30 07:12:00 corona-cases-bot app/web.1 2021-01-30T07:11:50.470+01 INF [corona.common:264] Requesting data from https://covid.ourworldindata.org/data/owid-covid-data.json ... 66742984 B received in 54511 ms
rymndhng commented

You need to set the timeouts explicitly. The default value 0 is to block indefinitely:

  • :connection-timeout
  • :socket-timeout

See the documented examples in the README: https://github.com/dakrone/clj-http#get

Bost commented

I've been testing your advice for a while and it looks like it helps. Thanks, I'm closing this issue.

  • 👍1
rymndhng commented

Glad to hear :)