Consider inspecting headers when HTTP 429 is encountered

[mzpqnxow]

I see that when there is an HTTP level failure for a specific CT server, there is a sleep/retry behavior implemented. Makes sense, clearly you anticipated spurious failures and are handling them:

certstream-server/lib/certstream/ct_watcher.ex

Lines 63 to 84 in 41c0547

	def http_request_with_retries(full_url, options \\ @default_http_options) do
	# Go ask for the first 512 entries
	Logger.info("Sending GET request to #{full_url}")
	user_agent = {"User-Agent", user_agent()}
	case HTTPoison.get(full_url, [user_agent], options) do
	{:ok, %HTTPoison.Response{status_code: 200} = response} ->
	response.body
	|> Jason.decode!
	{:ok, response} ->
	Logger.error("Unexpected status code #{response.status_code} fetching url #{full_url}! Sleeping for a bit and trying again...")
	:timer.sleep(:timer.seconds(10))
	http_request_with_retries(full_url, options)
	{:error, %HTTPoison.Error{reason: reason}} ->
	Logger.error("Error: #{inspect reason} while GETing #{full_url}! Sleeping for 10 seconds and trying again...")
	:timer.sleep(:timer.seconds(10))
	http_request_with_retries(full_url, options)
	end
	end

I'm entering this issue because I recently starting seeing 429s somewhat often on Digicert CT servers.

What are your thoughts on extending the logic in the retry to look at rate-limiting headers if present, and if present, use that value instead of the hard-coded sleep value? I haven't confirmed yet if the (soft) failures I see actually include a Retry-After header, but am guessing there's a half-decent chance they do

I know that the current solution keeps things simple while also not losing any data- to handle 429 fully would really just be a courtesy to the CT servers advising the client to slow down. One could also say it would make certstream-server more "correct" too, if you're into that sort of thing ;)

References

• Mozilla Developers resource doc for HTTP 429

Output from certstream-server with a short flurry of 429s from Digicert

16:47:12.204 [error] Unexpected status code 429 fetching url https://nessie2023.ct.digicert.com/log/ct/v1/get-entries?start=129276528&end=129276783! Sleeping for a bit and trying again...

16:47:12.207 [error] Unexpected status code 429 fetching url https://nessie2023.ct.digicert.com/log/ct/v1/get-entries?start=129277552&end=129277807! Sleeping for a bit and trying again...

16:47:12.210 [error] Unexpected status code 429 fetching url https://nessie2023.ct.digicert.com/log/ct/v1/get-entries?start=129277040&end=129277295! Sleeping for a bit and trying again...

16:47:15.670 [error] Unexpected status code 429 fetching url https://nessie2023.ct.digicert.com/log/ct/v1/get-entries?start=129279600&end=129279855! Sleeping for a bit and trying again...

16:47:22.315 [error] Unexpected status code 429 fetching url https://nessie2023.ct.digicert.com/log/ct/v1/get-entries?start=129277552&end=129277807! Sleeping for a bit and trying again...