Timeout exceeded while awaiting headers

[danielgrabowski]

Hi. First of all thanks for this project. It saved me some time :)

And now the problem I have. I scrape many pages but some of them seem to be problematic (actually the issue happens not only for the one I describe below but this is a good, repeatable example). The scrape fails with error message "request canceled (Client.Timeout exceeded while awaiting headers)".

Here's what I do.
I'm starting docker container:

docker run -p "9271:9271" --rm foomo/pagespeed_exporter -api-key MY_API_KEY

Then in the browser I open URL "http://localhost:9271/probe?target=https://www.morizon.pl/mieszkania/warszawa/"
After 30 seconds I get message in container logs:

time="2020-08-26T20:25:35Z" level=warning msg="target scraping returned an error" error="Get "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?alt=json&category=performance&category=seo&category=pwa&category=best-practices&category=accessibility&key=MY_API_KEY&prettyPrint=false&strategy=desktop&url=https%3A%2F%2Fwww.morizon.pl%2Fmieszkania%2Fwarszawa%2F\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" strategy=desktop target="https://www.morizon.pl/mieszkania/warszawa/"

When I open the PageSpeed API URL mentioned in the log (https://www.googleapis.com/pagespeedonline/v5/runPagespeed?alt=json&category=performance&category=seo&category=pwa&category=best-practices&category=accessibility&key=MY_API_KEY&prettyPrint=false&strategy=desktop&url=https%3A%2F%2Fwww.morizon.pl%2Fmieszkania%2Fwarszawa%2F) I usually get proper metrics.

Is there any way to configure the timeout (30s?). Do you have any ideas why the scrape with your tool fails while direct API request works fine?

I would be very grateful for help since not fixing this issue will force me to implement the exporter myself :(

[danielgrabowski]

I don't know "go" and I'm not quite familiar with Prometheus and PSI internals but it seems I managed to fix the issue myself :)
Checkout my pull request: #30
I tried this in my environment and it works now.