versioneye/crawl_r

Some Perl URLs cant be fetched

Closed this issue · 2 comments

reiz commented

In the logs of the cpan crawler I can see stuff like this:

[wtf] Failed to parse JSON response from https://fastapi.metacpan.org/v1/release/FLORA/Catalyst-Runtime-5.80009?join=author - execution expired (pid:197)

But if I try the URL in the browser I'm getting back a valid JSON in less then 1 second. Maybe a redirect issue?

timgluz commented

Ok, i will look into it - but it smells more like they are doing request throttling to prevent DDos;

timgluz commented

i increased timeout for fetch_json in the fetch_product_details function, it should fix this;