JustinBeckwith/linkinator

URL incorrectly reported as broken

Closed this issue · 2 comments

peastman commented

I use linkinator to find broken links in my documentation. It recently began reporting a particular URL as broken, even though it loads without problem in a web browser.

$ npx linkinator https://www.pnas.org/content/105/17/6290
🏊‍♂️ crawling https://www.pnas.org/content/105/17/6290
[503] https://www.pnas.org/content/105/17/6290
[200] https://www.cloudflare.com/5xx-error-landing/

  [503] https://www.pnas.org/content/105/17/6290
https://www.pnas.org/content/105/17/6290
  [200] https://www.cloudflare.com/5xx-error-landing/
ERROR: Detected 1 broken links. Scanned 2 links in 0.567 seconds.
JustinBeckwith commented

Greetings! Sadly, this likely falls into the "won't fix" category for linkinator. Making a basic HTTP request to this page returns an HTTP 503, which then does a redirect in JavaScript on the client. Linkinator does not actively execute JavaScript running on the page, as that would open up a ton of security and performance problems as opposed to doing a basic web crawl. This is a case of the web server breaking the rules of HTTP - they should return a 302! If they did that with the redirect url in a header, things would work as expected.

peastman commented

Thanks! It's good to at least understand what's happening.