Amazon CloudFront shouldn't be treated as crawler

Question

sunnydavis opened this issue 4 years ago · 3 comments

Amazon CloudFront is often the user agent for using CloudFront as reverse proxy / CDN so it's actually not a crawler but regular user requests.

Answer 1 · 2020-09-01T20:38:08.000Z

Fancy PRing that?

Thanks 👍🏻

Answer 2 · 2020-09-02T06:17:05.000Z

Sure, it's done. #410

Answer 3 · 2020-09-02T21:34:59.000Z

Merged 👍