The package is not detecting Ahrefs as a crawler
mkrvaric opened this issue · 1 comments
mkrvaric commented
This line from the access log, is not detected as a crawler. Why Ahrefs bot is not included?
54.36.149.243 - - [29/Jun/2023:08:10:28 +0000] "GET /hr/proizvod/tajice-nike-w-pro-365-tight-35524 HTTP/2.0" 200 16858 "-" "Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)"