User-Agent not set
edoardottt opened this issue · 1 comments
edoardottt commented
seems like the user agent is not set properly.
Should be investigated.
edoardottt commented
The User Agent is set for regular requests during crawling. However, colly performs some CONNECT requests with the default Go User Agent.
Example:
CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com:443
Connection: close
User-Agent: Go-http-client/1.1