error while visiting: Forbidden domain
Closed this issue · 1 comments
kiranupadhyak commented
Hi,
When I pass the domain https://php.com then it returns error while visiting: Forbidden domain
this is because the line scrape.go#L90 call the function trimProtocol scrape.go#L107 and it returns .com
.
Basically Trim
returns a slice of the string s with all leading and trailing Unicode code points contained in cutset removed.
Instead of Trim
if we use TrimPrefix then it returns the correct domain php.com
not .com
. i.e.
return strings.TrimPrefix(strings.TrimPrefix(requestURL, "http://"), "https://")
If I'm right then can you please update these chages.
FYR: https://play.golang.org/p/vjrqxFPsj_U
My code:
url := "https://php.com"
params := scraper.Parameters{
Async: true,
Emails: true,
PrintLogs: isDevelopment(),
Recursively: false,
Website: url,
}
s := scraper.New(params)
err := s.Scrape(&emails)
lawzava commented
@kiranupadhyak Thank you for the catch! Implemented & Pushed.