lawzava/scrape

error while visiting: Forbidden domain

Closed this issue · 1 comments

Hi,

When I pass the domain https://php.com then it returns error while visiting: Forbidden domain this is because the line scrape.go#L90 call the function trimProtocol scrape.go#L107 and it returns .com .

Basically Trim returns a slice of the string s with all leading and trailing Unicode code points contained in cutset removed.

Instead of Trim if we use TrimPrefix then it returns the correct domain php.com not .com. i.e.

return strings.TrimPrefix(strings.TrimPrefix(requestURL, "http://"), "https://")

If I'm right then can you please update these chages.

FYR: https://play.golang.org/p/vjrqxFPsj_U

My code:

url := "https://php.com"
params := scraper.Parameters{
		Async:               true,
		Emails:              true,
		PrintLogs:           isDevelopment(),
		Recursively:         false,
		Website:             url,
	}
s := scraper.New(params)
err := s.Scrape(&emails)

@kiranupadhyak Thank you for the catch! Implemented & Pushed.