crawlergo don't work with some sites like twitter
am0o0 opened this issue · 6 comments
Hey Qianlitp, thank you for this really awesome tool.
here is the whole problem in one picture:
Also when I tried to close the running crawlergo with Ctrl+C, It didn't let me get out of the process.
Maybe something wrong with your network? Try a different proxy?
well
the previous test was on a digital ocean UK droplet, and now I test in an azure US instance both on ubuntu 20.04 face with same problem.
I used the go version go1.18 linux/amd64 to build it and also used Linux version 79
I try to run crawlergo by appending this --request-proxy http://127.0.0.1:1080 and again, same problem.
Is there any option for sending the debug logs to you?
This log messages show that the browser has timed out accessing the site. Maybe your network is unable to access the site, or something wrong with the chrome browser.
Check the following operations
- Make sure you can access twitter using only your browser, without using crawlergo
-> if not, please try again with a newer browser version? Or check if your network can access twitter using curl command.
-> if you can, please reply your browser version, os environment, and crawlergo compilation information
The problem was related to chromium 79 that the download link exists on README, I downloaded the latest chromium version, and the problem is solved now!
I suggest removing the download link and forcing everyone to download the latest chromium version.
OK, I've deleted it.