Proxypool based on Python3
README_CN.md in Chinese
conda install --yes --file requirements.txt
If you set up redis authentication, you need to set 'REDIS_PASSWORD' in settings.py
Proxypool provides several crawlers on free proxies. It can be seen in proxy_web.txt
If needed, you can add crawlers to crawler.py
cd into the project folder and input the command below:
python run.py
Input http://127.0.0.1:5555 in the browser, then you can see the page by WebAPI encapsulation
/random：provide a proxy randomly
/count：provide the total amount of proxypool
It is very convenient to get proxy by WebAPI.
import requests
PROXY_POOL_URL = 'http://localhost:5555/random'
def get_proxy():
try:
response = requests.get(PROXY_POOL_URL)
if response.status_code == 200:
return response.text
except ConnectionError:
return None
- Project skeleton is based on Python3Webspider
- Comments and suggestions are most welcome!