| title | summary |
|---|---|
Title |
summary |
The package is intended to manage a set of proxies, rotating through at random, to prevent blocking. Proxies are currently dropped from the rotation if they fail twice in a row.
VERY EARLY STAGE
pip install ProxyManager
PROXY_LIST = [
'70.45.80.236:4602',
'70.45.80.236:4603',
'70.45.80.236:4604'
]
URL = 'http://www.myurl.com'
from ProxyManager.core import ProxyManager, Proxy
proxies = [i for i in PROXY_LIST] # OR [Proxy(i) for i in PROXY_LIST]
PM = ProxyManager(proxies)
result = PM.make_request(URL)
# Pass a fail function into make_request method, to check for domain/website specific indicators that the request has failed
# i.e. tell the ProxyManager, that this request has failed
def fail_function(response):
if '<something_wrong_about>' in response:
return True
return False
result = PM.make_request(URL, fail_func=fail_function) # or PM.fail_func = fail_functionIf you don't bring your own proxies, which is HIGHLY recommended, I have written a function that scrapes https://free-proxy-list.net/ to obtain a set of proxies.
from ProxyManager.free_proxies import get_free
FREE_PROXIES = get_free(10)
proxies = [Proxy('{}:{}'.format(i['ip'], i['port'])) for i in FREE_PROXIES]