pgaref/HTTP_Request_Randomizer

How to import the library to create the RequestProxy() object

vnjk opened this issue · 2 comments

vnjk commented

Hi, as above:
I have tried importing RequestProxy but with no success.

from project.http.requests.proxy.requestProxy import RequestProxy

Following the style of:

from project.http.requests.parsers.UrlParser import UrlParser

The error returned is:

Traceback (most recent call last):
File "crawler_copy.py", line 18, in
requests_proxy = RequestProxy()
File "/usr/local/lib/python2.7/dist-packages/http_request_randomizer-0.0.3-py2.7.egg/project/http/requests/proxy/requestProxy.py", line 21, in init
self.userAgent = UserAgentManager()
File "/usr/local/lib/python2.7/dist-packages/http_request_randomizer-0.0.3-py2.7.egg/project/http/requests/useragent/userAgent.py", line 8, in init
self.useragents = self.load_user_agents(self.agent_file)
File "/usr/local/lib/python2.7/dist-packages/http_request_randomizer-0.0.3-py2.7.egg/project/http/requests/useragent/userAgent.py", line 16, in load_user_agents
with open(useragentsfile, 'rb') as uaf:
IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/http_request_randomizer-0.0.3->py2.7.egg/project/http/requests/useragent/../data/user_agents.txt'

How can I import the library correctly? Please advise. Thank you

vnjk commented

nevermind, figured out the problem. cheers

@vnjk I fixed an issure in the setup.py file and now the easiest way would be to create a new project and import the pypi library (using requirements.txt for example) http-request-randomizer==0.0.5
After create a python file and test the code below:

import time
from http.requests.proxy.requestProxy import RequestProxy

if __name__ == '__main__':
    print "Hello"
    start = time.time()
    req_proxy = RequestProxy()
    print "Initialization took: {0} sec".format((time.time() - start))
    print "Size : ", len(req_proxy.get_proxy_list())
    print " ALL = ", req_proxy.get_proxy_list()

    test_url = 'http://icanhazip.com'

    while True:
        start = time.time()
        request = req_proxy.generate_proxied_request(test_url)
        print "Proxied Request Took: {0} sec => Status: {1}".format((time.time() - start), request.__str__())
        if request is not None:
            print "\t Response: ip={0}".format(request.text)
        print "Proxy List Size: ", len(req_proxy.get_proxy_list())

        print"-> Going to sleep.."
        time.sleep(10)

Thanks for reporting and sorry for the delay!