TeamHG-Memex/scrapy-rotating-proxies

Problem using proxy with scrapy

Nizarazo opened this issue · 1 comments

Hi

I am having problem using proxy with scrapy, I have private proxy with HTTPS supported.

I receive error stating:
2019-07-13 19:31:16 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://www.somewebsite.com via http://127.0.0.1:8050/execute> (failed 201 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [(
'SSL routines', 'ssl3_get_record', 'wrong version number')]>]

I added the proxy to the rotating proxy list as follows:
ROTATING_PROXY_LIST = [
'https://username:password@proxyip:proxyport'
]

Plus I added the two definitions in DOWNLOADER_MIDDLEWARES:
'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
'rotating_proxies.middlewares.BanDetectionMiddleware': 620,

I am using rotating proxy, scrapy and splash version:
Scrapy==1.6.0
scrapy-rotating-proxies==0.6.2
scrapy-splash==0.7.2

my open ssl version is:
OpenSSL 1.0.2k 26 Jan 2017
built on: reproducible build, date unspecified
platform: mingw64
options: bn(64,64) rc4(16x,int) des(idx,cisc,2,long) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include -I/mingw64/include -D_WINDLL -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -D_MT -DDSO_WIN32 -DL_ENDIAN -O3 -Wall -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE -DOPENSSL_IA32_SSE2
-DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
OPENSSLDIR: "/mingw64/ssl"

Plus I tried all the values of DOWNLOADER_CLIENT_TLS_METHOD such as: 'TLSv1.0' 'TLSv1.1' 'TLSv1.2' 'SSLv3' but it didn't change anything same error.

I believe it is a problem with SSL negotiation any ideas what is the problem?

Thanks