jingw/pyhdfs

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))

Closed this issue · 4 comments

Georege commented

Traceback (most recent call last):
File "D:\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
chunked=chunked)
File "D:\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "D:\Anaconda3\lib\http\client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "D:\Anaconda3\lib\http\client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "D:\Anaconda3\lib\http\client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "D:\Anaconda3\lib\http\client.py", line 1065, in _send_output
self.send(chunk)
File "D:\Anaconda3\lib\http\client.py", line 986, in send
self.sock.sendall(data)
ConnectionResetError: [WinError 10054] 远程主机强迫关闭了一个现有的连接。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\Anaconda3\lib\site-packages\requests\adapters.py", line 440, in send
timeout=timeout
File "D:\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "D:\Anaconda3\lib\site-packages\urllib3\util\retry.py", line 357, in increment
raise six.reraise(type(error), error, _stacktrace)
File "D:\Anaconda3\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "D:\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
chunked=chunked)
File "D:\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "D:\Anaconda3\lib\http\client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "D:\Anaconda3\lib\http\client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "D:\Anaconda3\lib\http\client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "D:\Anaconda3\lib\http\client.py", line 1065, in _send_output
self.send(chunk)
File "D:\Anaconda3\lib\http\client.py", line 986, in send
self.sock.sendall(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\workspace\phdfs\check_wrf.py", line 144, in
fs.copy_from_local(parname,"/test/fcst/china/10d_arwpost_sta/near/" + wrflisttime.format("YYYYMMDD") + "/" + parname,overwrite = True)
File "D:\Anaconda3\lib\site-packages\pyhdfs.py", line 753, in copy_from_local
self.create(dest, f, **kwargs)
File "D:\Anaconda3\lib\site-packages\pyhdfs.py", line 426, in create
metadata_response.headers['location'], data=data, **self._requests_kwargs)
File "D:\Anaconda3\lib\site-packages\requests\api.py", line 126, in put
return request('put', url, data=data, **kwargs)
File "D:\Anaconda3\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "D:\Anaconda3\lib\site-packages\requests\sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "D:\Anaconda3\lib\site-packages\requests\sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "D:\Anaconda3\lib\site-packages\requests\adapters.py", line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))

Georege commented

i can't sovle this problem!
can you help me?

jingw commented

This likely isn't something I can help with, as it's probably a problem with your network. Try troubleshooting the curl commands listed on https://hadoop.apache.org/docs/r1.0.4/webhdfs.html

wting861006 commented

@Georege @jingw Hi,Have you sloved this issue?I think this problem is caused by new tcp connection when request every time.
see the pyhdfs.HdfsClient() parameter:
requests_session – A requests.Session object for advanced usage. If absent, this class will use the default requests behavior of making a new session per HTTP request. Caller is responsible for closing session.
my code as below,but it not work:

import pyhdfs
import requests
session=requests.Session()
fs=pyhdfs.HdfsClient(hosts='192.168.8.189,50070',user_name='wangting',requests_session=session)
fs.create(path=savepath,data=file.read())
Georege commented

@wting861006 it's been a long time,i almost forget it,maybe the threads is too much for master，the master can't response all enough（too much small files or the master heap too small）。