getting lots of "borken pipe" errors while importing
aaronkaplan opened this issue · 4 comments
aaronkaplan commented
./create_db.py again:
After some time, while parsing the APNIC file, the subprocesses throw lots of error:
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/aaron/miniconda3/lib/python3.7/multiprocessing/queues.py", line 242, in _feed
send_bytes(obj)
File "/home/aaron/miniconda3/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/aaron/miniconda3/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/home/aaron/miniconda3/lib/python3.7/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
firefart commented
@aaronkaplan I think this is a follow up error on #9 as there happened an exception somewhere and this messes up the queue code.
I couldn't reproduce both errors, but I finally got some time to migrate the code to python3 byte strings which should solve the encoding errors.
Can you please try again with the latest changes?
aaronkaplan commented
Will do!
… On 23.06.2020, at 10:20, Christian Mehlmauer ***@***.***> wrote:
@aaronkaplan I think this is a follow up error on #9 as there happened an exception somewhere and this messes up the queue code.
I couldn't reproduce both errors, but I finally got some time to migrate the code to python3 byte strings which should solve the encoding errors.
Can you please try again with the latest changes?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
aaronkaplan commented
Works now, howerver... the data inserted into the pg DB is totally unreadable (i.e. it's not converted to utf-8 on line 63 in create_db.py)
Will send a PR.