tampe125/hashtag-scraper

Connection Issue

Closed this issue · 4 comments

trounce1 commented

Hi, i keep getting this error after a minute or so of running

Traceback (most recent call last):
File "hashtag-scraper.py", line 135, in
scraper.run()
File "hashtag-scraper.py", line 96, in run
for tweet in tweets:
File "/usr/local/lib/python2.7/dist-packages/twitter/api.py", line 4383, in GetStreamSample
for line in resp.iter_lines():
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 720, in iter_lines
for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 679, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read, 203 more expected)', IncompleteRead(0 bytes read, 203 more expected))

Do you know why this happens, how to fix this or what to do

Thanks

tampe125 commented

Hello,

this happens when you can't keep up with Twitter feed. If you can't read tweets fast enough, they will close the connection.
I just made a change to automatically resume the connection if it was closed.
Can you please give it a try? If everything works fine, I'll close this issue.

trounce1 commented

I'm just going to try it now, i will let you know how it works out

trounce1 commented

it appears to break again

13:53:24|[INFO] Fetched more than 500 hashtags, dumping them to file
13:58:46|[INFO] Fetched more than 500 hashtags, dumping them to file
14:03:04|[INFO] Fetched more than 500 hashtags, dumping them to file
14:07:47|[INFO] Fetched more than 500 hashtags, dumping them to file
14:12:45|[INFO] Fetched more than 500 hashtags, dumping them to file
14:17:51|[INFO] Fetched more than 500 hashtags, dumping them to file
14:20:55|[WARNING] Connection error, resuming...
14:20:57|[INFO] Request tweet stream...
14:20:57|[INFO] Got stream, starting to analyze tweets
Traceback (most recent call last):
File "hashtag-scraper.py", line 146, in
scraper.run()
File "hashtag-scraper.py", line 102, in run
for tweet in tweets:
File "/usr/local/lib/python2.7/dist-packages/twitter/api.py", line 4382, in GetStreamSample
resp = self._RequestStream(url, 'GET')
File "/usr/local/lib/python2.7/dist-packages/twitter/api.py", line 4813, in _RequestStream
raise TwitterError(str(e))
twitter.error.TwitterError: ('Connection aborted.', BadStatusLine("''",))

trounce1 commented

i keep getting this twitter error. i have no idea why. it comes after the connection dies