hang during syncs of large collections

[tomtor]

After running about an hour gsync always ends hanging in a random transfer.

When running in debug mode and after manual interrupt:

DEBUG: drive/:708:update(): * uploading next chunk...
DEBUG: drive/:708:update(): * uploading next chunk...
DEBUG: drive/:708:update(): * uploading next chunk...
DEBUG: drive/:708:update(): * uploading next chunk...

^CDEBUG: sync/:200:_sync(): Interrupted

Interrupted
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
tom@swan:/media/scratch/backup/gdrive$

[tomtor]

On the current master with some minor changes to get it compiled I get the following stacktrace:

f+++++++++ 2007/maastricht-07/imgp0534.pef
10485760 86% 155.15KB/s 0:01:06^C
Interrupted
sent 2797971312 bytes received 0 bytes 778014.47 bytes/sec
DEBUG: KeyboardInterrupt(): File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/EGG-INFO/scripts/gsync", line 82, in main
Crawler(src, dest).run()
File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/crawler.py", line 193, in run
self._walk(srcpath, self._walk_callback, self._dev)
File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/crawler.py", line 163, in _walk
self._sync(absfile)
File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/init.py", line 263, in call
self._sync(path)
File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/init.py", line 315, in _sync
self.dst.create(dst_path, src_file)
File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/file/init.py", line 443, in create
self.create_file(path, src_obj)
File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/file/__init.py", line 350, in create_file
self._update_data(path, src)
File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/sync/file/remote/__init.py", line 132, in _update_data
progress_callback=progress
File "/usr/local/lib/python2.7/dist-packages/gsync-0.1.14-py2.7.egg/libgsync/drive/init.py", line 874, in update
status, res = req.next_chunk()
File "build/bdist.linux-x86_64/egg/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(_args, *_kwargs)
File "build/bdist.linux-x86_64/egg/apiclient/http.py", line 867, in next_chunk
headers=headers)
File "build/bdist.linux-x86_64/egg/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(_args, *_kwargs)
File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 497, in new_request
redirections, connection_type)
File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1569, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1316, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1285, in _conn_request
response = conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib/python2.7/ssl.py", line 341, in recv
return self.read(buflen)
File "/usr/lib/python2.7/ssl.py", line 260, in read
return self._sslobj.read(len)

[tomtor]

I made a dirty hack which got me uploading through the night:

diff --git a/libgsync/drive/__init__.py b/libgsync/drive/__init__.py
index 7ea043e..9da03e8 100644
--- a/libgsync/drive/__init__.py
+++ b/libgsync/drive/__init__.py
@@ -6,6 +6,12 @@
 """The GSync Drive module that provides an interface to the Google Drive"""

 import os, sys, re, datetime, shelve, time
+import signal
+
+def handler(signum, frame):
+    print 'Signal handler called with signal', signum
+
+signal.signal(signal.SIGALRM, handler)

 from contextlib import contextmanager

@@ -871,7 +877,9 @@ class Drive(object):
                     while res is None:
                         debug(" * uploading next chunk...")

+                        signal.alarm(10)
                         status, res = req.next_chunk()
+                        signal.alarm(0) # Disable the signal
                         if status:
                             progress_callback(status)

[mullenkamp]

I have the same problem...but I don't have a dirty hack... :(

[fnkr]

+1