gsync only create directories

Question

gsync only create directories

Opened this issue 10 years ago · 30 comments

Hello,

I have a directory that I wish to sync with google drive.
My directory, temp, contain one text file, with the name apan.txt and content "apan"

$ gsync -r --progress -v temp/ drive://temp
./
apan.txt
0 0% 0.00B/s 0:00:00
sent 0 bytes received 0 bytes 0.00 bytes/sec

However, only the directory temp is created on the google drive.

What do I do wrong?

/Richard

fnkr commented 10 years ago

+1

Answer 1 · 2014-06-24T09:27:20.000Z

I get this same issue

Answer 2 · 2014-07-07T07:16:07.000Z

Hi,
I also get same issue, some of the progress output :

$ time gsync --owner --group --progress --perms -u -d -r -vi /opt/ict/monday/ drive://ict/

<f+++++++++ 2012/work-jp.docx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/~$noplanning.docx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/PLANNING (edit).xlsx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/Planned 2012.docx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/strategic (c).docx
0 0% 0.00B/s 0:00:00
<f+++++++++ 2012/presentation LATEST.pptx
0 0% 0.00B/s 0:00:00

all shows 0.00B/s ..when checking through Google Drive webpage.. only folder is created but no content...

Answer 3 · 2014-07-08T00:30:24.000Z

Hi.. I fixed it by using method as this thread issue : #66
thanks to @GyroLand
...but some error occured while syncing :

return req.execute()
  File "/usr/lib/python2.6/site-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/apiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)

Error: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/xxxxxxxxxxxxxxxxxxxxxx?alt=json&setModifiedDate=false&newRevision=true returned "Internal Error">
sent 16005692534 bytes  received 0 bytes  4471186.31 bytes/sec
sent 0 bytes  received 0 bytes  0.00 bytes/sec
sent 0 bytes  received 0 bytes  0.00 bytes/sec

... also many files were not sync maybe because of above error?
it takes about 7 hours+ to sync document file total size of 8GB office files .doc, ppt, pdf, etc (this is by using 12 Mbps leased line dedicated Internet)

Answer 4 · 2014-07-12T14:41:06.000Z

Yes, i have the same, i think this is some kind of a time out error or something. Anyway, if you don't use the -u flag and run the program several times, it will upload also the failed files. And it takes much less than the first time, as it uploads only the missing files.

Answer 5 · 2014-07-12T14:59:34.000Z

I have the same issue and can't get files uploaded(with or without the -u
flag). I've tried many times and the files still get ignored(or seem to).

Thanks, Phil

On Sat, Jul 12, 2014 at 10:41 AM, GyroLand notifications@github.com wrote:

Yes, i have the same, i think this is some kind of a time out error or
something. Anyway, if you don't use the -u flag and run the program several
times, it will upload also the failed files. And it takes much less than
the first time, as it uploads only the missing files.

—
Reply to this email directly or view it on GitHub
#69 (comment).

Answer 6 · 2014-07-23T03:25:36.000Z

Also having the directories but no files issue. I don't think it's a timeout error, running with -v shows that it's not spending much time on any given file.

Answer 7 · 2014-07-23T12:43:51.000Z

I've yet to decipher what's happening myself. In my case I found a
workaround by uploading my files manually, killing the md5sum and running
gsync. It's been working well ever since. Must be done strange semi-rare
condition.
On Jul 22, 2014 11:25 PM, "nmatthews" notifications@github.com wrote:

Also having the directories but no files issue. I don't think it's a
timeout error, running with -v shows that it's not spending much time on
any given file.

—
Reply to this email directly or view it on GitHub
#69 (comment).

Answer 8 · 2014-09-19T16:52:39.000Z

$ gsync --version
0.1.14

I'm also having this problem. It seems to have started only recently; old files seem to update (I think), but new ones don't upload. Directories are created, but are unpopulated on Google Drive. I'm wondering if it has to do with having a directory path that's more than one level deep. Here's one example:

gsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/Dane/Recipes/ drive://orchestrator/Recipes

I've tried this with just the "-r -v --progress" options, as well, and it still doesn't upload the files. (I refreshed the Google Drive page a few times, just to be sure.) My network activity applet reports that while (supposedly) uploading directories full of small text files, the download speed is ~12KiB/s, and the upload speed is ~14KiB/s. (My Internet connection is much, much faster than this, so I don't think it's an uplink issue.)

I've tried doing this sync multiple times, and the files haven't yet been uploaded, so the solution, above, about just trying it a few times seems not to be working for me.

Thanks for your work on this! It's truly an indespensible tool!

Answer 9 · 2014-09-19T17:05:30.000Z

It looks like editing /usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py, and changing:

body = {}
        for k, v in properties.iteritems():
                body[k] = _Drive.utf8(v)

...to:

body = {}
        for k, v in properties.iteritems():
            if v is not None:
                body[k] = _Drive.utf8(v)

...solves the problem, per #66.

It does, however, produce this when it runs into a file that ends in a ~ character:

Monsters/Lvl. 3 - Undead Horrid Housecat~
           0   0%     0.00B/s    0:00:00DEBUG: 'Exception':   File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 712, in update
    status, res = req.next_chunk()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 874, in next_chunk
    return self._process_response(resp, content)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 901, in _process_response
    raise HttpError(resp, content, uri=self.uri)

Answer 10 · 2014-09-21T15:54:42.000Z

Worked for me.

Thanks, Phil

On Fri, Sep 19, 2014 at 1:05 PM, dmutters notifications@github.com wrote:

It looks like editing
/usr/local/lib/python2.7/dist-packages/libgsync/drive/init.py, and
changing:

body = {}
for k, v in properties.iteritems():
body[k] = _Drive.utf8(v)

...to:

body = {}
for k, v in properties.iteritems():
if v is not None:
body[k] = _Drive.utf8(v)

...solves the problem, per #66
#66.

—
Reply to this email directly or view it on GitHub
#69 (comment).

Answer 11 · 2014-10-31T11:02:44.000Z

Solution not working for me. Making the change it will not run at all.

Answer 12 · 2014-10-31T18:54:33.000Z

It worked for a while, for me, and then broke. This is what I did to fix it (as root):

(Install pip using your distro's package manager.)
pip uninstall gsync
rm -vfr /usr/local/lib/python2.7/dist-packages/libgsync /usr/local/bin/gsync
pip install gsync
pip install --upgrade gsync

It's once again working for me with only the occasional hiccup.

Answer 13 · 2014-10-31T19:42:00.000Z

the fix worked for me for quite some time and just the other I realized it
had stopped.
On Oct 31, 2014 2:54 PM, "dmutters" notifications@github.com wrote:

It worked for a while, for me, and then broke. This is what I did to fix
it (as root):

(Install pip using your distro's package manager.)

pip uninstall gsync

rm -vfr /usr/local/lib/python2.7/dist-packages/libgsync
/usr/local/bin/gsync

pip install gsync

pip install --upgrade gsync

It's once again working for me with only the occasional hiccup.

—
Reply to this email directly or view it on GitHub
#69 (comment).

Answer 14 · 2014-11-01T01:23:33.000Z

I'm thinking this might be caused by something else.

Found a "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup." error in the cache files link for the files API.

If you

grep -R "selfLink" ~/.gsync/ | more

and look for the https://www.googleapis.com/drive/v2/files/.......... links and try them, I got the response from Google about a limit.

If you also get the key

https://www.googleapis.com/drive/v2/files/0B42Fqxu8L6YfYkZW.........

Then we are all using the same key.

Thank you,

Answer 15 · 2014-11-01T02:32:06.000Z

bingo.

Thanks, Phil

On Fri, Oct 31, 2014 at 9:23 PM, johanels notifications@github.com wrote:

I'm thinking this might be caused by something else.

Found a "Daily Limit for Unauthenticated Use Exceeded. Continued use
requires signup." error in the cache files link for the files API.

If you

grep -R "selfLink" ~/.gsync/ | more

and look for the https://www.googleapis.com/drive/v2/files/..........
links and try them, I got the response from Google about a limit.

If you also get the key

https://www.googleapis.com/drive/v2/files/0B42Fqxu8L6YfYkZW.........

Then we are all using the same key.

Thank you,

—
Reply to this email directly or view it on GitHub
#69 (comment).

Answer 16 · 2014-11-01T06:04:39.000Z

Ok, sow how do I fix this?

Answer 17 · 2014-11-01T06:28:17.000Z

My logs are full of this notice. Thanks for pointing it out.

How can we see what key we're using? That would be a pretty big problem if we're all using the same key, and there's a limit on how much data transfer can be ascribed to any one key.

Answer 18 · 2014-11-01T06:50:02.000Z

Hi,

The Google JSON key information is loaded in /usr/local/lib/python2.7/dist-packages/libgsync/drive/client_json.py and are the same from the source, so everyone is using the same key.

https://github.com/iwonbigbro/gsync/blob/master/libgsync/drive/client_json.py

I created a new project on Google Developers site, created the JSON auth and changed the file on my machine. I see the requests in Google Developers Dashboard, but number of requests matches number of errors, and gsync still does not upload.

https://console.developers.google.com

Will keep trying.

Answer 19 · 2014-11-01T08:42:06.000Z

Thanks, johanels. Let us know when you get it sorted! Hopefully, such a fix can be integrated into the gsync code...

Answer 20 · 2014-11-04T19:22:46.000Z

Same issue here, only empty directories created.

Answer 21 · 2014-12-04T05:19:16.000Z

#66 fix also worked here

Answer 22 · 2015-01-06T13:38:48.000Z

As johanels indicated, the gsync utility is set to use pre-defined Google API key / credentials when shipped.

This means that by default all users will utilise the data limit (typically 10m API requests per day).

Therefore, you need to create your own Google API Project with associated key / credentials.

Method to resolve this:

Goto: https://console.developers.google.com/project/
Create a new project and go to that Projects Dashboard, then:
Enable an API -> Set Drive API to On
APIs & Auth -> Consent Screen. Configure the consent screen, set a project name 'gsync-someotherthing' (or suchlike) and any relevant information
APIs & Auth -> Credentials -> Create new client ID. Select 'installed application'. Type: Other
As root, edit /usr/local/lib/python2.7/dist-packages/libgsync/drive/client_json.py and modify the client_id and client_secret to reflect your newly generated client credentials
Also apply fix #66
As your required (suggest non-root..) user, run gsync --authenticate and follow the instructions as per the standard docs
Uploading of files will work, against your own personal API requests quota

All the best,

Jez

Answer 23 · 2015-01-06T19:31:05.000Z

Thank you Jez. Seems to be running and I can see Response codes in the Development Console.

Answer 24 · 2015-01-06T20:04:12.000Z

Great. Have fun.

The solution is clearly to move the client_json requirement to within a
per user .gsync folder.

I plan on submitting a patch for this in the near future.

Jez

On Tuesday, 6 January 2015, johanels notifications@github.com wrote:

Thank you Jez. Seems to be running and I can see Response codes in the
Development Console.

—
Reply to this email directly or view it on GitHub
#69 (comment).

This email is confidential in that it is intended for the exclusive
attention of the addressee(s) indicated. If you are not the intended
recipient, this email should not be read or disclosed to any other person.
Please notify the sender immediately and delete this email from your
computer system. Any opinions expressed are not necessarily those of the
company from which this email was sent and, whilst to the best of our
knowledge no viruses or defects exist, no responsibility can be accepted
for any loss or damage arising from its receipt or subsequent use of this
email.

Answer 25 · 2015-01-08T13:10:22.000Z

Hi Jez, I've been following this fix and am struggling as I don't seem to have the directory:

/usr/local/lib/python2.7/dist-packages/libgsync/drive/

I'm on Centos 6.6 server and have tried uninstalling and re-installing GSync following carrying out an installation of Python 2.7.

I would appreciate your guidance.

Many thanks,
Simon

Answer 26 · 2015-01-08T16:41:56.000Z

Hi

So I don't have access to a CentOS 6.6 right this second.

However, the issue is highly likely to be just that libgsync is in a different place to the one on my system.

To find it, you could do either:

jtucker@elmo:~/Desktop$ pip show -f gsync
---
Name: gsync
Version: 0.1.14
Location: /usr/local/lib/python2.7/dist-packages
Requires: google-api-python-client, docopt, httplib2, oauth2client, python-dateutil, urllib3
Files:
Cannot locate installed-files.txt

So libgsync on mine is under: /usr/local/lib/python2.7/dist-packages

(I'm presuming you've used pip, if you've used the git version, you should know where you've installed it ;-)

or

updatedb
locate drive | grep libgsync

or

find / -name "drive" -type d -print | grep libgsync

All the best,

Jez

Answer 27 · 2015-01-08T16:46:28.000Z

Thank you oh so much. The first command you issued showed me that instead of dist-packages, it was within site-packages.

I shall progress now. Many many thanks.

Answer 28 · 2015-03-10T19:48:32.000Z

https://github.com/google/skicka already handles this, and appears to also meet my basic needs for syncing content up and down to google drive.

Answer 29 · 2015-10-14T12:14:31.000Z

Hi there, I'm still getting the exceeded 30GB issue, but my client_json.py file has the google API secret in it as followed by the steps above. It's giving me the following error:

DEBUG: 'Exception': File "/usr/local/lib/python2.7/site-packages/libgsync/drive/init.py", line 703, in update
return req.execute()
File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 142, in positional_wrappe r
return wrapped(_args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 729, in execute
raise HttpError(resp, content, uri=self.uri)

DEBUG: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/0B2oFXY6ToU6fNE04aUY wclZFMTg?alt=json&setModifiedDate=true&newRevision=true returned "Internal Error">: File "/usr/loc al/lib/python2.7/site-packages/libgsync/crawler.py", line 150, in run
self._walk(srcpath, self._walkCallback, self._dev)
File "/usr/local/lib/python2.7/site-packages/libgsync/crawler.py", line 124, in _walk
self._sync(f)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/init.py", line 31, in call
self._sync(path)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/init.py", line 197, in _sync
self.dst.update(dstPath, srcFile)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/init.py", line 325, in update
self.updateFile(path, srcObj)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/__init.py", line 228, in updat eFile
self.__updateStats(path, src)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/__init.py", line 273, in updat eStats
self._updateStats(path, src, mode, uid, gid, mtime, atime)
File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/remote/__init.py", line 148, in _updateStats
'setModifiedDate': GsyncOptions.times
File "/usr/local/lib/python2.7/site-packages/libgsync/drive/init.py", line 703, in update
return req.execute()
File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 142, in positional_wrappe r
return wrapped(_args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 729, in execute
raise HttpError(resp, content, uri=self.uri)

Error: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/0B2oFXY6ToU6fNE04aUY wclZFMTg?alt=json&setModifiedDate=true&newRevision=true returned "Internal Error">
sent 307341525 bytes received 0 bytes 60409.28 bytes/sec

When I go in to the link, it tells me the following:

{
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "dailyLimitExceededUnreg",
"message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.",
"extendedHelp": "https://code.google.com/apis/console"
}
],
"code": 403,
"message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."
}
}

but the issue is that I already have a google API Secret with an account that has a 1TB limit.

Please can I have further assistance?