Error 403 - cannot download dataset with PyDrive

[laszloturanyi]

Hi,

I followed the PyDrive method to download the data. However after succesfully authenticating, I get the following Error 403

Authentication successful.
authorized access to google drive API!
Parsing JSON metadata...
Downloading 70001 files...
- done processing 0/70001 filesTraceback (most recent call last):
  File "/home/tlaci/anaconda3/envs/pydrive/lib/python3.8/site-packages/pydrive/files.py", line 235, in FetchMetadata
    metadata = self.auth.service.files().get(fileId=file_id,
  File "/home/tlaci/anaconda3/envs/pydrive/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/tlaci/anaconda3/envs/pydrive/lib/python3.8/site-packages/googleapiclient/http.py", line 907, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/drive/v2/files/1nGp-wUlRHFJ6aJitkZZlHVTICTLKLJfT?alt=json returned "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=971122666526">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_ffhq_aging.py", line 374, in <module>
    run_cmdline(sys.argv)
  File "download_ffhq_aging.py", line 369, in run_cmdline
    run(**vars(args))
  File "download_ffhq_aging.py", line 348, in run
    download_files(specs, dst_dir, output_size, drive=drive, **download_kwargs)
  File "download_ffhq_aging.py", line 209, in download_files
    raise exc_info[1].with_traceback(exc_info[2])
  File "download_ffhq_aging.py", line 219, in _download_thread
    pydrive_utils.pydrive_download(drive, spec['file_url'], spec['file_path'])
  File "/media/tlaci/F85E3F6B5E3F2234/FFHQ-Aging-Dataset/pydrive_utils.py", line 40, in pydrive_download
    pydrive_file.GetContentFile(save_path)
  File "/home/tlaci/anaconda3/envs/pydrive/lib/python3.8/site-packages/pydrive/files.py", line 210, in GetContentFile
    self.FetchContent(mimetype, remove_bom)
  File "/home/tlaci/anaconda3/envs/pydrive/lib/python3.8/site-packages/pydrive/files.py", line 42, in _decorated
    self.FetchMetadata()
  File "/home/tlaci/anaconda3/envs/pydrive/lib/python3.8/site-packages/pydrive/auth.py", line 75, in _decorated
    return decoratee(self, *args, **kwargs)
  File "/home/tlaci/anaconda3/envs/pydrive/lib/python3.8/site-packages/pydrive/files.py", line 239, in FetchMetadata
    raise ApiRequestError(error)
pydrive.files.ApiRequestError: <HttpError 403 when requesting https://www.googleapis.com/drive/v2/files/1nGp-wUlRHFJ6aJitkZZlHVTICTLKLJfT?alt=json returned "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=971122666526">

Opening the link the .json message shows a bit more detailed message.

{
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "dailyLimitExceededUnreg",
    "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.",
    "extendedHelp": "https://code.google.com/apis/console"
   }
  ],
  "code": 403,
  "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."
 }
}

If the authentication happened how can the problem be "Unauthenticated Use Exceeded"?

[royorel]

Can you please verify that the ffhq-dataset folder is in your shared with me folder on the google drive account you used pydrive with?

[laszloturanyi]

I can verify that. However, after 24 hours I could downloaded it. I think it is rather related with the limitations at the owner's side.

"Continued use requires signup." This message seems to be misleading.

[royorel]

Thanks for the update!

These are indeed limitations imposed by google, which we have no way to control.
Same thing with the error messages, these are google error messages, not ours, so we can't change them.