GDriveReaderDataPipe complains "using a sharing/viewing link instead of a download link"
Opened this issue · 0 comments
huaixv commented
🐛 Describe the bug
Google Drive redirects users to a Virus Warining page, when the file size is large and it cannot scan for virus.
This results in GDriveReaderDataPipe complaining "using a sharing/viewing link instead of a download link".
The following snippet shold reproduce this issue.
A easy solution is replacing the https://drive.google.com/uc? in URL with https://drive.usercontent.google.com/download?.
from functools import partial
import torch
import torchtext
torch.utils.data.datapipes.utils.common.DILL_AVAILABLE = (
torch.utils._import_utils.dill_available()
)
torchtext.disable_torchtext_deprecation_warning()
from torch.utils.data import DataLoader
from torchtext.datasets import CNNDM
cnndm_datapipe = CNNDM(split="test")
cnndm_dataloader = DataLoader(cnndm_datapipe, shuffle=True, batch_size=None)
batch = next(iter(cnndm_dataloader))gives the following error:
Traceback (most recent call last):
File "/home/pc/torch/test.py", line 17, in <module>
batch = next(iter(cnndm_dataloader))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
data = self._next_data()
^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 675, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 41, in fetch
data = next(self.dataset_iter)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 152, in __next__
return self._get_next()
^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 140, in _get_next
result = next(self.iterator)
^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 224, in wrap_next
result = next_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/datapipe.py", line 383, in __next__
return next(self._datapipe_iter)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/sharding.py", line 76, in __iter__
for i, item in enumerate(self.source_datapipe):
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combinatorics.py", line 129, in __iter__
for x in self.datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 193, in wrap_generator
response = gen.send(request)
^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchtext/data/datasets_utils.py", line 339, in __iter__
for _, stream in self.source_datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 193, in wrap_generator
response = gen.send(request)
^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combining.py", line 55, in __iter__
yield from dp
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/fileopener.py", line 66, in __iter__
yield from get_file_binaries_from_pathnames(self.datapipe, self.mode, self.encoding)
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/utils/common.py", line 213, in get_file_binaries_from_pathnames
for pathname in pathnames:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combining.py", line 55, in __iter__
yield from dp
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combining.py", line 55, in __iter__
yield from dp
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/util/cacheholder.py", line 455, in __iter__
for filename in self.source_datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/util/saver.py", line 53, in __iter__
for filepath, data in self.source_datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/callable.py", line 124, in __iter__
for data in self.datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/callable.py", line 124, in __iter__
for data in self.datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/selecting.py", line 72, in __iter__
for data in self.datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/util/tararchiveloader.py", line 54, in __iter__
for data in self.datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/fileopener.py", line 66, in __iter__
yield from get_file_binaries_from_pathnames(self.datapipe, self.mode, self.encoding)
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/utils/common.py", line 213, in get_file_binaries_from_pathnames
for pathname in pathnames:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/util/cacheholder.py", line 230, in __iter__
yield from self.source_datapipe
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/util/cacheholder.py", line 375, in __iter__
for item in self.source_datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 152, in __next__
return self._get_next()
^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 140, in _get_next
result = next(self.iterator)
^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combining.py", line 468, in get_next_element_by_instance
yield self._find_next(instance_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combining.py", line 437, in _find_next
value = next(self._datapipe_iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combining.py", line 55, in __iter__
yield from dp
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/combining.py", line 55, in __iter__
yield from dp
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/util/cacheholder.py", line 455, in __iter__
for filename in self.source_datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/util/saver.py", line 53, in __iter__
for filepath, data in self.source_datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/util/hashchecker.py", line 67, in __iter__
for file_name, data in self.source_datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/callable.py", line 124, in __iter__
for data in self.datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/iter/callable.py", line 124, in __iter__
for data in self.datapipe:
File "/home/pc/torch/venv/lib/python3.11/site-packages/torch/utils/data/datapipes/_hook_iterator.py", line 181, in wrap_generator
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/load/online.py", line 212, in __iter__
yield _get_response_from_google_drive(url, timeout=self.timeout, **self.query_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pc/torch/venv/lib/python3.11/site-packages/torchdata/datapipes/iter/load/online.py", line 140, in _get_response_from_google_drive
raise RuntimeError(
RuntimeError: Google drive link https://drive.google.com/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs&confirm=t internal error: headers don't contain content-disposition. This is usually caused by using a sharing/viewing link instead of a download link. Click 'Download' on the Google Drive page, which should redirect you to a download page, and use the link of that page.
This exception is thrown by __iter__ of GDriveReaderDataPipe(skip_on_error=False, source_datapipe=OnDiskCacheHolderIterDataPipe, timeout=None)
Versions
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Arch Linux (x86_64)
GCC version: (GCC) 13.2.1 20230801
Clang version: 17.0.0
CMake version: version 3.29.0
Libc version: glibc-2.39
Python version: 3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801] (64-bit runtime)
Python platform: Linux-6.2.0-36-generic-x86_64-with-glibc2.39
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Vendor ID: GenuineIntel
Model name: 12th Gen Intel(R) Core(TM) i9-12900K
CPU family: 6
Model: 151
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 2
CPU(s) scaling MHz: 53%
CPU max MHz: 5200.0000
CPU min MHz: 800.0000
BogoMIPS: 6374.40
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 640 KiB (16 instances)
L1i cache: 768 KiB (16 instances)
L2 cache: 14 MiB (10 instances)
L3 cache: 30 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-23
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.3.0
[pip3] torchdata==0.7.1
[pip3] torchtext==0.18.0
[pip3] triton==2.3.0
[conda] Could not collect