Unable to download pretrained weights
Opened this issue · 8 comments
When running the following to download the pretrained weights:
cd ROOT_DIR/PreDiff
python ./scripts/download_pretrained.py --model all
There are two errors.
sed: -e expression #1, char 31: unterminateds' command`- The files are 2.4k and contain an Google Drive - Virus scan warning.
Example stdout:
python ./scripts/download_pretrained.py --model all
sed: -e expression #1, char 31: unterminated `s' command
--2024-03-01 20:41:16-- https://docs.google.com/uc?export=download&confirm=&id=10OicEQuOPzSKDp5WYF3zDHsL-COywe98
Resolving docs.google.com (docs.google.com)... 142.251.33.110, 2607:f8b0:400a:80a::200e
Connecting to docs.google.com (docs.google.com)|142.251.33.110|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://drive.usercontent.google.com/download?id=10OicEQuOPzSKDp5WYF3zDHsL-COywe98&export=download [following]
--2024-03-01 20:41:16-- https://drive.usercontent.google.com/download?id=10OicEQuOPzSKDp5WYF3zDHsL-COywe98&export=download
Resolving drive.usercontent.google.com (drive.usercontent.google.com)... 142.251.215.225, 2607:f8b0:400a:80b::2001
Connecting to drive.usercontent.google.com (drive.usercontent.google.com)|142.251.215.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2444 (2.4K) [text/html]
Saving to: ‘/fsx/PreDiff/pretrained/vae/pretrained_sevirlr_vae_8x8x64_v1.pt’
/fsx/PreDiff/pretrained/vae/pre 100%[=============================================================================>] 2.39K --.-KB/s in 0.001s
2024-03-01 20:41:16 (2.55 MB/s) - ‘/fsx/PreDiff/pretrained/vae/pretrained_sevirlr_vae_8x8x64_v1.pt’ saved [2444/2444]
When looking at pretrained_sevirlr_vae_8x8x64_v1.pt we see:
<!DOCTYPE html><html><head><title>Google Drive - Virus scan warning</title><meta http-equiv="content-type" content="text/html; charset=utf-8"/><style nonce="6TFDa1DXVpjcW1LC24sRVg">.goog-link-button{position:relative;color:#15c;text-decoration:underline;cursor:pointer}.goog-link-button-disabled{color:#ccc;text-decoration:none;cursor:default}body{color:#222;font:normal 13px/1.4 arial,sans-serif;margin:0}.grecaptcha-badge{visibility:hidden}.uc-main{padding-top:50px;text-align:center}#uc-dl-icon{display:inline-block;margin-top:16px;padding-right:1em;vertical-align:top}#uc-text{display:inline-block;max-width:68ex;text-align:left}.uc-error-caption,.uc-warning-caption{color:#222;font-size:16px}#uc-download-link{text-decoration:none}.uc-name-size a{color:#15c;text-decoration:none}.uc-name-size a:visited{color:#61c;text-decoration:none}.uc-name-size a:active{color:#d14836;text-decoration:none}.uc-footer{color:#777;font-size:11px;padding-bottom:5ex;padding-top:5ex;text-align:center}.uc-footer a{color:#15c}.uc-footer a:visited{color:#61c}.uc-footer a:active{color:#d14836}.uc-footer-divider{color:#ccc;width:100%}.goog-inline-block{position:relative;display:-moz-inline-box;display:inline-block}* html .goog-inline-block{display:inline}*:first-child+html .goog-inline-block{display:inline}sentinel{}</style><link rel="icon" href="//ssl.gstatic.com/docs/doclist/images/drive_2022q3_32dp.png"/></head><body><div class="uc-main"><div id="uc-dl-icon" class="image-container"><div class="drive-sprite-aux-download-file"></div></div><div id="uc-text"><p class="uc-warning-caption">Google Drive can't scan this file for viruses.</p><p class="uc-warning-subcaption"><span class="uc-name-size"><a href="/open?id=10OicEQuOPzSKDp5WYF3zDHsL-COywe98">pretrained_sevirlr_vae_8x8x64.pt</a> (322M)</span> is too large for Google to scan for viruses. Would you still like to download this file?</p><form id="download-form" action="https://drive.usercontent.google.com/download" method="get"><input type="submit" id="uc-download-link" class="goog-inline-block jfk-button jfk-button-action" value="Download anyway"/><input type="hidden" name="id" value="10OicEQuOPzSKDp5WYF3zDHsL-COywe98"><input type="hidden" name="export" value="download"><input type="hidden" name="confirm" value="t"><input type="hidden" name="uuid" value="f8745934-ca2f-490f-8df3-709a8bf952ed"></form></div></div><div class="uc-footer"><hr class="uc-footer-divider"></div></body></html>
This also means the next step of course does not work:
cd ROOT_DIR/PreDiff
MASTER_ADDR=localhost MASTER_PORT=10001 python ./scripts/prediff/sevirlr/train_sevirlr_prediff.py --gpus 2 --pretrained --save tmp_sevirlr_prediff
Is there any chance you can store the data in S3?
I can confirm that I have had the same issue - i think we should move all trained weights to S3 and if some information can be shared on how the data was trained so that it can be reproduced (for example some estimates on time it took to train using what/how many GPUs - assuming of course trained using GPUs)
Thank you very much for reporting the downloading issue from Google Drive.
I have migrated the pretrained weights to OneDrive and addressed the download process in #21 . Please let me know if you continue to experience any downloading issues.
I can confirm that I have had the same issue - i think we should move all trained weights to S3 and if some information can be shared on how the data was trained so that it can be reproduced (for example some estimates on time it took to train using what/how many GPUs - assuming of course trained using GPUs)
Thank you for your suggestion. I have added the estimation on the training time in #21 :
Lines 109 to 116 in 68310b3
Thank you for the details - can confirm that I am now able to download the pre-trained weights
--2024-03-21 10:58:58-- https://hkustconnect-my.sharepoint.com/:u:/g/personal/zgaoag_connect_ust_hk/EU6tZPSExoZIgoAwu5hTkjoBBJu1RBFepjFsP68Msb6JFA?e=31aIfa%5C
Resolving hkustconnect-my.sharepoint.com (hkustconnect-my.sharepoint.com)... 13.107.136.10, 13.107.138.10
Connecting to hkustconnect-my.sharepoint.com (hkustconnect-my.sharepoint.com)|13.107.136.10|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://hkustconnect-my.sharepoint.com/personal/zgaoag_connect_ust_hk/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fzgaoag%5Fconnect%5Fust%5Fhk%2FDocuments%2Fprojects%2FPreDiff%2Fpretrained%2Fmetrics%2Fpretrained%5Fi3d%5F600%2Ept&parent=%2Fpersonal%2Fzgaoag%5Fconnect%5Fust%5Fhk%2FDocuments%2Fprojects%2FPreDiff%2Fpretrained%2Fmetrics&ga=1 [following]
--2024-03-21 10:58:59-- https://hkustconnect-my.sharepoint.com/personal/zgaoag_connect_ust_hk/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fzgaoag_connect_ust_hk%2FDocuments%2Fprojects%2FPreDiff%2Fpretrained%2Fmetrics%2Fpretrained_i3d_600.pt&parent=%2Fpersonal%2Fzgaoag_connect_ust_hk%2FDocuments%2Fprojects%2FPreDiff%2Fpretrained%2Fmetrics&ga=1
Reusing existing connection to hkustconnect-my.sharepoint.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 269602 (263K) [text/html]
Saving to: 'EU6tZPSExoZIgoAwu5hTkjoBBJu1RBFepjFsP68Msb6JFA@e=31aIfa%5C.1'
EU6tZPSExoZIgoAwu5hTkjoBBJu1R 100%[=================================================>] 263.28K 455KB/s in 0.6s
2024-03-21 10:58:59 (455 KB/s) - 'EU6tZPSExoZIgoAwu5hTkjoBBJu1RBFepjFsP68Msb6JFA@e=31aIfa%5C.1' saved [269602/269602]
'download' 不是内部或外部命令，也不是可运行的程序
或批处理文件。 'download' is not an internal or external command, nor is it a runnable program
or batch files.How to solve this problem？thanks
--2024-03-21 10:58:58-- https://hkustconnect-my.sharepoint.com/:u:/g/personal/zgaoag_connect_ust_hk/EU6tZPSExoZIgoAwu5hTkjoBBJu1RBFepjFsP68Msb6JFA?e=31aIfa%5C Resolving hkustconnect-my.sharepoint.com (hkustconnect-my.sharepoint.com)... 13.107.136.10, 13.107.138.10 Connecting to hkustconnect-my.sharepoint.com (hkustconnect-my.sharepoint.com)|13.107.136.10|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://hkustconnect-my.sharepoint.com/personal/zgaoag_connect_ust_hk/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fzgaoag%5Fconnect%5Fust%5Fhk%2FDocuments%2Fprojects%2FPreDiff%2Fpretrained%2Fmetrics%2Fpretrained%5Fi3d%5F600%2Ept&parent=%2Fpersonal%2Fzgaoag%5Fconnect%5Fust%5Fhk%2FDocuments%2Fprojects%2FPreDiff%2Fpretrained%2Fmetrics&ga=1 [following] --2024-03-21 10:58:59-- https://hkustconnect-my.sharepoint.com/personal/zgaoag_connect_ust_hk/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fzgaoag_connect_ust_hk%2FDocuments%2Fprojects%2FPreDiff%2Fpretrained%2Fmetrics%2Fpretrained_i3d_600.pt&parent=%2Fpersonal%2Fzgaoag_connect_ust_hk%2FDocuments%2Fprojects%2FPreDiff%2Fpretrained%2Fmetrics&ga=1 Reusing existing connection to hkustconnect-my.sharepoint.com:443. HTTP request sent, awaiting response... 200 OK Length: 269602 (263K) [text/html] Saving to: 'EU6tZPSExoZIgoAwu5hTkjoBBJu1RBFepjFsP68Msb6JFA@e=31aIfa%5C.1'
EU6tZPSExoZIgoAwu5hTkjoBBJu1R 100%[=================================================>] 263.28K 455KB/s in 0.6s
2024-03-21 10:58:59 (455 KB/s) - 'EU6tZPSExoZIgoAwu5hTkjoBBJu1RBFepjFsP68Msb6JFA@e=31aIfa%5C.1' saved [269602/269602]
'download' 不是内部或外部命令，也不是可运行的程序 或批处理文件。 'download' is not an internal or external command, nor is it a runnable program or batch files.How to solve this problem？thanks
Thank you for reporting this issue. Could you please provide the command you used that caused the above error? I just tested the downloading script ./script/download_pretrained.py and it executed correctly.
The error you encountered might be related to the downloading of pretrained_i3d_600.pt used in FVD. You may want to notice that we also provide an alternative pretrained_i3d_400.pt. You can use pretrained_i3d_400.pt instead if you don't specifically require the weights pretrained_i3d_600.pt.
I followed the steps given by redme to execute the command, whether it was executing the python ./scripts/download_pretrained.py --model all command in the terminal or directly in pycharm, I ran into this issue