steilerDev/icloud-photos-sync

Unable to add Asset with filesize > 2 GB

Closed this issue · 6 comments

steilerDev commented 
  last run results

0% | Elapsed: 3m41s | 15/60811 assets downloaded
0% | Elapsed: 58m29s | 429/60796 assets downloaded
0% | Elapsed: 41m56s | 392/60367 assets downloaded

Is it possible to handle file larger than 2GiB ?

[2023-01-09T14:00:51.380Z] INFO CLI-Interface: SyncWarning (WARN): Error while writing state caused by File size (2387209258) is greater than 2 GiB
[2023-01-09T14:00:51.380Z] WARN Error-Handler: SyncWarning (WARN): Error while writing state caused by File size (2387209258) is greater than 2 GiB
[2023-01-09T14:00:52.430Z] DEBUG Photos-Library: Writing asset ARCv8Z3JSEgBaRFgOJghRDIW9FKX
[2023-01-09T14:00:55.101Z] INFO CLI-Interface: undefined
[2023-01-09T14:00:55.390Z] INFO CLI-Interface: Experienced fatal error at 1/9/2023, 3:00:55 PM: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (2387209258) is greater than 2 GiB (Error Code: 989aac0b-2a56-4aea-acc3-0fd8bb922c31)
[2023-01-09T14:00:55.390Z] INFO CLI-Interface: undefined
[2023-01-09T14:00:55.390Z] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (2387209258) is greater than 2 GiB (Error Code: 989aac0b-2a56-4aea-acc3-0fd8bb922c31)

Thanks !

Originally posted by @Laffreux-dev in #154 (comment)

Laffreux-dev commented

here is some new error reporting for this one

[2023-01-11T11:41:19.751Z] DEBUG i-Cloud-Photos: Starting download of asset AearpB9K3qKmwQ67X5NuvL7s7lCS
[2023-01-11T11:41:19.852Z] INFO CLI-Interface:  [0m [31mundefined [39m [0m
[2023-01-11T11:41:20.005Z] INFO CLI-Interface:  [0m [31mExperienced fatal error at 1/11/2023, 12:41:19 PM: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (4132634065) is greater than 2 GiB (Error Code: 2aa0071d-936b-4df8-b239-5d2ff2a26ab0) [39m [0m
[2023-01-11T11:41:20.005Z] INFO CLI-Interface:  [0m [31mundefined [39m [0m
[2023-01-11T11:41:20.005Z] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (4132634065) is greater than 2 GiB (Error Code: 2aa0071d-936b-4df8-b239-5d2ff2a26ab0)

[2023-01-11T13:51:52.501Z] DEBUG i-Cloud-Photos: Starting download of asset AcIHZZP8R/gj0wUEkjZbCYQMqptn
[2023-01-11T13:51:52.599Z] INFO CLI-Interface:  [0m [31mundefined [39m [0m
[2023-01-11T13:51:52.886Z] INFO CLI-Interface:  [0m [31mExperienced fatal error at 1/11/2023, 2:51:52 PM: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (3780015937) is greater than 2 GiB (Error Code: 06d8091b-8e25-4c50-8e6f-a03d3cab0302) [39m [0m
[2023-01-11T13:51:52.887Z] INFO CLI-Interface:  [0m [31mundefined [39m [0m
[2023-01-11T13:51:52.887Z] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (3780015937) is greater than 2 GiB (Error Code: 06d8091b-8e25-4c50-8e6f-a03d3cab0302)

[2023-01-11T14:27:43.469Z] DEBUG i-Cloud-Photos: Starting download of asset Af9vgaL9KahY7FA4sV9geHRza4qM
[2023-01-11T14:27:43.504Z] INFO CLI-Interface:  [0m [33mSyncWarning (WARN): Error while writing state caused by File size (2234834664) is greater than 2 GiB [39m [0m
[2023-01-11T14:27:43.504Z] WARN Error-Handler: SyncWarning (WARN): Error while writing state caused by File size (2234834664) is greater than 2 GiB
[2023-01-11T14:27:45.410Z] DEBUG Photos-Library: Writing asset Af9vgaL9KahY7FA4sV9geHRza4qM
[2023-01-11T14:27:46.541Z] INFO CLI-Interface:  [0m [31mundefined [39m [0m
[2023-01-11T14:27:46.784Z] INFO CLI-Interface:  [0m [31mExperienced fatal error at 1/11/2023, 3:27:46 PM: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (2234834664) is greater than 2 GiB (Error Code: 5765bef1-5668-4bfa-a0b7-2fbb01f96c4b) [39m [0m
[2023-01-11T14:27:46.785Z] INFO CLI-Interface:  [0m [31mundefined [39m [0m
[2023-01-11T14:27:46.909Z] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (2234834664) is greater than 2 GiB (Error Code: 5765bef1-5668-4bfa-a0b7-2fbb01f96c4b)
steilerDev commented

During verification, I'm loading the whole file into a Buffer - it seems there is a 2GB limit on that buffer: https://stackoverflow.com/questions/70092824/rangeerror-err-fs-file-too-large-file-size-3472064213-is-greater-than-2-gb

Probably need to redo some logic to handle this more efficiently using streams....

Laffreux-dev commented

the good news is that there is a solution :)

steilerDev commented

Looking at the error - I hope this is the root cause :D

Since I don't verify the checksum (because I don't understand how Apple calculates the checksum), this might actually be a very easy fix :)

Laffreux-dev commented

The synchronization is soon complete...

$ ls -l _All-Photos/ | wc -l
50429

$ du -sh _All-Photos/
1.1T    _All-Photos/

and the files of more than 2G are present!

$ find . -size +2G -exec ls -lhs {} \+ | sort -n
2.1G -rw-r--r-- 1 admin users 2.1G Nov 13  2020 ./AafQbJscV3jzDDRt8uG1F5HwqNRi.mov
2.1G -rw-r--r-- 1 admin users 2.1G Sep 13  2020 ./AUc5p31XoZU_70pbQu9qrdkaz3Zu.mp4
2.1G -rw-r--r-- 1 admin users 2.1G Sep 18  2020 ./AQRGVs3gY5wZATXhcSzvT-iroe7y.mov
2.2G -rw-r--r-- 1 admin users 2.2G Jun 20  2021 ./AfiKitbnvfhvBy4mC20blTEkgJtD.mov
2.2G -rw-r--r-- 1 admin users 2.2G Mar 13  2021 ./AQxS5_Q__PoQe4HwfOyl6ZLoqrHz.mov
2.2G -rw-r--r-- 1 admin users 2.2G Oct 27  2020 ./AeCcl6LGpxzN_YxL9UUAroikk89W.mov
2.3G -rw-r--r-- 1 admin users 2.3G Aug 18 05:00 ./Acy00AH9KsoFD5PnCeWAonWlntZa.mp4
2.3G -rw-r--r-- 1 admin users 2.3G Sep 11  2020 ./AacVlGkK_rHXZcLYB1CohQhyewac.mov
2.4G -rw-r--r-- 1 admin users 2.4G May 30  2022 ./AT4X03ap5IDjOnsW2fiCeeuwRjfa.mov
2.4G -rw-r--r-- 1 admin users 2.4G Oct 28  2020 ./AYmZduvy5nLcs__WG6DZyeoIqGbA.mov
2.4G -rw-r--r-- 1 admin users 2.4G Oct  3  2020 ./AVWPDY8ZMWfUwfBgW-f8u3Hz8X84.mp4
2.4G -rw-r--r-- 1 admin users 2.4G Sep 23  2020 ./ARb5K1AEHl9VuzgkIJnsUmkGuKOb.mov
2.8G -rw-r--r-- 1 admin users 2.8G Oct  3  2020 ./ATsa_QRUDfoal3BGeZ-X4H_zNM4K.mp4
3.4G -rw-r--r-- 1 admin users 3.4G Jun 25  2022 ./AaEqpF5FY4Ou1Dm8Q594s_3tWqqE.mov
3.5G -rw-r--r-- 1 admin users 3.5G Jun  1  2022 ./AQ24wImRIdQEjdA9G4_kXPCbNxon.mp4
3.6G -rw-r--r-- 1 admin users 3.6G Dec  1  2020 ./AQY3sUYpD-f17Y3-Ml55Vcei_3be.mov
3.8G -rw-r--r-- 1 admin users 3.8G Sep 13  2020 ./AWMBo46dTvAdF4KiSler0MidNhIG.mp4
3.9G -rw-r--r-- 1 admin users 3.9G Jan 12  2021 ./AaMmFa0HODbWKU9Acw_5kZUFesWO.mov
4.7G -rw-r--r-- 1 admin users 4.7G Sep 26  2020 ./ARrQgj3uKktUcOMvRGL90oN8ZVCq.mp4
4.8G -rw-r--r-- 1 admin users 4.8G Nov  5  2021 ./AYfwm0Sa0nPaydFoGve3lO-RC4gr.mp4
5.1G -rw-r--r-- 1 admin users 5.1G Aug 15 06:45 ./AaXZkkUTcq6HCZ3PZT-KsZc8oyoS.mp4
5.1G -rw-r--r-- 1 admin users 5.1G Oct  3  2020 ./ATVOPjaeL32jVqEVc8EotSknYYUR.mp4
6.9G -rw-r--r-- 1 admin users 6.9G Sep 13  2020 ./AWjRjos8PzGNDMwQbZo4qbJigeq1.mp4
 10G -rw-r--r-- 1 admin users  10G Jan  8 00:55 ./AYiekI0FG-xhMznta-PqIQaUCH53.mov
 13G -rw-r--r-- 1 admin users  13G Mar 15  2022 ./ARbBPzEWaWQfmOUWNfREebDU7sqs.mp4
steilerDev commented

Closed with f66ad1a