Errors using rsync over mtpfs
Smoutdab opened this issue · 0 comments
Smoutdab commented
When backing up my Samsung Galaxy A52 using rsync over USB and mtpfs, I see several
errors "No data available (61)" and "failed verification -- update discarded"
I have found this previous similar report:
https://unix.stackexchange.com/questions/665525/android-file-copy-gives-errors-with-simple-mtpfs-and-rsync-on-fedora
but no solution
The errors are the same on two different computers with two different USB cables.
"cp -a" doesn't show the problem
The most interesting thing is the file sizes: they are all an integral multiple
of 512 bytes, less 12. During the "read" of the file, one fewer bytes is returned
than the "stat" size.
Snippet from an "strace" of "rsync"
171597 openat(AT_FDCWD, "toc.xhtml", O_RDONLY) = 3
171597 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=3572, ...}, AT_EMPTY_PATH) = 0
171597 read(3, "<?xml version=\"1.0\" encoding=\"UT"..., 3572) = 3571
171597 read(3, "", 1) = 0
171597 write(2, "rsync: [sender] read errors mapp"..., 212) = 212
171597 close(3) = 0
and compare with a file that isn't erroring.
note that the first read above returns one less byte than the size returned by "stat"
172205 openat(AT_FDCWD, "ad-card.xhtml", O_RDONLY) = 3
172205 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=3759, ...}, AT_EMPTY_PATH) = 0
172205 read(3, "<?xml version=\"1.0\" encoding=\"UT"..., 3759) = 3759
172205 close(3) = 0
When repeatedly running md5sum, on one of the files, the output is always the same.
The systems are up-to-date Gentoo, running the latest stable kernel 5.16.5. I
have repeated the test with the latest rsync git checkout from
https://github.com/WayneD/rsync.git, v3.2.4pre3-3-g81f71f6f with the same
results.
[I] sys-fs/simple-mtpfs
Available versions: 0.4.0
Installed versions: 0.4.0(10:25:28 10/10/20)
Homepage: https://github.com/phatina/simple-mtpfs
Description: Simple MTP fuse filesystem driver
$ rsync --version
rsync version 3.2.3 protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
batchfiles, inplace, append, no ACLs, xattrs, optional protect-args,
iconv, symtimes, prealloc, stop-at, no crtimes
Optimizations:
SIMD, asm, openssl-crypto
Checksum list:
md5 md4 none
Compress list:
zlibx zlib none
# /work/src.git/rsync/rsync --delete -a /mnt/phone/ .
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.nianticproject.ingress/cache/UnityShaderCache/08ec8082de60d312a88711c39719aacf": No data available (61)
ERROR: Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml failed verification -- update discarded.
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/11/oruli_mMybATsowWgQLNaFc3poA.cnt": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/20/B1sh1wVqAJvyMvYizfCtuiDhOow.cnt": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/38/HNkayh6xlr6SQbyVU7aAlNuTkeM.cnt": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/75/Rd6X195JFD2oiU5houPfvm5oe5M.cnt": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/DCIM/Camera/20211016_092257.jpg": No data available (61)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1326) [sender=v3.2.4pre3-3-g81f71f6f]
different computer, different USB cable, not root
$ rsync -a /mnt/phone/ .
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.nianticproject.ingress/cache/UnityShaderCache/08ec8082de60d312a88711c39719aacf": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/11/oruli_mMybATsowWgQLNaFc3poA.cnt": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/20/B1sh1wVqAJvyMvYizfCtuiDhOow.cnt": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/38/HNkayh6xlr6SQbyVU7aAlNuTkeM.cnt": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/75/Rd6X195JFD2oiU5houPfvm5oe5M.cnt": No data available (61)
rsync: [sender] read errors mapping "/mnt/phone/DCIM/Camera/20211016_092257.jpg": No data available (61)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]
List file sizes
-rw-r--r-- 1 root root 3572 Dec 3 12:26 /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
-rw-r--r-- 1 root root 3572 Jan 29 10:05 /mnt/phone/Android/data/com.nianticproject.ingress/cache/UnityShaderCache/08ec8082de60d312a88711c39719aacf
-rw-r--r-- 1 root root 6132 Sep 1 17:58 /mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/11/oruli_mMybATsowWgQLNaFc3poA.cnt
-rw-r--r-- 1 root root 6132 Sep 1 17:58 /mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/20/B1sh1wVqAJvyMvYizfCtuiDhOow.cnt
-rw-r--r-- 1 root root 6132 Jan 15 11:17 /mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/38/HNkayh6xlr6SQbyVU7aAlNuTkeM.cnt
-rw-r--r-- 1 root root 84980 Sep 7 19:02 /mnt/phone/Android/data/com.twitter.android/cache/image_cache/v2.ols100.1/75/Rd6X195JFD2oiU5houPfvm5oe5M.cnt
-rw-r--r-- 1 root root 2502644 Oct 16 09:22 /mnt/phone/DCIM/Camera/20211016_092257.jpg
The file sizes are all an integral multiple of 512 bytes, less 12
; 512 * 7 - 12
3572
; 512 * 12 - 12
6132
; 512 * 166 - 12
84980
; 512 * 4888 - 12
2502644
repeated reads of a file return the same data
# md5sum /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
36f02268095daa7293d5994b108ae6ce /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
# md5sum /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
36f02268095daa7293d5994b108ae6ce /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
# md5sum /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
36f02268095daa7293d5994b108ae6ce /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
# md5sum /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
36f02268095daa7293d5994b108ae6ce /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
copy just a single file several times, then run "rsync" with -vvv
# rsync -a /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml .
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml": No data available (61)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]
# rsync -a /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml .
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml": No data available (61)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]
# rsync -vvv -a /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml .
sending incremental file list
[sender] make_file(toc.xhtml,*,0)
send_file_list done
send_files starting
server_recv(2) starting pid=171400
recv_file_name(toc.xhtml)
received 1 names
recv_file_list done
get_local_name count=1 .
generator starting pid=171400
delta-transmission disabled for local transfer or --whole-file
recv_generator(toc.xhtml,1)
send_files(1, /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml)
send_files mapped /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml of size 3572
calling match_sums /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
toc.xhtml
sending file_sum
false_alarms=0 hash_hits=0 matches=0
rsync: [sender] read errors mapping "/mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml": No data available (61)
sender finished /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
generate_files phase=1
recv_files(1) starting
WARNING: toc.xhtml failed verification -- update discarded (will try again).
recv_files(toc.xhtml)
recv mapped toc.xhtml of size 3571
got file_sum
recv_generator(toc.xhtml,1)
send_files(1, /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml)
send_files mapped /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml of size 3571
calling match_sums /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
toc.xhtml
sending file_sum
false_alarms=0 hash_hits=0 matches=0
sender finished /mnt/phone/Android/data/com.kobobooks.android/files/c15309fdbf7616dba9c74d4b1eb5f327/epubs/a730fadf-913a-4b0d-904b-02b7065236a1-2/ops/xhtml/toc.xhtml
recv_files(toc.xhtml)
recv mapped toc.xhtml of size 3571
got file_sum
set modtime, atime of .toc.xhtml.KUMetq to (1638534375) 2021/12/03 12:26:15, (1643794931) 2022/02/02 09:42:11
renaming .toc.xhtml.KUMetq to toc.xhtml
send_files phase=1
recv_files phase=1
generate_files phase=2
send_files phase=2
send files finished
total: matches=0 hash_hits=0 false_alarms=0 data=7143
recv_files phase=2
recv_files finished
generate_files phase=3
generate_files finished
sent 7,301 bytes received 794 bytes 5,396.67 bytes/sec
total size is 3,572 speedup is 0.44
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]
[sender] _exit_cleanup(code=23, file=main.c, line=1330): about to call exit(23)
#