Using rsync on large files failed
Closed this issue · 12 comments
Hy,
First Thanks for your great app. It works like a charm.
Yet, some OpenStack consideration haven't been integrated. For example, it's impossible to manage file which size is upper to 5Go.
Is it possible to handle this with a kind of automatic splitting?
regards
Hello flotho,
I haven't test yet but maybe there is an option with fuse to do so
if you type
#hubicfuse -h
you will see an option
==> -o [no_]splice_write use splice to write to the fuse device
maybe if you mount your hubic storage unit with this it can resolve the issue.
I'll tonight I'll tell you if it works !
Yes, let me know if this works or not.
Ok, it seems it doesn't. But I have some clues.
On my server, the rsync command failed because all storage of /tmp (as tmpfs) is used.
This is not a bug of rsync because "killall -s SIGKILL rsync" let the usage of /tmp at 100%
But a "killall -s SIGKILL hubicfuse" clear the memory.
We have to find a way to make caching process of large file use maybe the hard drive insteadof memory.
FYI I'll try to mount /tmp on my hardrive (with much more space than my memory) and see how it run
I come back to tell you what I've done
-> Mount /tmp on hard drive make rsync transfert complete, but it freeze at the end of transfert.
I think it's because it take time to transfert a large amount of data on hubic server for now, rsync end transfert and freeze for arround 5 min with a file of 4GB.
Maybe it will end shortly, I don't know.
I keep you in touch
I may be an issue with the renaming. I was unable to cp or rsync any files on my hubicfuse mountpoint due to that renaming issue. Maybe you'll confirm that ?
(cp/rsync process went in status D (unkillable), I had to kill hubicFuse.)
IIUC renaming is not supported by hubic, so with rsync you should copy in-place to avoid the final renaming which will be done via a copy+remove. I don't remember the correct rsync option, but I know there is something supported by rsync to copy in-place.
With
# hubicfuse /hubic -o noauto_cache,sync_read,allow_other,no_splice_write,no_splice_move
# rsync -vPW --inplace /storage/4GB.file /hubic/default
the operation fails. The file is normaly copied, stay in the folder for one or two minute then disapears and the rsync doesn't quit. If we send a CTRL+C at this point, a rsync process stay with a status D. The only solution to get read of it is to run umount -l /hubic then kill hubicfuse.
No problem with multiple medium files using the same rsync command.
EDIT : The 4GB.file is now visible on my hubic web interface, but not in my hubic mountpoint... But the rsync process ended. All is matter of time !
EDIT2 : Now visible in the mountpoint. Thanks !
Hy,
Is it normal that I have this message while trying your latest command :
fuse: unknown option `no_splice_write'
@flotho, don't know it is working for me. maybe an old fuse version? Anyway at this point I'll close this issue, nothing more to do.
BTW lechatleon, as Hubic does not support setting timestamp how does your rsync command is supposed to work? Some time ago I had to use and option to teach rsync to use the file size and not the stamp (--size-only IIRC).
Actually, I don't use rsync to "sync" directories. I used it only to copy big files to the mountpoint because cp failed. Sorry, cannot help ! (But I'm testing BittorentSync on a hubic mountpoint, and it seems to work.)
Ok, understood.