sync --delete hangs with many files
adutchak opened this issue · 0 comments
Hello,
we are observing that sometimes application hangs when we run sync --delete
. The amount of data we are having on both source and destination sides is ~17Tb, ~9M files.
It seems to be that this happens when the app attempts to delete < 20k files at a time, with smaller deletion scope it runs ok.
Unfortunately, in case of this problem, the app just freezes on rm s3:***
info message without errors or exits. Enabling debug log level makes no difference.
One of the theories can be that the number of arguments(...dstURLs
) for generateCommand()
method becomes too long: https://github.com/peak/s5cmd/blob/master/command/sync.go#L512 and it might have something to do with ARG_MAX
size limit on OS. In our case this limit is ~2M, and those 20k file names are slightly more than 2M.