peak/s5cmd

sync --delete hangs with many files

adutchak opened this issue · 0 comments

Hello,
we are observing that sometimes application hangs when we run sync --delete. The amount of data we are having on both source and destination sides is ~17Tb, ~9M files.
It seems to be that this happens when the app attempts to delete < 20k files at a time, with smaller deletion scope it runs ok.

Unfortunately, in case of this problem, the app just freezes on rm s3:*** info message without errors or exits. Enabling debug log level makes no difference.

One of the theories can be that the number of arguments(...dstURLs) for generateCommand() method becomes too long: https://github.com/peak/s5cmd/blob/master/command/sync.go#L512 and it might have something to do with ARG_MAX size limit on OS. In our case this limit is ~2M, and those 20k file names are slightly more than 2M.