How to make a copy of more than 1,000,000 messages
Closed this issue · 6 comments
How to make a copy of more than 1,000,000 messages in one file? What would not occupy an inode.
Current implementation indeed saves one message per file (or two files, if you use the -full command-line switch).
It is possible to modify the code to put a bunch of messages together, maybe in a tar/zip file? I'll be glad to hear ideas (and, if possible — a pull request with suggested implementation).
Note that I never tested the system with so many messages (I did test with up to about 10K messages).
To save immediately to the archive, you still need to save the file somewhere first. Those. He will be saved in memory or in a temporary catholic.
It's better just to the file, so at least predictable behavior will b
We can write files to an archive in batches - say write 1000 files, archive them into one tar file, delete them (leave just the tar file), and then continue to the next 1000 messages.
This sounds like a complex addition to this tool, I hope to be able to do it from an outside tool, say in a pipe. This is the reason for printing the file names to the standard output — to allow another tool to process them as they are written.
rabbitmq-dump-queue -full -output-dir=/some/dir | tar -czf /some/archive.tgz --remove-files -T -
This will add each file to a gzip tarball as they're being written to disk. After the message is added to the archive, tar automatically deletes it. The filesystem will act as a buffer, containing only a limited number of files based on how fast tar can archive them. You may want to disable gzipping to get faster perf if rabbitmq-dump-queue is too fast.
(inactive issue; closing.)