Allow for output to stdout?

[billcrook]

Ask a question
The -o output file parameter is a required option in rct. It would be useful to allow an option to output to standard out. This would allow the tool to be used in typical unix piping on the command line. For instance, I could then compress the output such as:
rct -f jsonl -s backup.rdb -k 'somekey:.*' | bzip2 > output.jsonl.bz2

And of course this would open up all the unix tools such as head, sort, awk, sed, etc... What do you think?

[leonchen83]

@billcrook
if we use stdout. there have some cases to consider. because the source not only rdb file, but also an redis URI like redis://127.0.0.1:6379.
there are 2 cases to consider

1. the source is rdb file like rct -f jsonl -s ./backup.rdb in this case we can ignore -o option and no problem.
2. the source is redis URI like rct -f jsonl -s redis://127.0.0.1:6379, in this case we have another 3 cases to consider.

• 2.1 if network very well and no disconnected with redis://127.0.0.1:6379, we can also ignore -o option and no problem.

• 2.2. if disconnected with redis://127.0.0.1:6379 a little while and after reconnect triggered redis psync process,we can also ignore -o option and no problem.

• 2.3 if disconnected with redis://127.0.0.1:6379 a long while and after reconnect triggered redis full sync process, in this case we ignore -o option and wirte jsonl to stdout , then we may get an invalid and duplicate json. the stdout may following

{"key":"k1", "value": "v1"}
{"key":"k2", "value": "v2"}
{"key":"k3", // disconncet and triggered full sync in here
{"key":"k1", "value": "v1"} // after reconnectted re-export json form "k1"
{"key":"k2", "value": "v2"}
{"key":"k3", "value": "v3"}

in case 2.3 ,if we add -o option and output json to file , we can recreate the file and refill the data when triggered redis full sync, so finally we can get an valid json file.

above is why we use -o option instead of stdout

[billcrook]

I see, makes sense. I can get by without this feature, it was just an idea I had that might be useful to others. Feel free to close this issue if you deem it unnecessary at this time.

FYI, I'm about to start using rct in production, I'll let you know how it goes :)

[leonchen83]

I'm very glad to see this tool used in your production env. if any question or bug report , feel free to ask. and thank you for your support to help us to improve this tool.