Can cdc_rsync be used as a local Windows rsync tool?
prarsena opened this issue · 10 comments
I saw the following in another issue thread :
adding support for Windows to Windows cdc_rsync (including local syncs, i.e. not over network), see
main...sync_windows
Between the speed improvements you highlight from this tool and the fact that Windows doesn't have native support for rsync, I would love to use this tool to make backups for local drives.
But I wasn't able to understand the status of that work. Is the current status of the sync_windows (or win_support) branch at the point to support local Win rsync?
If the support comes, would it work like similarly to the following format? cdc_rsync E:\ F:\MyDrive\ -rv
The status is, the code is in review. I expect it to land next week. My reviewer is currently on vacation.
If the support comes, would it work like similarly to the following format?
cdc_rsync E:\ F:\MyDrive\ -rv
Yes, exactly.
I somewhat doubt, though, that cdc_rsync will perform any better than standard Windows tools like xcopy and robocopy in this case. Note that both tools can skip unchanged files by modification date. cdc_rsync only gives a benefit if the bandwidth between the local and the target system is lower than the disk IO speeds on the two systems. In case of a local copy the bandwidth is very high (basically, memory speed), so I don't expect any gain.
To illustrate what I mean, imagine there's a 1 GB file F:\MyDrive\foo.dat and the same file with a single byte inserted in the beginning at E:\foo.dat. In order to sync this file, cdc_rsync creates a temp file F:\MyDrive\foo.dat__cdc_rsync_temp__, copies a chunk of E:\foo.dat to it and appends the rest from F:\MyDrive\foo.dat. Finally, it replaces F:\MyDrive\foo.dat by F:\MyDrive\foo.dat__cdc_rsync_temp__. I highly doubt this is any faster than a plain copy.
I would suggest you try it out once the changes are in, and compare. I'm very curious what the results are.
Thank you for your detailed response, and explaining that the benefit of cdc_rsync comes from the network improvements. I appreciate it. I will keep an eye on this project and test that out.
When would a test version for windows->windows remote machine be available for testing? I would be willing to give it a try
I hacked together a release build that contains the changes. I tested it to a local Docker container and it worked. Currently, I only have one Windows machine, so I can't test true remote copies, but I've applied for one.
https://github.com/google/cdc-file-transfer/suites/10439823585/artifacts/516488099
I'm super curious whether it works for you. Please let me know what issues you run into. Deploying the binaries properly is a surprisingly hard problem.
Note that we switched from scp to sftp, so you need to update your CDC_SCP_COMMAND to CDC_SFTP_COMMAND (but most likely you can just replace scp.exe by sftp.exe). Please see the readme contained in the zip.
How do I get the copy to go on windows? Any hints on what command line to use would be appreciated
Sorry for the delay, we had some distractions here lately. For local Windows syncing, i.e. copying files locally, you would just remove the user@host, e.g.
cdc_rsync C:\path\to\source\* C:\path\to\dest -rv
For remote Windows-to-Windows copies just add user@host:
cdc_rsync C:\path\to\source\* user@host:C:\path\to\dest -rv
The code will assume it's copying to Windows because the path starts with C: and deploy the right server file, i.e. cdc_rsync_server.exe.
No worries on the delay, totally understand. I tried it and this is what I get:
Error: CreateProcess() failed: The system cannot find the file specified.; Failed to start cdc_rsync_server process; Failed to start server
I have the remove drive mapped so security wise the source server is logged into the destination server.
cdc-file-transfer-binaries-v1.0.0-x64
My code:
cdc_rsync D:\tmp\111\* D:\tmp\ggg -vr
Error: No remote host specified in destination 'D:\tmp\ggg'. Expected [user@]host:destination.
cdc-file-transfer-binaries-main-x64
My code:
cdc_rsync D:\tmp\111\* D:\tmp\ggg -vr
Port 51048: Server is listening
�EOF detected; Failed to receive packet of size 4; Failed to receive packet; Message pump thread is down; Failed to dequeue packet; Failed to receive SetOptionsRequest; Failed to receive options�2113 file(s) and 1949 folder(s) found
2532 file(s) and 2323 folder(s) found
Error: No remote host specified in destination 'D:\tmp\ggg'.
I have the same issue. Any workarounds?
Error: No remote host specified in destination 'D:\tmp\ggg'.
I have the same issue. Any workarounds?
Same for me. Same error using windows to windows local sync:
Port 51890: Server is listening
EOF detected; Failed to receive packet of size 4; Failed to receive packet; Message pump thread is down; Failed to dequeue packet; Failed to receive SetOptionsRequest; Failed to receive options582 file(s) and 21 folder(s) found