Beaker puppet_module_install and scp_to is slow in larger environments
patrobinson opened this issue · 1 comments
This was originally raised in #143, but not implemented as the original speed issue was resolved by forcing a chunk_size. My problem is I have a large number of modules I need to copy across for testing (40+) and using puppet_module_install_on takes up-to 10 minutes!
There are two problems here:
- puppet_module_install_on uses the ignore method which actually slows down copying, because each individual file is scped one by one! The resulting spawning and reaping of child processes takes a long time.
- scp blind copies and uses encryption
I've implemented using rsync instead of scp and will submit a PR shortly, here's a comparison of time it takes to copy across modules in my environment (1GB connection and sub 1ms latency):
- puppet_module_install_on with scp => 463 seconds
- puppet_module_install_on with rsync (sync nothing) => 332 seconds
- scp_to => 179 seconds
- rsync_to (sync nothing) => 21 seconds
- rsync_to (sync all) => 19 seconds
Rsync speed starting with no files and all files are comparable, the speed increase seems to be lack of encryption/decryption. This is probably because the modules consist of lots of small files (around 4,000 in total).
Ultimately puppet_module_install_on with rsync should use the '--exclude' argument to ignore certain files, but it's unlikely I'll include this in my PR.