rsync just hangs

[feliperaul]

What went wrong?

For the last 5 days, I have been trying to set up rsync backup to sync around 5 million files (280GB) between two VPSs on DigitalOcean.

The behavior is very inconsistent. Most of the times, running the command from the command line (backup perform -t shared_folders), it just sits forever on these lines:

[2018/06/08 13:46:04][info] Syncer::RSync::Push Started... [2018/06/08 13:46:04][info] Running system utility 'rsync'...

And it just stays there. And I mean 10+ hours with no progress whatsoever.

Using htop, I can see that the destination server initially spins up a rsync --server, but later on (after 30 min or 1 hour) it finishes and the command is still running on the source server, and the backup gem just sits there forever.

After using htop on the source server, I manually started rsync on the command line using the exact same command, which was something like this:

rsync --archive --delete --progress --exclude=tmp/ --exclude=cache/ --whole-file --no-inc-recursive -e /usr/bin/ssh -p 22 /home/deploy/www/shared/public xx.xx.xx.xx:/mnt/volume_nyc3_01/backup

And even tough it's the exact same command it was running via the gem, this time it works flawlessly, specially if I use --progress (I can then see the files being uploaded).

What steps did you follow?

1. gem install backup --version 4.3.0
2. backup check
3. backup perform [OPTIONS]
4. Check the file that is produced by backup

How is your copy of backup configured?

config.rb file is only comments, like the default one generated by the gem.

This is the model:

Model.new(:shared_folders, 'Backup ') do

  ##
  # RSync::Push [Syncer]
  #
  # The default `mode` is :ssh, which does not require the use
  # of an rsync daemon on the remote. If you wish to connect
  # directly to an rsync daemon, or via SSH using daemon features,
  # :rsync_daemon and :ssh_daemon modes are also available.
  #
  sync_with RSync::Push do |rsync|
    rsync.mode     = :ssh
    rsync.host     = "xx.xx.xx.xx"
    rsync.path     = "/mnt/volume_nyc3_01/backup"
    rsync.mirror   = true
    # Disabled compress because destination server is a VPS with only 1GB of ram and it was killing rsync because of out of memory. 
    # rsync.compress = true

    rsync.additional_rsync_options = ['--whole-file','--no-inc-recursive']

    rsync.directories do |directory|
      directory.add "/home/deploy/www/shared/public"

      # Exclude files/folders.
      # Each pattern will be passed to rsync's `--exclude` option.
      #
      # Note: rsync is run using the `--archive` option,
      #       so be sure to read the `FILTER RULES` in `man rsync`.
      # directory.exclude '*~'
      directory.exclude 'tmp/'
      directory.exclude 'cache/'
    end
  end

  ##
  # Slack [Notifier]
  #
  notify_by Slack do |slack|
    slack.on_success = true
    slack.on_warning = true
    slack.on_failure = true
    slack.webhook_url = 'https://hooks.slack.com/services.xxxx'
    slack.username = 'backup_gem'
  end

  Logger.configure do
    # Console options:
    console.quiet = false

    # Logfile options:
    logfile.enabled   = true
    logfile.log_path  = 'backup_gem_log'
    logfile.max_bytes = 5_000_000
  end

end

Tell us about the computer that runs the backup gem

• Operating system: Ubuntu 16.04.3 LTS
• Ruby version: 2.3.6
• Tools or services used by backup: MySQL 5.7.9, GnuPG 1.4.16

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.