This repo contains a few utility scripts to manage Qubes OS backups using Borg Backup into Rsync.net.
Backup of data is essential in any workflow. Two top concerns of any backup strategy are the security and the cost of maintaining backups.
Borg Backup offers a reasonably secure, open source backup software solution.
Rsync.net offers special pricing discounts for borg users.
Qubes OS offsers separation of concerns and security domains on your desktop computer.
Let's combine them FTW!
This repo contains template files and scripts to setup your own backup workflow. I am using it for my own workflow, which may or may not be appropriate for your own needs.
Please review all files and scripts and adapt this flow to your preferred backup strategy.
I make no guarantees of backup performance, reliability and integrity if you follow this template.
USE AT YOUR OWN RISK
This assumes the following:
- You have setup an account at rsync.net (or any backup provider that allows ssh+borg access);
- You have installed borg and borgmatic;
- If using Qubes, you have enabled the
crondservice on the AppVMs you'll be making backup from.
- On your personal
~/.ssh/configfile, create an entry named "rsyncnet" (see item 3 below) to serve as alias to connect to rsync.net for administration purposes; - Create the root repository dir in rsync.net (
ssh rsyncnet mkdir -p repos); - Copy (or link) the public part of the ssh key you'll use to connect to rsync.net to the file
admin-key.openssh.pub; - Create the file
template/bakserverto serve as template for the AppVMs to connect to the backup server (see item 3 below - usebakserver.templateas an example).
For each VM you'll want to backup data from, create a new key:
# Replace REPO-NAME. The full path on rsyncnet will be repos/REPO-NAME
$ ./new-key.sh REPO-NAME
# Example
$ ./new-key.sh vm-personalDuring key creation you'll need to enter the passphrase to the repo. I suggest generating and storing the passphrase on KeePassX on your VaultVM with high entropy parameters.
The script will also display your passphrase, so only do this when you can be certain nobody is watching your screen.
After generating the key for the new repo, run assemble-authorized-keys.sh to generate an authorized_keys file to be installed on rsync.net. On the first few runs, manually check if the file is correct (specially the admin key section - if that gets messed up you may get locked out of the remote server).
Upload the file to rsync.net with upload-authorized-keys.sh.
The host-keys/REPO-NAME dir should be copied and installed on the target AppVM (see item 4 below). Also, modify the file borgmatic.cfg to your needs (at a minimum you'll need to modify the source_directories section).
I usually config ssh connection aliases on ~/.ssh/config.d and then assemble the file ~/.ssh/config by concatenating everything in this dir.
The alias to connect to rsync.net for administration purposes (i.e. anything except backing up from a VM) should look like:
# File ~/.ssh/config.d/rsyncnet
Host rsyncnet
HostName <SERVER SENT BY>.rsync.net
User <USERNAME SENT BY RSYNC.NET>
IdentityFile ~/.ssh/rsyncnet-admin-key.openssh
The template file template/bakserver should have the HostName and User fields filled with the same information (you can use the bakserver.sample file as an example).
To generate the actual ssh config file run:
$ cat ~/.ssh/config.d/* > ~/.ssh/configIf you don't use this method for configuring aliases, you'll need to manually edit your ~/.ssh/config file on your admin VM and on all the AppVMs used to backup data with the appropriate aliases.
Another alternative is to start using this method by copying your existing ~/.ssh/config into the ~/.ssh/config.d dir and taking from there.
The dir generated by the new-key.sh script should have everything needed to configure an AppVM to backup to rsync.net. The files are:
- bak.crontab: Crontab entry for automated backups;
- bakserver: The ssh config the VM will use to connect to the backup server (see item 3 above);
- borgbak: A simple script copied to ~/bin to allow faster usage of borg on rsync.net (seem item 5 below);
- borgmatic.cfg: The borgmatic config file;
- borgpass: The password for the backup repo in plain text (see security considerations below);
- borgrepo: The name of the remote backup repo;
- install.sh: Install script (read it to see what it does and if it's appropriate to your use case);
- REPO-NAME.openssh: The private key to backup;
- REPO-NAME.openssh.pub: The public key to install on the backup server.
This dir is meant to be installed on the target VM as ~/.ssh/backups. Changing this location means changing the location of the key on the bakserver alias and the borgbak utility.
The command line to execute any borg action on rsync.net looks like the following:
$ borg $ACTION --remote-path=borg1 bakserver:repos/$REPO(:$ARCHIVE) $REST
Password: (type password)It can get tiring typing that all time, so the borgbak script simplifies it by extracting the repo from the file ~/.ssh/backups/borgrepo, the passphrase from the file ~/.ssh/backups/borgpass and the action, archive and rest of the command from the next arguments.
The most commonly used borg commands get simplified to:
$ borgbak list
$ borgbak list personal-2017-08-03T12:56:09.339289
$ borgbak info personal-2017-08-03T12:56:09.339289
$ borgbak extract personal-2017-08-03T12:56:09.339289 home/user/test-fileComplex borg actions may still be issued by using the original borg command and passing the full arguments.
A few security considerations for this script. They do not represent all potential issues on a given borg installation, only the most relevant ones for this workflow.
Some of the decisions made by this flow can be modified to suit your desired security profile.
This script currently uses the repokey encryption mode of borg, which means the encryption key file (the actual key bits) is stored on the server. The key file is protected by the key passphrase, which never leaves the client.
If the backup server is compromised, the key file can be extracted. However, that alone should not give access to the actual key bits, as long as the passphrase is strong enough (or if there is a bug or vulnerability on borg's encryption algorithm).
This workflow could be modified to use keyfile mode by generating the key locally (in the new-key.sh script). Using this mode, you must also backup the key file on a different medium or risk losing access to your backups (see remarks about backup accessibility).
This script assembles an authorized_keys file to be installed on the backup server that restricts the actions of the individual client backup keys to the borg command on a given repo dir.
This is the recommended procedure when multiple sources will write to the same backup server, as it allows each source to have an independent ssh key and borg passphrase, and compromise of one source does not automatically compromise all.
The exception to this is the maintenance key used to connect to the server for full access. Obviously, this key is specially important, should be stored encrypted and only used from a trusted machine.
The current ssh line assembled for each client is the following (minus the newlines):
command="
cd repos/$HOST;
borg1 serve --append-only --restrict-to-path repos/$HOST\",
no-port-forwarding,
no-X11-forwarding,
no-pty,
no-agent-forwarding,
no-user-rc
ssh-rsa AAAA...
Another security feature of the assembled authorized_keys is to restrict every access of the backup clients to append-only mode.
A compromise of the client VM, the ssh key and/or the borg passphrase does not automatically imply loss of data, although it may imply disclosure of the encrypted data.
All remarks concerning Borg's append-only mode apply.
The passphrase used to access a given borg repo is stored as plain text in the target VM and in the VM used to create the keys.
The passphrase must be stored as plaintext in order to allow automated backups.
If the VM (or a given backup client) is compromised and this password is leaked, the backups from this VM are also compromised.
Therefore, each VM (or backup client host) must use a different, strong passphrase such that the compromise of one passphrase does not imply the compromise of all backup repos.
The same considerations apply to the ssh keys used to connect to the backup server as the ones to the borg repo passphrase.
The keys must be passwordless (in order to allow automated backups) and also unique among the clients such that a compromise of one key does not immediately give access to the others.
It is the recommended Qubes philosophy to use a dedicated VM for a given set of activities.
One possible application of this is to use a VM dedicated to backup maintenance (the VM which will have the admin-keys.openssh) and this VM will generate the keys and passphrases for the other VMs.
In this situation, if the backup maintenance VM is compromised, all keys and passphrases may be compromised.
To mitigate against this, after verifying that the new repo is working, you should remove all files (except the *.pub files) from the host-keys dir.
You could also generate the ssh key and passphrase on the client VM and initialize the repository from there.
Losing access to the backed up data in the event of loss of the primary data is as bad as not having a backup in the first place.
To prevent losing access to your backups using this workflow on rsync.net you should have an offline, secondary backup of the relevant keys and passphrases.
At the very least, you should have an offline copy of:
- The e-mail credentials used to register on rsync.net
- The passphrases for each borg repo
- The rsync.net user password
You should also try to maintain copies of:
- The admin private key used to perform backup maintenance
- The key file for each borg repo
This should be stored offline (preferably on multiple media) and on a different site. Obviously, you should also store all this information encrypted with a strong master password.
One possible approach to achieve this is to save all this information in a KeePassX file and then print it using PyPaperBak.
You can also further mitigate risk of data loss by having a copy of all data on rsync.net either locally on another computer/disk or on another provider.
Rsync.net supports the s3cmd utility, so you could clone your data to S3 or Glacier.
Another option is to rsync it to some other backup provider.
Another strategy altogether would be to use Borg to backup to a local computer and then rclone it to rsync.net (or backblaze or S3).