2 more questions.. diffs are not cleaned up and /.snapshots/ exclude still shows warnings

Question

2 more questions.. diffs are not cleaned up and /.snapshots/ exclude still shows warnings

zilexa opened this issue 4 years ago · 9 comments

Just 2 more questions if you don't mind! Thanks for the help so far:

I don't use Snapper for anything else but snapraid-btrfs (I use btrbk for backups). I only used sync and cleanup. Now, since I run your script, I notice in the Snapper-GUI, there is also a diff type snapshot created. Those diffs are not cleaned up with your runner script.
I ran the script 4 times to test the email server, and now I have 4 diff snapshots for each disk, even though cleanup=true.
Is this expected behaviour?
In my snapraid.conf I have exclude /.snapshots/ but I still get these warnings with snapraid-btrfs and with your script:

[OUTERR] WARNING! Ignoring mount point '/mnt/disks/cache/Users/.snapshots/7/snapshot/.snapshots' because it appears to be in a different device

I understand it cannot harm since the actual snapshots within are subvolumes themselves so they are excluded. But why the message? It is driving me crazy. Note my disks are actually mounted at /mnt/disk/diskname but each contains 3 subvolumes, I am only covering one called Users. My snapper config files contain the exact same path as in the snapraid.conf (/mnt/disks/cache/Users for example):

# Defines the data disks to use
# The name and mount point association is relevant for parity, do not change it
# WARNING: Adding here your /home, /var or /tmp disks is NOT a good idea!
# SnapRAID is better suited for files that rarely changes!
# Format: "data DISK_NAME DISK_MOUNT_POINT"
data d0 /mnt/disks/cache/Users
data d1 /mnt/disks/data1/Users
data d2 /mnt/disks/data2/Users

# Excludes hidden files and directories (uncomment to enable).
#nohidden

# Defines files and directories to exclude
# Defines files and directories to exclude
# Remember that all the paths are relative at the mount points
# Format: "exclude FILE"
# Format: "exclude DIR/"
# Format: "exclude /PATH/FILE"
# Format: "exclude /PATH/DIR/"  
exclude /.snapshots/             # snapraid-btrfs (snapper) snapshot folder
exclude /.snapraid/              # snapraid content subvolume folder
exclude /.filerun.trash/         # filerun trash folder
exclude /.stversions/            # Syncthing deleted files (by peer, synced to this device)
exclude /incoming/incomplete/    # temp torrent download subvolume folder
exclude *.unrecoverable
exclude /tmp/
exclude /lost+found/
exclude .tmp/
exclude .cache/
exclude /lost+found/
exclude .AppleDouble
exclude ._AppleDouble
exclude .DS_Store
exclude .Thumbs.db
exclude .fseventsd
exclude .Spotlight-V100
exclude .TemporaryItems
exclude .Trashes
exclude .AppleDB
exclude .err
exclude .bak
exclude .dmp
exclude .dump
exclude .old
exclude .tmp
exclude .temp

Answer 1 · 2021-03-26T20:05:18.000Z

Hello!

I don't use Snapper for anything else but snapraid-btrfs (I use btrbk for backups). I only used sync and cleanup. Now, since I run your script, I notice in the Snapper-GUI, there is also a diff type snapshot created. Those diffs are not cleaned up with your runner script.
I ran the script 4 times to test the email server, and now I have 4 diff snapshots for each disk, even though cleanup=true.
Is this expected behaviour?

This is not the expected behaviour. Three questions:

I know you mentioned it, but can you double-check that you have cleanup = true in your config?
Do you see a line item Running cleanup... in your logs?
If you run snapraid-btrfs cleanup manually, does it remove all but the last snapraid-btrfs=synced snapshot?

In my snapraid.conf I have exclude /.snapshots/ but I still get these warnings with snapraid-btrfs and with your script:
[OUTERR] WARNING! Ignoring mount point '/mnt/disks/cache/Users/.snapshots/7/snapshot/.snapshots' because it appears to be in a different device
I understand it cannot harm since the actual snapshots within are subvolumes themselves so they are excluded. But why the message? It is driving me crazy. Note my disks are actually mounted at /mnt/disk/diskname but each contains 3 subvolumes, I am only covering one called Users. My snapper config files contain the exact same path as in the snapraid.conf (/mnt/disks/cache/Users for example):
...
exclude /.snapshots/             # snapraid-btrfs (snapper) snapshot folder
...

I have that exact exclude /.snapshots/ line item in my /etc/snapraid.conf and I don't get that warning. Do you get the same warning if you run snapraid-btrfs sync directly?

Answer 2 · 2021-03-26T21:00:40.000Z

Yup cleanup is there
Also in the log I count 4 times Running cleanup...
And yes, when I run snapraid-btrfs sync I do get the /.snapshots/.... warning.

BTW, my btrfs root of my system disk looks like this:

@ 
@home
@docker

@ is mounted as /
Apart from the docker subvolume, this is default setup of Ubuntu when you select btrfs as filesystem during setup wizard.

.snapshots is a subvolume created by Snapper by default in /.snapshots.

[snapraid-btrfs]
; path to the snapraid-btrfs executable (e.g. /usr/bin/snapraid-btrfs)
executable = /usr/bin/snapraid-btrfs
; specify snapper-configs and/or snapper-configs-file as specified in snapraid-btrfs; only one instance of each can be specified in this config
snapper-configs = 
snapper-configs-file = 
; specify whether snapraid-btrfs should run the pool command after the sync, and optionally specify pool-dir 
pool = false
pool-dir =
; specify whether snapraid-btrfs-runner should automatically clean up all but the last snapraid-btrfs sync snapshot after a successful sync
cleanup = true

[snapper]
; path to snapper executable (e.g. /usr/bin/snapper)
executable = /usr/bin/snapper

[snapraid]
; path to the snapraid executable (e.g. /usr/bin/snapraid)
executable = /usr/local/bin/snapraid
; path to the snapraid config to be used
config = /etc/snapraid.conf
; abort operation if there are more deletes than this, set to -1 to disable
deletethreshold = -1
; if you want touch to be ran each time
touch = false

[logging]
; logfile to write to, leave empty to disable
file = snapraid.log
; maximum logfile size in KiB, leave empty for infinite
maxsize = 5000

[email]
; when to send an email, comma-separated list of [success, error]
sendon = success,error
; set to false to get full programm output via email
short = true
subject = SnapRAID Status Report
from =
to = 
; maximum email size in KiB
maxsize = 500

[smtp]
host = 
; leave empty for default port
port = 587
; set to "true" to activate
ssl = false
tls = true
user = 
password =

[scrub]
; set to true to run scrub after sync
enabled = false
percentage = 12
older-than = 10

2021-03-26 18:14:14,761 [INFO  ] ************************************************************
2021-03-26 18:14:14,762 [INFO  ] Diff results: 0 added,  0 removed,  0 moved,  0 modified
2021-03-26 18:14:14,762 [INFO  ] No changes detected, no sync required
2021-03-26 18:14:14,762 [INFO  ] Running cleanup...
2021-03-26 18:14:15,280 [INFO  ] ************************************************************
2021-03-26 18:14:15,281 [INFO  ] All done

Answer 3 · 2021-03-27T00:38:08.000Z

Okay. Let's focus on the cleanup part first, and let's decouple whether it's an issue with the snapraid-btrfs-runner script vs upstream snapraid-btrfs. Run the following and provide the outputs requested:

snapraid-btrfs --quiet diff
snapraid-btrfs --quiet ls | grep snapraid     # show the output of this post-diff snapshot list
snapraid-btrfs --quiet sync
snapraid-btrfs --quiet ls | grep snapraid     # show the output of this post-sync snapshot list
snapraid-btrfs --quiet cleanup
snapraid-btrfs --quiet ls | grep snapraid     # show the output of this post-cleanup snapshot list

No need to paste the output of the diff and sync commands. Thanks.

Answer 4 · 2021-03-27T12:44:19.000Z

$ snapraid-btrfs --quiet ls | grep snapraid  
1  | single |       | za 27 mrt 2021 13:35:47 CET | asterix |         | snapraid-btrfs diff | snapraid-btrfs=diff
1  | single |       | za 27 mrt 2021 13:35:47 CET | asterix |         | snapraid-btrfs diff | snapraid-btrfs=diff
1  | single |       | za 27 mrt 2021 13:35:51 CET | asterix |         | snapraid-btrfs diff | snapraid-btrfs=diff

$ snapraid-btrfs --quiet ls | grep snapraid 
1  | single |       | za 27 mrt 2021 13:35:47 CET | asterix |         | snapraid-btrfs diff | snapraid-btrfs=diff  
2  | single |       | za 27 mrt 2021 13:36:21 CET | asterix |         | snapraid-btrfs sync | snapraid-btrfs=synced
1  | single |       | za 27 mrt 2021 13:35:47 CET | asterix |         | snapraid-btrfs diff | snapraid-btrfs=diff  
2  | single |       | za 27 mrt 2021 13:36:21 CET | asterix |         | snapraid-btrfs sync | snapraid-btrfs=synced
1  | single |       | za 27 mrt 2021 13:35:51 CET | asterix |         | snapraid-btrfs diff | snapraid-btrfs=diff  
2  | single |       | za 27 mrt 2021 13:36:21 CET | asterix |         | snapraid-btrfs sync | snapraid-btrfs=synced

$ snapraid-btrfs --quiet ls | grep snapraid 
2  | single |       | za 27 mrt 2021 13:36:21 CET | asterix |         | snapraid-btrfs sync | snapraid-btrfs=synced
2  | single |       | za 27 mrt 2021 13:36:21 CET | asterix |         | snapraid-btrfs sync | snapraid-btrfs=synced
2  | single |       | za 27 mrt 2021 13:36:21 CET | asterix |         | snapraid-btrfs sync | snapraid-btrfs=synced

And when I check Snapper-GUI indeed I only see 1 snapshot besides the default (0). Seems to go well now.

Still get this though:

$ snapraid-btrfs --quiet sync
Self test...
Loading state from /.snapraid/snapraid.content...
Scanning disk d0...
WARNING! Ignoring mount point '/mnt/disks/cache/Users/.snapshots/2/snapshot/.snapshots' because it appears to be in a different device
Scanning disk d1...
WARNING! Ignoring mount point '/mnt/disks/data1/Users/.snapshots/2/snapshot/.snapshots' because it appears to be in a different device
Scanning disk d2...
WARNING! Ignoring mount point '/mnt/disks/data2/Users/.snapshots/2/snapshot/.snapshots' because it appears to be in a different device
WARNING! UUID is unsupported for disks: 'd0', 'd1', 'd2'. Not using inodes to detect move operations.
Using 129 MiB of memory for the file-system.
Initializing...

Snapraid should not require the full path right? exclude /.snapshots/ should work.

Answer 5 · 2021-03-27T15:04:25.000Z

Seems to go well now.

Good to hear. Now please show the contents of your snapraid-btrfs-runner.conf and the exact command you use to run snapraid-btrfs-runner.

Still get this though:

Understood. I'm still focused on the cleanup question right now because that's something that is fully controlled within this script. Once we sort that out I can try to help with the exclude for /.snapshots/.

Answer 6 · 2021-03-27T21:59:40.000Z

My conf hasn't changed:

[snapraid-btrfs]
; path to the snapraid-btrfs executable (e.g. /usr/bin/snapraid-btrfs)
executable = /usr/bin/snapraid-btrfs
; specify snapper-configs and/or snapper-configs-file as specified in snapraid-btrfs; only one instance of each can be specified in this config
snapper-configs = 
snapper-configs-file = 
; specify whether snapraid-btrfs should run the pool command after the sync, and optionally specify pool-dir 
pool = false
pool-dir =
; specify whether snapraid-btrfs-runner should automatically clean up all but the last snapraid-btrfs sync snapshot after a successful sync
cleanup = true

[snapper]
; path to snapper executable (e.g. /usr/bin/snapper)
executable = /usr/bin/snapper

[snapraid]
; path to the snapraid executable (e.g. /usr/bin/snapraid)
executable = /usr/local/bin/snapraid
; path to the snapraid config to be used
config = /etc/snapraid.conf
; abort operation if there are more deletes than this, set to -1 to disable
deletethreshold = -1
; if you want touch to be ran each time
touch = false

[logging]
; logfile to write to, leave empty to disable
file = snapraid.log
; maximum logfile size in KiB, leave empty for infinite
maxsize = 5000

[email]
; when to send an email, comma-separated list of [success, error]
sendon = success,error
; set to false to get full programm output via email
short = true
subject = SnapRAID Status Report
from = 
to = 
; maximum email size in KiB
maxsize = 500

[smtp]
host = 
; leave empty for default port
port = 587
; set to "true" to activate
ssl = false
tls = true
user = 
password = 

[scrub]
; set to true to run scrub after sync
enabled = false
percentage = 12
older-than = 10

Command used: python3 snapraid-btrfs-runner.py
without sudo

Answer 7 · 2021-03-27T22:32:40.000Z

Command used: python3 snapraid-btrfs-runner.py

The snapraid-btrfs-runner.conf is in the same directory that you run the script from? If not or if there's a different conf there, use --conf to explicitly specify it.

without sudo

I don't think this is the issue but it could be worth trying it with sudo or as root. On my setup, I have cleanup = true and the script successfully runs the diff, sync, cleanup, and scrub every night for the past few months. I run snapraid and snapraid-btrfs as root, though. I'd be curious if you could test this and see if there's a difference. Thanks.

Answer 8 · 2021-09-03T21:10:50.000Z

I know this is old, but I just ran into this when I was testing. The diffs weren't getting cleaned up because if you look at the help page for snapraid-btrfs cleanup it states:
Delete all snapshots created by snapraid-btrfs before the last one a successful sync has been completed with.
Because I was testing and the runner saw no changes in the diff, it did not perform a sync, which means the diff occurred after the last sync not before. Therefore the cleanup does not cleanup the diff.

Answer 9 · 2021-09-29T02:28:48.000Z

@FlexibleToast , thanks for the comment. I'm closing this issue as it appears that the cleanup command is working as expected. Thanks.