Failed to find dump folder
Migsi opened this issue · 9 comments
Hi, I just upgraded heavyscript on two of my TrueNAS Sclae machines, which went fine so far. But strangely, on one machine I get errors regarding the dump folder, even though I did everything the same. The output is as follows:
...
🅃 🄰 🅂 🄺 🅂 :
-Backing up ix-applications dataset
-Syncing catalog(s)
Please wait for output from both tasks..
find: ‘./database_dumps’: No such file or directory
du: cannot access './database_dumps/*': No such file or directory
🅂 🅈 🄽 🄲
Catalog sync complete
...
The script seems to work fine otherwise, so I'm not sure where this comes from. This happens, no matter if I'm calling the script manually or via parameters, as in a cron job. I called the script as root.
Does this system have cnpg apps?
k3s kubectl get deployments --all-namespaces | grep -E '^(ix-.*\s).*-cnpg-main-' | awk '{gsub(/^ix-/, "", $1); print $1}'
There are:
lldap
authelia
And they are stopped via the recently added "Stop All" flag. But I don't get it, how does this affect (finding) the dump folder?
Can you run the script with
bash -x heavy_script.sh backup --create 14Then show me the output.
Not at my machine rn, but at least the email reports of the cron job suggest the issue is gone now. Heavyscript since updated iteself to the most recent version though (from 2.0.2 to 2.1.0) and I've updated a few more apps to the "new common" Truecharts releases. However, I did not touch the afforementioned lldap and authelia. This is the most recent output I've got via mail by cron job execution:
...
🅃 🄰 🅂 🄺 🅂 :
-Backing up ix-applications dataset
-Syncing catalog(s)
Please wait for output from both tasks..
🄱 🄰 🄲 🄺 🅄 🄿 🅂
Number of backups was set to 30
Failed to get cnpg pod for lldap.
Failed to back up lldap's database.
Failed to get cnpg pod for authelia.
Failed to back up authelia's database.
App Name Total Size
home-assistant 84M
[0%] ...
[10%] Basic validation complete...
[15%] Backing up ddns-route53...
[21%] Backing up pihole...
[27%] Backing up nextcloud...
[32%] Backing up lldap...
[38%] Backing up traefik...
[44%] Backing up miraculix...
[50%] Backing up egmontix...
[55%] Backing up authelia...
[61%] Backing up pgadmin...
[67%] Backing up pihole-tc-workaround...
[72%] Backing up storj-node...
[78%] Backing up nextcloud-tc-workaround...
[84%] Backing up grocy...
[90%] Backing up home-assistant...
[95%] Taking snapshot of ix-applications...
[100%] Backup 'HeavyScript_2023_05_29_02_00_02' complete...
HeavyScript_2023_05_29_02_00_02
Deleted the oldest backup(s) for exceeding limit:
HeavyScript_2023_05_10_02_00_02
🅂 🅈 🄽 🄲
Catalog sync complete
...
Unil I get to the machine its a shot in the dark, but I'd say it looks like the script has some issues with handling shut down apps. The previously not found backup folder might have been created by a now working backup of e.g. home-assistant, which now also uses cnpg.
Test out the latest 2.1.1 release and let me know if that works for your stopped cnpg applications.
The most recent run printed this:
...
🅃 🄰 🅂 🄺 🅂 :
-Backing up ix-applications dataset
-Syncing catalog(s)
Please wait for output from both tasks..
🅂 🅈 🄽 🄲
Catalog sync complete
Validation errors:
* values.persistence.data.enabled: Field was not expected
* values.persistence.data.mountPath: Field was not expected
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Error from server (NotFound): pods "lldap-cnpg-main-1" not found
Validation errors:
* values.persistence.data.enabled: Field was not expected
* values.persistence.data.mountPath: Field was not expected
🄱 🄰 🄲 🄺 🅄 🄿 🅂
Number of backups was set to 30
Failed to get cnpg pod for lldap.
Failed to back up lldap's database.
App Name Total Size
authelia 32K
home-assistant 261M
[0%] ...
[10%] Basic validation complete...
[15%] Backing up ddns-route53...
[21%] Backing up pihole...
[27%] Backing up nextcloud...
[32%] Backing up egmontix...
[38%] Backing up miraculix...
[44%] Backing up pgadmin...
[50%] Backing up traefik...
[55%] Backing up lldap...
[61%] Backing up authelia...
[67%] Backing up home-assistant...
[72%] Backing up nextcloud-tc-workaround...
[78%] Backing up pihole-tc-workaround...
[84%] Backing up storj-node...
[90%] Backing up grocy...
[95%] Taking snapshot of ix-applications...
[100%] Backup 'HeavyScript_2023_06_05_02_00_03' complete...
HeavyScript_2023_06_05_02_00_03
Deleted the oldest backup(s) for exceeding limit:
HeavyScript_2023_05_17_02_00_01
🅄 🄿 🄳 🄰 🅃 🄴 🅂
...
Apparently authelia was started automatically during an update, so that completed fine I guess.
lldap is an upstream issue, that Truecharts is currently working on to fix.
The errors waiting for the pod to become active have been silenced in the next release and can be tested with the lifecycle branch of heavyscript.
You can switch to this branch with the heavyscript menu, or by using:
heavyscript git --branch
Then choosing lifecycle.
There is nothing that can be currently done with the validation errors, but those errors are also silenced in the lifecycle branch that should be merged as soon as ample testing has been done.
The issues should be solved with the latest release. Please re-open if they are not. Upstream issues have also been fixed
lldap is an upstream issue, that Truecharts is currently working on to fix.
The errors waiting for the pod to become active have been silenced in the next release and can be tested with the
lifecyclebranch of heavyscript.You can switch to this branch with the heavyscript menu, or by using:
heavyscript git --branchThen choosing lifecycle.
There is nothing that can be currently done with the validation errors, but those errors are also silenced in the lifecycle branch that should be merged as soon as ample testing has been done.
Apologize for responding late, but I can confirm the errors (if there are any left?) are hidden now. The script runs through smoothly again! Though I still hesitate to auto update all my apps due to other issues, not related to heavyscript itself (TLS settings clearing itself).
Regarding the validation errors, I can immagine those being hard to circumvent, when they are about unexpected hostPath mounts. I didn't check the script for what it tries to do at that point, but I can immagine it tries to gather all relevant data storages of an app to do a proper backup? If thats the case, you could implement a check if a given hostpath is a dataset itself and snapshot it or the nearest parent which is an actual dataset?