Backup breaks startup
Closed this issue · 4 comments
After pregenerating my world, the files in the minecraft folder are getting really large. This causes the backup to take really long wich results in a timeout when starting the systemd service.
My workaround at the moment is a separate script wich makes the backup and is executed by crontab every day.
Hmmm, this is an interesting one. How big are we talking?
One thing people have done when related issues have come up (the issue is the systemd startup timeout value and that backing up your server is taking longer than the 600 second timeout) is you can edit the systemd service with:
sudo nano /etc/systemd/system/minecraft.service
In there you will see a line that says TimeoutStartSec=600. This means 600 seconds. This is almost certainly what is failing (you can see that it's probably timing out waiting for it with sudo systemctl status minecraft or journalctl -xe
Increasing that timeout has helped with really large servers in the past. You are the first one to bring up pregenerated worlds but that is an interesting consideration. Perhaps I need to spin off the backup as a separate process (shouldn't be too hard theoretically, pretty similar to what you are doing with crontab but just being called from the script directly instead).
After you've increased the timeout press Ctrl+X to save the file in nano and then type:
sudo systemctl daemon-reload
I'd say give adjusting the timeout a try as a shorter term fix and I can definitely look into adding in a separate backup thread/process. The only issue I could see is you can't be backing up the server files during a write and if the server is running you may have some sketchy/corrupt data in the backups potentially.
The server could also crash if the backup utility is accessing those files when it requests an exclusive write lock on the file to record some new/changed block data in that area of the map files. The busier your server is (as in number of players online concurrently) the more likely it is to happen, and it's most likely to happen in the most populated areas of the server since that is where the most block changes will be constantly happening.
If backups like these are running ideally nobody should be online, but even then the map files do get touched from time to time as tiles are "ticked" (think operations like the daily reset for villagers profession inventories, or crops growing which will grow when nobody is on) and other longer term operations still run even though most stuff is suspended when there's nobody online (the monsters don't move if nobody is around and usually despawn altogether if they aren't in a protected area for example, most things freeze/stop/are deleted from memory).
The server should honestly be locking the files but maybe it's not locking them if it hasn't actively been using that file. If your backup utility/script locks the files (or prevents the server from exclusively locking them when it wants to write to them) the server would crash. It's probably going to run fine 95%+ of the time but if you see weird errors or backups that are smaller than they should be / missing it's likely that you ran into some files locking issues during that run. It could manifest as a server crash or a problem in the backup, it's hard to predict these things but it's a concern for sure.
Hopefully adjusting the timeout gets it going for you in the immediate term!
Thank you for the fast response.
At the moment one Backup is about 3GB when I first got the issue the Backup file had a size of about 2.5GB. I stopped pregenerating at this point because at the moment I only have a 16GB sd card.
I worked with systemd before but I didn't know/found the Timeout option. I will definitely give it a try.
Also thank you for the long explanation at the end. I already expected that my solution could break something when someone is on the sever but I didn't consider the ticking when everyone is offline.
No worries at all! That sounds like a pretty good setup. Upgrading to a SSD would be a strong recommendation as it would reduce this problem as part of the reason it's taking so long (even for 3GB) is because it's the SD card. There's no way even with an older model 2.5" SATA SSD that it would take 600 seconds for such an operation if that makes sense.
I do have a guide for doing this on the Pi here: https://jamesachambers.com/new-raspberry-pi-4-bootloader-usb-network-boot-guide/
Basically you could get a low size drive like a 128GB SSD such as a Kingston A400 and a 2.5" SATA adapter for < $50 altogether both new (and cheaper if you want to bargain hunt or already have a spare old drive around). Even a smaller 128GB model would give you enough for the default 10 rotating backups too even at 3GB each with room to spare.
Just another option to consider possibly but maybe since your world is pregenerated once everything is set up (with a longer timeout) it's going to work just fine. If it's lacking in performance though or you aren't happy with it that would for sure be the next logical step and would definitely resolve all of these. It would boost your pregeneration speed substantially, but you would likely find you don't need it with a SSD (as I haven't with a few players online concurrently in my personal sessions at least).
Definitely let me know if I can help further!
I tried it and with the new timeout it works just perfect.
I actually have a M.2 SATA SSD lying around but I won't be able to buy an adapter until January. So over the holidays the micro SD must do the job. Until then I automatically copy my backups over to my NAS with rsync and delete them on the Minecraft Server
About the pregeneration: I also tried a rented server with 2GB of RAM and SSD advertised. Every time new chunks were generated my TPS dropped below 14. That's the reason I tried the Raspberry Pi and pregeneration in the first place.