Detect failed self update and rollback if failed
Closed this issue · 6 comments
If the self-update fails, the launcher should be able to recover.
@StefanKert did we understand this correctly? 😁
Yep, this is the interesting part when it comes to updating the launcher. In fact we should think about how we can make sure that we counter the risk of a broken installation of an update and if there are still atomic steps involved that can lead to a state that is unrecoverable, how can we notice that in a scaled way and how can we rollback the faulty update.
We have 2 cases I think:
-
The update process fails, and the installation is broken.
Maybe solved by not overwriting the launcher executable but backing up (renaming) the old executable and then moving the new one. We can then check the executable hash or sth like that and rollback to the backup if it's different.
-
The update process fails but the updated executable is faulty (e.g. contains a bug or does not work on the system for some reason)
I don't think we can recover from this case. We would need the updater to run and check the launcher which is (I think) not really possible when running the launcher as a service because the update happens on system shutdown.
Maybe solved by not overwriting the launcher executable but backing up (renaming) the old executable and then moving the new one. We can then check the executable hash or sth like that and rollback to the backup if it's different.
That sounds reasonbale. So the rename would be the last action that we can perform and it should probably be also the last step so that we can recover in case the laucherupdate crashes in between. So similar to enabling a Feature Flag right?
I don't think we can recover from this case. We would need the updater to run and check the launcher which is (I think) not really possible when running the launcher as a service because the update happens on system shutdown.
Do we have something like --dry-run?
So similar to enabling a Feature Flag right?
I'm not sure I understand 😅
Do we have something like --dry-run?
No, but I see how that would be helpful in this situation.
Another thing that could lead to problems though is that currently we do the update on service shutdown (= system shutdown). If we take too long to check the validity the user/os might force shutdown and interrupt the update process.
@StefanKert After the update we run the doctor command on the updated launcher from the selfupdate which checks the launcher health.
Then we rollback to the previous version if the doctor detects a problem
Example:
023-01-16 12:18:33.878 +01:00 [INF] Launcher update successful
2023-01-16 12:19:35.392 +01:00 [INF] Waiting for launcher to shut down.
2023-01-16 12:19:38.175 +01:00 [INF] Copying launcher executable from "C:/ProgramData\fiskaltrust\service\c626f3e7-0401-4c7d-b431-5c0cc565cc98\fiskaltrust.Launcher\fiskaltrust.Launcher.exe" to "C:\Users\paul.volavsek\source\middleware-launcher\src\fiskaltrust.Launcher\bin\Debug\net7.0\win-x64\publish\fiskaltrust.Launcher.exe".
2023-01-16 12:19:39.517 +01:00 [INF] Running launcher health check.
2023-01-16 12:19:39.534 +01:00 [INF] doctor --launcher-configuration C:\Users\paul.volavsek\source\middleware-launcher\tmp\test.json --cashbox-id "c626f3e7-0401-4c7d-b431-5c0cc565cc98" --access-token "BJktVxsLVkNlb980XkNbskWFhdtPl2PCWapfxLfqJabsrH+8hsSVbV+CxkN2yDn6258Me9WYhGuHsDKPEPFrsNk=" --sandbox "True" --launcher-version "2.0.0-preview4.23016.59245"
2023-01-16 12:19:42.821 +01:00 [INF Doctor]
[12:19:42 INF] ✅ Load ECDH Curve
[12:19:42 INF] ✅ Download cashbox configuration
[12:19:42 INF] ✅ Parse cashbox configuration in launcher configuration
[12:19:42 INF] ✅ Parse cashbox configuration
[12:19:42 INF] ✅ Decrypt cashbox configuration
[12:19:42 INF] ✅ Setup data protection
[12:19:42 INF] ✅ Decrypt launcher configuration
[12:19:42 INF] ✅ Setup monarch services
[12:19:42 INF] ✅ Setup monarch ProcessHostService
[12:19:42 INF] ✅ Build monarch WebApplication
[12:19:42 INF] ✅ Start monarch WebApplication
[12:19:42 INF] ✅ Start plebian processhostservice client
[12:19:42 INF] ✅ Setup plebian services
[12:19:42 INF] ✅ Build plebian Host
[12:19:42 INF] ✅ Start plebian Host
[12:19:42 INF] ✅ Shutdown launcher gracefully
[12:19:42 INF] Doctor found no issues.
2023-01-16 12:19:42.821 +01:00 [INF] Launcher update successful
Can we close this issue?