Database initialization failure makes device not functional

[SLDP-8255]

---

name: Database initialization failure makes device not functional
about: PostgreSQL database initialization

What happened?

I experienced a kind of critical problem with PostgreSQL database that was not succesfully initialized during first boot. A cause of that was a power outage probably while the initialization was executing. An effect of that is unfunctional database and therefore other Chirpstack components like the App server.
I found an issue in the firstbootinit script which performs initialization of PostgreSQL database, for example in the following line:
https://github.com/brocaar/chirpstack-gateway-os/blob/e858d9ff30f4d9e77d360c7cd86c32978c206544/meta/recipes-database/firstbootinit/firstbootinit/firstbootinit.sh#L7
there are no any checks of return codes, so in the case of any failure subsequent commands in the script are executed and at the end a flag is set to indicate that the database has been initialized without checking whether the database has been initialized in actual.

At a next boot, the operations won't be executed because the flag /var/lib/firstbootinit/postgres_initdb prevents the initialization of an already initialized database.
This creates the possibility that, for example, a power outage or other failure during database initialization will make the device non-functional without administrator intervention.

What did you expect?

A better solution would be to check the return codes of each operation in the initialization procedure and only execute the next line in the script when the previous one was successful.

Steps to reproduce this issue

To reproduce this issue in actual, there is need to try suddenly disconnecting a power supply while a device is booting first time.

Your Environment

Component	Version
ChirpStack Gateway OS	v3.5.1