Upgrade to Postgres 10 images
Closed this issue · 9 comments
Project is currently using Postgres 9.5 images, update to version 10 images.
There does not appear to be a direct upgrade path from the version 9.5 images to the version 10 images. The version 10 image will upgrade from a version 9.6 image. So best do this now.
Reference:
Environments to Update:
-
devex-von-dev
- postgresql
- wallet-db
-
devex-von-test
- postgresql
- wallet-db
-
devex-von-bc-tob-dev
- postgresql
- wallet-db
-
devex-von-bc-tob-test
- postgresql
- wallet-db
-
devex-von-bc-tob-prod
- postgresql
- wallet-db
-
ontvon-von-dev
- postgresql
- wallet-db
-
ontvon-von-test
- postgresql
- wallet-db
-
ontvon-von-prod
- postgresql
- wallet-db
-
devex-von-permitify-dev
- wallet-db
-
devex-von-permitify-test
- wallet-db
It looks like it is possible to upgrade a Postgres 9.5 image to a Postgres 9.6 image, and then to a Postgres 10.
The process seems fairly simple and would be way faster than having to reload data in each environment. Works great in docker, fails in OpenShift. Looking into logs and possible reasons.
There is a low level SELinux issue with gluster-file-db (gluster-file) that is not allowing a unix socket file to be created on persistent storage.
Tests confirm gluster-block is not affected and it is possible to perform the upgrade processes using gluster-file provided the socket file is not created anywhere on the persistent storage.
There are, however, a number of layers that make it difficult to point the socket file at a different location. It appears the pg_upgrade --old-options are set automatically and -c unix_socket_directories= option is automatically set to the current directory. The automated upgrade script does a cd "$HOME"/data so the logs are written to persistent storage. Things can be run manually in different directories but the results are not completely stable since there are additional steps the script performs that can easily be missed when trying to run things manually.
I've given all of my findings to the ops guys, and there is a ticket in with Red Hat engineering.
The automated approach is blocked by the platform issue mentioned above. I'm waiting for an update on the timing of a fix ... At this point I'm considering more extreme approaches such as backup/restore or starting fresh where possible.
Leaning toward backup/restore as it would also excessive our recovery strategy for the wallets.
In #651, the support for database migration/upgrade was tested thoroughly. Specifically our requirements; upgrade to Postgres V10 from V9.5.
So we have a migration path and a DR plan all rolled into a single solution.
PostgreSQL v10 images have been queued up to be deployed to the prod environments. Upgrades of those environments will be done as part of the migration to the upgraded SLN. Until that time any deployments to prod should be done at the component level to avoid deploying the new version of PostgreSQL.
@esune, I see you already have PostgreSQL v10 in the Permitify pipeline; deployed to dev, and staged and ready to be deployed to test once that environment is upgraded. Once that is done please mark the devex-von-permitify-test environment as complete in this ticket.
Lessons learned while restoring devex-von-bc-tob-test:
- 22GB Wallet database
- Backs up in 11.5 minutes and compresses to 4.9GB
- Takes almost 2 hours to restore.
@esune, The only environment left to upgrade is the Permitify test environment.
devex-von-permitify-test was updated correctly.