Alternate Liquibase Locking solution which makes able an application to recover from a terminated Schema update using Kubernetes API.
Liquibase is a database version control system for relational databases http://liquibase.org/
Kubernetes is a container orchestration platform http://kubernetes.io/
Liquibase Locks the database while it performs schema updates and when it checks which change sets are executed. If an application instance stops during this period the database remain locked and it prevents any other instances to lock the database, therefore no other instance can be started until the database is manually fixed.
This behavior is OK for an application which is deployed manually especially if that application runs on a single instance, but it's not acceptable in a multi instance environment where an automated system starts and stops instances in order to match the actual load. Kubernetes is solution for managing such environments, it can continuously monitor the running services and start new instances when an instance goes down or the load increases. Unfortunately the locking mechanism mentioned above can compromise the high availability by rendering new instances useless.
Fortunately Liquibase is easily extendable and Kubernetes has a great API.
This library is only active when a project is running in a Kubernetes Pod, in this case the API client can connect without additional authentication and check the status of the Pods, so it can decide that the pod which locked the database is still running or not. When Liquibase checks for it's lock entry, this library will release the current lock if the pod it's granted to is inactive or is the same as the pod which tries to lock the db (Container was restarted)
Currently two changes are implemented:
In the Standard implementation the lockedby value is intended to provide debug information for human operators. In this version it's changed to namespace:podName format and it's used to decide which pod holds the lock.
The Standard implementation checks for the lock and if the database is locked it waits for a while and if the lock can not be obtained, it fails. This version checks the lockedby attribute and looking up the pod to decide that the lock is still relevant or not, and if it's obsolete removes it. This allows the system to recover from a failure during schema update.
A partial schema update can cause problems if it's not rolled back properly, I tested this solution with PostgreSQL and according to my experiences Liquibase can rollback changes when it's used with PostgreSQL, but it can leave behind a dirty DB Schema with some other database engines
The library checks for connection with the Kubernetes API and if it's not able to connect the custom locking will not be activated, everything works as in the standard implementation.
Using the library is quite easy: it has to be added as a dependency and 2 environment variables has to be set.
<groupId>com.github.patricio78</groupId>
<artifactId>liquibase-kubernetes</artifactId>
<version>2.3.1</version>
implementation 'com.github.patricio78:liquibase-kubernetes:1.0.1-SNAPSHOT'
Two environment variables are required:
- POD_NAME
- POD_NAMESPACE
These can be provided by utilizing Kubernetes Donward API The required environment variable configuration looks like this:
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace##RBAC config
If you use RBAC in your cluster the following permissions are required for the pod's Service Account
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: reader-role
rules:
- apiGroups: [""]
resources: ["pods", "pods/status"]
verbs: ["get", "watch", "list"]