Simple time-based lock service with HTTP interface.
Oath to solve the Thundering herd problem during multiple applications startup in the Kubernetes cluster.
Designed atThe Problem
Starting multiple applications simultaneously on the same host may cause a performance bottleneck. In Kubernetes this usually happens when applications are automatically deployed to a newly added Node. In the worst-case scenario, application startup may be slowed down so dramatically that they fail to pass the healthcheck. They are then restarted by Kubernetes just to start fighting for shared resources again, in an endless loop.
The Solution
Kubernetes allows a Pod to have additional, Init container, and postpone application startup until Init container finishes execution. The solution is to deploy Lock service as a DaemonSet on a Pod, and each init container will sequentially acquire this lock. So moments of application container starts will be distributed in time.
Components
See Readmes in subfolders for details.
-
Lock
HTTP service to be deployed one instance per Node (as a DaemonSet). Returns code
200 OK
as a response to the first request. Returns423 Locked
to the subsequent requests until timeout exceeded. May depend on additional endpoint check. -
Init
Lightweight client for the Lock service. To be deployed as Init Container alongside the main application container. Periodically tries to acquire the lock. Once succeeded, terminates, allowing the main container to start running.
-
K8s-health
Optional component. Performs healthcheck of Kubernetes DaemonSets. May be used by Lock service to postpone lock acquiring until all DaemonSets on the Node are up and running.
How to build locally
The project is built using Make.
Go
0. Installwget https://golang.org/dl/go1.16.10.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.16.10.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
Dep and update dependencies
1. InstallThe project uses Dep for dependency management. You can find installation instructions on the project page.
mkdir -p ~/go/{bin,src}
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
Then run Make:
make dep
go mod init
go mod tidy
go mod vendor
2. Set target platform for Go binaries
Optional step, default is linux
. You can use Make task as shown below.
Unix example:
export GOOS=openbsd
Windows:
set GOOS=windows
3. Build binaries
Run Make:
make
Or with target platform specified:
make darwin build
Binaries will be located in sbfolder's bin
folders:
init/bin/init
k8s-health/bin/health
lock/bin/lock
4. Or Build Docker images
First, you need to specify Docker user name as a variable: DOCKER_USER
.
Then run Make:
make docker-build
5. Or Build Docker images and push them to the repo
First, you need to specify Docker credentials as environment variables: DOCKER_URL
, DOCKER_USER
, and DOCKER_PW
.
Then run Make:
make docker-push
Release Notes
1.0.2
- Bumped Go version to 1.16. Bumped Kubernetes version to 1.16.
1.0.1
- Added connection timeouts for http and tcp connections; Added keep-alive for http connections.
1.0.0
- Initial version.
Contributing
Please feel free to submit issues, fork the repository and send pull requests!