built-in support for glusterfs client on worker nodes
Closed this issue · 7 comments
We are implementing glusterfs cluster to be used as storage for persistent volumes using Heketi API and StorageClasses.
The scenario works only when the glusterfs client package is installed on the worker nodes manually when it comes to pod mounting. Since the manual package installation can be overwritten anytime the reconciler is run, it is requested to make it part of worker node image so that it does not have to be installed manually and also works in autoscale scenario.
Hey @m-usmanayub 👋
for cases like this, we usually recommend running a Daemonset that installs the binary onto the host system. One key advantage of this approach is that you do not depend on the cloud provider and can update the binary per your own preferences.
How do you feel about this approach?
(Sorry for the accidental close and half-finished comment, I fat-fingered that one.)
Hey @m-usmanayub 👋
for cases like this, we usually recommend running a Daemonset that installs the binary onto the host system. One key advantage of this approach is that you do not depend on the cloud provider and can update the binary per your own preferences.
How do you feel about this approach?
Hi @timoreimann
The approach seems realistic. Do you have any example of this?
Also in such a case we need to know the OS flavor (Ubuntu, CentOS etc) running on the worker nodes and install the relevant OS Package (deb/rpm) by creating a customized docker image. am I right?
@m-usmanayub I'd try to create a Daemonset that mounts the root host-filesystem, chroots into it, checks if the package is already installed, and if not updates the repository and installs the package. We currently run Debian, so the logic could look something like this (untested):
#!/bin/bash
chroot /host /bin/bash -u <<'EOF'
export GLUSTERFS_PACKAGE=glusterfs-client
dpkg -l ${GLUSTERFS_PACKAGE} > /dev/null
case $? in
0)
echo "nothing to do: glusterfs package is already installed"
;;
1)
echo "installing glusterfs package"
apt-get update && apt-get install -y ${GLUSTERFS_PACKAGE}
;;
*)
echo "failed to check if glusterfs is installed" >&2
exit 1
;;
esac
sleep infinity
EOFYou might not even need a custom container image if you choose to inline the script as arguments to a container that comes with bash (e.g., a Debian-based image).
For mounting the host file-system, you'll need your Daemonset to specify volume and volume mount fields accordingly. A small blog post on the topic is available here, and our own doks-debug Daemonset does it as well. (Adjust the manifest to your needs, of course.)
Hope this gets you started. Note that while we do not have plans to do so, we could change the distro and/or distro version in the future. Such a change would likely be bundled with a Kubernetes minor release, however, so if you keep an eye on the release notes you should be able to respond to any updates we might do that affect how your Daemonset works.
Thanks for the explanation, I was able to run a DaemonSet using the logic mentioned however with some changes. As you said we would have to keep a check on the distro and its version of the worker nodes. Please feel free to close the issue if you believe that the package cannot be/will never be made part of the worker nodes. Thanks.
I am posting the yaml definition for anyone who would like to use.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: gfs-client
namespace: kube-system
labels:
app: gfs-client
spec:
selector:
matchLabels:
name: gfs-client
template:
metadata:
labels:
name: gfs-client
spec:
hostNetwork: true
containers:
- name: ubuntu
securityContext:
privileged: true
image: ubuntu:bionic
args:
- /bin/bash
- "-c"
- |
chroot /host /bin/bash -u <<'EOF'
export GLUSTERFS_PACKAGE=glusterfs-client
apt list --installed | grep ${GLUSTERFS_PACKAGE} > /dev/null
case $? in
0)
echo "nothing to do: glusterfs package is already installed"
;;
1)
echo "installing glusterfs package"
apt update && apt install -y ${GLUSTERFS_PACKAGE}
;;
*)
echo "failed to check if glusterfs is installed" >&2
exit 1
;;
esac
sleep infinity
EOF
volumeMounts:
- name: host
mountPath: /host
- name: docker
mountPath: /var/run/docker.sock
priorityClassName: system-cluster-critical
volumes:
- name: host
hostPath:
path: /
- name: docker
hostPath:
path: /var/run/docker.sock
type: SocketA very nice solution! In the spirit of sharing, I've updated the installation script above ☝️ to install a more recent version of glusterfs-client. At the time of writing, the master nodes are based on Debian Buster which are shipped with version 5.5-3. Due to some strange and persistent issues with missing files / non-empty directories showing as empty on the gluster volumes, I had to update both the server and the client to 8.3-1 from upstream. Here's my code:
image: ubuntu:groovy
args:
- /bin/bash
- "-c"
- |
chroot /host /bin/bash -u <<'EOF'
export GLUSTERFS_PACKAGE=glusterfs-client
apt show ${GLUSTERFS_PACKAGE}
apt list --installed | grep ${GLUSTERFS_PACKAGE} > /dev/null
case $? in
0)
echo "nothing to do: glusterfs package is already installed"
;;
1)
echo "installing glusterfs package"
wget -O - https://download.gluster.org/pub/gluster/glusterfs/7/rsa.pub | apt-key add -
DEBID=$(grep 'VERSION_ID=' /etc/os-release | cut -d '=' -f 2 | tr -d '"')
DEBVER=$(grep 'VERSION=' /etc/os-release | grep -Eo '[a-z]+')
DEBARCH=$(dpkg --print-architecture)
echo deb https://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/${DEBID}/${DEBARCH}/apt ${DEBVER} main > /etc/apt/sources.list.d/gluster.list
apt update && apt install -y ${GLUSTERFS_PACKAGE}
;;
*)
echo "failed to check if glusterfs is installed" >&2
exit 1
;;
esac
sleep infinity
EOF
I'll be closing out this ticket given a workaround is possible. I could imagine a future feature where customers can choose from a limited list of packages / node properties to enable on bootstrap. This needs broader discussion, however, and consideration for how DOKS worker node images may possibly evolve. We'll continue to track this possibility internally.