## Basic settings
- 5 nodes
- Debian 9
- 1 GB RAM and 1 vCPU
- 1 network interface
- Update, install dependencies and GlusterFS (server and client are needed in all nodes)
apt install thin-provisioning-tools glusterfs-server glusterfs-client
- Check glusterfs-server status in all nodes
systemctl status glusterfs-server.service
- Install Heketi on one of the GlusterFS nodes, the project does not provide .deb packages (only rpm) so we need to install via tarball:
- Link: https://github.com/heketi/heketi/releases/
wget https://github.com/heketi/heketi/releases/download/v6.0.0/heketi-v6.0.0.linux.amd64.tar.gz
tar xzvf heketi-v6.0.0.linux.amd64.tar.gz
cd heketi
cp heketi heketi-cli /usr/local/bin/
heketi -v
- We also need to manually create the heketi user and the directory structures for the configuration:
groupadd -r -g 515 heketi
useradd -r -c "Heketi user" -d /var/lib/heketi -s /bin/false -m -u 515 -g heketi heketi
mkdir -p /var/lib/heketi && chown -R heketi:heketi /var/lib/heketi
mkdir -p /var/log/heketi && chown -R heketi:heketi /var/log/heketi
mkdir -p /etc/heketi
-
Now, we need to setup password-less ssh login between gluster nodes to let heketi execute commands as root
-
In the first node where Heketi is installed (glusterfs-micro01), execute the following:
ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
chown heketi:heketi /etc/heketi/heketi_key*
ssh-copy-id -i /etc/heketi/heketi_key.pub root@[glusterfs-micro01|glusterfs-micro02|glusterfs-micro03|glusterfs-micro04|glusterfs-micro05]
-
You need to setup SSH to accept root password for login correctly or paste the pub_key of the source server in the auth_keys file of the destination host.
-
Test SSH access password-less
ssh -i /etc/heketi/heketi_key root@glusterfs-micro01
ssh -i /etc/heketi/heketi_key root@glusterfs-micro02
ssh -i /etc/heketi/heketi_key root@glusterfs-micro03
ssh -i /etc/heketi/heketi_key root@glusterfs-micro04
ssh -i /etc/heketi/heketi_key root@glusterfs-micro05
{
"_port_comment": "Heketi Server Port Number",
"port": "8888",
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": true,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "SUPERPASSWORD"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "PASSWORD"
}
},
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": [
"Execute plugin. Possible choices: mock, ssh",
"mock: This setting is used for testing and development.",
" It will not send commands to any node.",
"ssh: This setting will notify Heketi to ssh to the nodes.",
" It will need the values in sshexec to be configured.",
"kubernetes: Communicate with GlusterFS containers over",
" Kubernetes exec api."
],
"executor": "ssh",
"_sshexec_comment": "SSH username and private key file information",
"sshexec": {
"keyfile": "/etc/heketi/heketi_key",
"user": "root",
"port": "22",
"fstab": "/etc/fstab"
},
"_kubeexec_comment": "Kubernetes configuration",
"kubeexec": {
"host" :"http://K8S.HOST",
"cert" : "PATH.TO.CERT.FILE",
"insecure": false,
"user": "admin",
"password": "SUPERPASSWORD",
"namespace": "k8s namespace",
"fstab": "/etc/fstab"
},
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",
"brick_max_size_gb" : 1024,
"brick_min_size_gb" : 1,
"max_bricks_per_volume" : 33,
"_loglevel_comment": [
"Set log level. Choices are:",
" none, critical, error, warning, info, debug",
"Default is warning"
],
"loglevel" : "debug"
}
}
- Now, lets create a systemctl unit to manage heketi: /etc/systemd/system/heketi.service
[Unit]
Description=Heketi Server
Requires=network-online.target
After=network-online.target
[Service]
Type=simple
User=heketi
Group=heketi
PermissionsStartOnly=true
PIDFile=/run/heketi/heketi.pid
Restart=on-failure
RestartSec=10
WorkingDirectory=/var/lib/heketi
RuntimeDirectory=heketi
RuntimeDirectoryMode=0755
ExecStartPre=[ -f "/run/heketi/heketi.pid" ] && /bin/rm -f /run/heketi/heketi.pid
ExecStart=/usr/local/bin/heketi --config=/etc/heketi/heketi.json
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGINT
TimeoutStopSec=5
[Install]
WantedBy=multi-user.target
- Start the service and check its status and listening port:
systemctl daemon-reload
systemctl start heketi.service
systemctl status heketi.service
systemctl enable heketi.service
netstat -putan | grep 8888
-
Test the service from local node and remote nodes
-
Local node
root@glusterfs-micro01:~|⇒ curl http://glusterfs-micro01:8888/hello
Hello from Heketi#
- Remote node
root@glusterfs-micro03:~|⇒ curl http://glusterfs-micro01:8888/hello
Hello from Heketi#
- And check the auth configured in heketi.json
root@glusterfs-micro01:~|⇒ heketi-cli --server http://glusterfs-micro01:8888 --user admin --secret "SUPERPASSWORD" cluster list
Clusters:
- All working, great.
- We have 5 hosts with glusterfs
root@glusterfs-micro01:~|⇒ host glusterfs-micro01.ironshared.com
glusterfs-micro01.ironshared.com has address 10.200.1.121
root@glusterfs-micro01:~|⇒ host glusterfs-micro02.ironshared.com
glusterfs-micro02.ironshared.com has address 10.200.1.122
root@glusterfs-micro01:~|⇒ host glusterfs-micro03.ironshared.com
glusterfs-micro03.ironshared.com has address 10.200.1.123
root@glusterfs-micro01:~|⇒ host glusterfs-micro04.ironshared.com
glusterfs-micro04.ironshared.com has address 10.200.1.124
root@glusterfs-micro01:~|⇒ host glusterfs-micro05.ironshared.com
glusterfs-micro05.ironshared.com has address 10.200.1.125
- Create topology.json FILE
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"10.200.1.121"
],
"storage": [
"10.200.1.121"
]
},
"zone": 1
},
"devices": [
"/dev/vdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"10.200.1.122"
],
"storage": [
"10.200.1.122"
]
},
"zone": 1
},
"devices": [
"/dev/vdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"10.200.1.123"
],
"storage": [
"10.200.1.123"
]
},
"zone": 1
},
"devices": [
"/dev/vdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"10.200.1.124"
],
"storage": [
"10.200.1.124"
]
},
"zone": 1
},
"devices": [
"/dev/vdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"10.200.1.125"
],
"storage": [
"10.200.1.125"
]
},
"zone": 1
},
"devices": [
"/dev/vdb"
]
}
]
}
]
}
-
Where /dev/vdb must be a RAW block device attached to each gluster node in topology.json file
-
Now we have to load the FILE
export HEKETI_CLI_SERVER=http://glusterfs-micro01:88888
export HEKETI_CLI_USER=admin
export HEKETI_CLI_KEY=SUPERPASSWORD
heketi-cli topology load --json=/opt/heketi/topology.json
Creating cluster ... ID: 87fcab5a999fd7a27d84444cea12e765
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node 10.200.1.121 ... Unable to create node: New Node doesn't have glusterd running
Creating node 10.200.1.122 ... Unable to create node: New Node doesn't have glusterd running
Creating node 10.200.1.123 ... Unable to create node: New Node doesn't have glusterd running
Creating node 10.200.1.124 ... Unable to create node: New Node doesn't have glusterd running
Creating node 10.200.1.125 ... Unable to create node: New Node doesn't have glusterd running
- And then it fails. :(
- Then we forget about env variables and try to do it in one liner style
heketi-cli --server http://glusterfs-micro01:8888 --user admin --secret "SUPERPASSWORD" topology load --json=/etc/heketi/topology.json
-
But fails anyway. Then check the following: dns resolution, chage hostnames in topology.json to IPs, ssh access, gluster peer probe "IP" test, netstat looking for glusterd... all is ok.
-
To continue...