This tutorial can be used to deploy Rook+Ceph on OpenShift 4.1. Assumes installation on AWS but these instructions have also been tested with RHV and VMware based OCP deployments.
$ git clone https://github.com/rook/rook.gitWe will use the Rook operator to deploy Ceph. The following yaml files are used to create CRDs, RoleBindings and other objects needed to support Rook and the Ceph cluster.
$ cd cluster/examples/kubernetes/ceph
$ oc create -f common.yamlBefore proceeding we need to create EBS volumes for Ceph to use. These volumes will be presented as raw block devices to the nodes. My reference cluster is deployed on AWS and uses 3x m5.2xlarge worker nodes. I added 1x400 EBS volume to each node. For other platforms, just ensure raw (unformatted/unpartitioned) devices are presented to the nodes you want to deploy Ceph on. If you opt for the default installation, discovery pods will be deployed to automatically enumerate available storage.
Next, create the operator (verify it is running by executing oc get pods -n rook-ceph | grep operator). Then create the cluster by sourcing cluster.yaml.
$ oc create -f operator-openshift.yaml
$ oc create -f cluster.yamlIf you get a CephCluster error, build with kubetctl manually with command in UI.
Verify the deployment of the cluster by running oc get pods -n rook-ceph as shown below.
$ oc get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-24ghk 2/2 Running 0 17m
csi-cephfsplugin-gs42d 2/2 Running 0 17m
csi-cephfsplugin-lmjss 2/2 Running 0 17m
csi-cephfsplugin-provisioner-0 2/2 Running 0 17m
csi-rbdplugin-fsf6l 2/2 Running 0 17m
csi-rbdplugin-jl84w 2/2 Running 0 17m
csi-rbdplugin-mt9xc 2/2 Running 0 17m
csi-rbdplugin-provisioner-0 4/4 Running 0 17m
rook-ceph-agent-84g4n 1/1 Running 0 17m
rook-ceph-agent-gmjq6 1/1 Running 0 17m
rook-ceph-agent-hljgk 1/1 Running 0 17m
rook-ceph-mgr-a-649ccccddb-494r5 1/1 Running 0 3m21s
rook-ceph-mon-a-d746fd87b-phr7w 1/1 Running 0 4m20s
rook-ceph-mon-b-7f959f6fc7-445g9 1/1 Running 0 4m7s
rook-ceph-mon-c-57f698c8b4-4ntp8 1/1 Running 0 3m46s
rook-ceph-operator-fbdb6784c-zcns5 1/1 Running 0 17m
rook-ceph-osd-0-f8c5bb5d9-xsjkq 1/1 Running 0 2m19s
rook-ceph-osd-1-78d4875965-76khn 1/1 Running 0 2m19s
rook-ceph-osd-2-55cd9ffc5c-d9d74 1/1 Running 0 2m19s
rook-ceph-osd-prepare-ip-10-0-204-150-6fjfj 0/2 Completed 0 2m54s
rook-ceph-osd-prepare-ip-10-0-206-173-w8qzk 0/2 Completed 0 2m54s
rook-ceph-osd-prepare-ip-10-0-215-44-sf8t6 0/2 Completed 0 2m54s
rook-discover-fp68m 1/1 Running 0 17m
rook-discover-h49qs 1/1 Running 0 17m
rook-discover-wr69x 1/1 Running 0 17mNext we create a CephFilesystem object using the Rook operator. We also create a debug pod to interface with the deployed Ceph cluster.
$ oc create -f filesystem.yaml
$ oc create -f toolbox.yamlExecute a shell on the debug pod to get the status of the Ceph cluster.
$ oc -n rook-ceph exec -it $(oc -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bashThen run ceph status and observe the output.
# ceph status
cluster:
id: d1a84232-f042-4e50-8541-051456744d7a
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 9m)
mgr: a(active, since 9m)
mds: myfs:1 {0=myfs-b=up:active} 1 up:standby-replay
osd: 3 osds: 3 up (since 8m), 3 in (since 8m)
data:
pools: 2 pools, 200 pgs
objects: 22 objects, 2.2 KiB
usage: 3.0 GiB used, 1.2 TiB / 1.2 TiB avail
pgs: 200 active+clean
io:
client: 853 B/s rd, 1 op/s rd, 0 op/s wrFinally we need to create the StorageClass objects for CephFS and RBD.
$ oc create -f csi/cephfs/storageclass.yaml$ oc create -f csi/rbd/storageclass.yamlWhen you are finished verify the StorageClass objects were created as shown:
$ oc get sc
NAME PROVISIONER AGE
csi-cephfs cephfs.csi.ceph.com 8m36s
rook-ceph-block rbd.csi.ceph.com 9m29s
...
To verify basic functionality, create a project to test the create of some PVCs.
$ oc new-project csi-test$ cd rook/cluster/examples/kubernetes/ceph
$ oc create -f csi/cephfs/pvc.yaml -n csi-test
$ oc create -f csi/cephfs/pod.yaml -n csi-testVerify the CephFS PVC and test pod as follows:
$ oc get pvc cephfs-pvc -n csi-test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-pvc Bound pvc-0ce77415-b3aa-11e9-9478-123b081d1798 1Gi RWO csi-cephfs 12s$ oc get pod csicephfs-demo-pod -n csi-test
NAME READY STATUS RESTARTS AGE
csicephfs-demo-pod 1/1 Running 0 26sFirst, modify the contents of csi/cephfs/pvc.yaml and change the storageClassName from rook-csi to rook-ceph-block. Then run the following commands:
$ cd rook/cluster/examples/kubernetes/ceph
$ oc create -f csi/rbd/pvc.yaml -n csi-test
$ oc create -f csi/rbd/pod.yamlVerify the RBD PVC and test pod as follows:
$ oc get pvc rbd-pvc -n csi-test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rbd-pvc Bound pvc-a53e879c-b3aa-11e9-9478-123b081d1798 1Gi RWO csi-rbd 19s$ oc get pods csirbd-demo-pod -n csi-test
NAME READY STATUS RESTARTS AGE
csirbd-demo-pod 1/1 Running 0 66sAs an extra level of validation we can deploy the OpenShift EFK stack and configure Elasticsearch PVCs to use the csi-rbd storage class.
To do this, follow the instructions referenced in the beginning of the document. When you get to step five (5. Create a cluster logging instance:) use the following yaml:
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
name: "instance"
namespace: "openshift-logging"
spec:
managementState: "Managed"
logStore:
type: "elasticsearch"
elasticsearch:
nodeCount: 3
storage:
# storage class is modified here
storageClassName: rook-ceph-block
size: 100G
redundancyPolicy: "SingleRedundancy"
visualization:
type: "kibana"
kibana:
replicas: 1
curation:
type: "curator"
curator:
schedule: "30 3 * * *"
collection:
logs:
type: "fluentd"
fluentd: {}Continue with the official documentation to complete the install. After install, verify the Elasticsearch pods are using storage provided by the csi-rbd storage class:
$ oc get pvc -n openshift-logging
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
elasticsearch-elasticsearch-cdm-b68hqpif-1 Bound pvc-3829d7b7-b3a1-11e9-84c0-0206e4668600 94Gi RWO csi-rbd 3h9m
elasticsearch-elasticsearch-cdm-b68hqpif-2 Bound pvc-3847ea96-b3a1-11e9-84c0-0206e4668600 94Gi RWO csi-rbd 3h9m
elasticsearch-elasticsearch-cdm-b68hqpif-3 Bound pvc-386668ad-b3a1-11e9-84c0-0206e4668600 94Gi RWO csi-rbd 3h9mapiVersion: v1
baseDomain: redhat-demo.com
controlPlane:
hyperthreading: Enabled
name: master
platform:
aws:
zones:
- us-east-1a
- us-east-1b
- us-east-1c
type: m5.xlarge
replicas: 3
compute:
- hyperthreading: Enabled
name: worker
platform:
aws:
type: m5.2xlarge
zones:
- us-east-1d
# WTF us-east-1e doesn't support m5.2xlarge?
# - us-east-1e
- us-east-1f
replicas: 3
metadata:
name: ckeller-aws
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineCIDR: 10.0.0.0/16
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
region: us-east-1
userTags:
adminContact: ckeller@redhat.com
pullSecret: ‘’
sshKey: ‘’