Openshift 4

Deploy an Openshift 4 cluster using UPI method (User Provisioned Infrastructure) in a disconnected scenario without a cloud provider to simulate a real baremetal environment.

The cluster is deployed on top of libvirt/kvm using Terraform and every step of the installation and configuration is automated to be able to destroy and deploy a new cluster in less than 30 minutes.

Red Hat Quay has been chosen as the registry to store the cluster images.

Warning

This is a demo environment and it should never be used in production.

Requirements

The requirements.sh script will install all needed requirements:

make require

QEMU

The provider does not currently support to create volumes with different mode than root:root so QEMU agent must run as priviledged:

sed -i '/^#user/s/^#//' /etc/libvirt/qemu.conf
sed -i '/^#group/s/^#//' /etc/libvirt/qemu.conf

Restart libvirt daemon:

systemctl restart libvirtd

Pull secret

Set the current environment:

export OCP_ENVIRONMENT="lab"

Get your pull secret from cloud.redhat.com and copy it to the src/openshift-install/${OCP_ENVIRONMENT} folder (create the folder for your environment if it does not exist):

src/openshift-install
 └── ${OCP_ENVIRONMENT}
     └── pull-secret.json

Install Openshift 4

Deploy Openshift 4 cluster:

make

Mirror registry (disconnected)

Mirror the images used during the installation to the external registry:

./output/openshift-install/${OCP_ENVIRONMENT}/mirror-release-image.sh

Go to the Quay GUI to verify that the images have been mirrored (e.g. https://registry.ocp.bmlab.int:5443).

Bootstrap

Power on bootstrap node:

virsh start ocp-bootstrap # --console to get logs

Control plane

Wait until bootstrap is ready to power on master nodes:

for i in $(seq 0 2); do virsh start "ocp-master0$i"; done

Wait until the master nodes are up and running:

./openshift-install wait-for bootstrap-complete \
    --dir output/openshift-install/${OCP_ENVIRONMENT}

Finally, shutdown the bootstrap node:

virsh shutdown ocp-bootstrap

Worker nodes

Power on worker nodes:

for i in $(seq 0 2); do virsh start "ocp-worker0$i"; done

Approve all pending CSR to allow nodes registration (run the command twice to ensure that all the certificates are signed):

export KUBECONFIG="$(pwd)/output/openshift-install/${OCP_ENVIRONMENT}/auth/kubeconfig"

oc get csr \
    -o go-template='
      {{range .items}}
        {{if not .status}}
          {{.metadata.name}}{{"\n"}}
        {{end}}
      {{end}}' |\
        xargs oc adm certificate approve

Wait until the installation is completed:

./openshift-install wait-for install-complete \
	--dir output/openshift-install/${OCP_ENVIRONMENT}

Verify the Openshift cluster is correctly created:

oc get nodes

Day-2 operations

Once Openshift 4 has been deployed in can be configured with the instructions in day-two folder resources.

Upgrade cluster

Mirror the images for the new version to the external registry:

./output/openshift-install/${OCP_ENVIRONMENT}/mirror-release-image.sh \
    "4.y.z" --apply-release-image-signature

Start the cluster upgrade to the desired version:

./output/openshift-install/${OCP_ENVIRONMENT}/update-cluster.sh "4.y.z"