OCP4 on VMware vSphere UPI Automation
The goal of this repo is to make deploying and redeploying a new OpenShift v4 cluster a snap. Using the same repo and with minor tweaks, it can be applied to any version of OpenShift higher than the current version of 4.4.
As it stands right now, the repo works for several installation use cases:
- DHCP with OVA template
- DHCP with PXE boot (needs helper node)
- Static IPs for nodes (lack of isolated network to let helper run DHCP server)
- w/o Cluster-wide Proxy (HTTP and SSL/TLS with certs supported)
- Restricted network (with or without DHCP)
- No Cloud Provider (Useful for mixed clusters with both virtual and physical Nodes)
This repo is most ideal for Home Lab and Proof-of-Concept scenarios. Having said that, if prerequisites (below) can be met and if the vCenter service account can be locked down to access only certain resources and perform only certain actions, the same repo can then be used for DEV or higher environments. Refer to this link for more details on required permissions for a vCenter service account.
Quickstart
This is a concise summary of everything you need to do to use the repo. Rest of the document goes into details of every step.
- Setup helper node or ensure appropriate services (DNS/DHCP/LB/etc.) are available and properly referenced.
- Edit
group_vars/all.yml
, the following must be changed while the rest can remain the same- pull secret
- ip and mac addresses, host/domain names
- enable/disable fips mode
- networkType: OpenShiftSDN (default), OVNKubernetes (uncomment this setting to use)
- isolationMode: NetworkPolicy (default), Multitenant, Subnet (uncomment and set to use Multitenant or Subnet)
- master_schedulable: false (default), set to true to install 3-node cluster
- set
worker_vms
to empty list (i.e.[]
) while removing or commenting out list items for worker node details
- set
- vcenter details
- datastore name
- datacenter name
- username and passwords of admin/service accounts
- validate current Govc version is set
- enable/disable registry/proxy/ntp with their details, as required
- Customize
ansible.cfg
and use/copy/modifystaging
inventory file as required - Run one of the several install options
Infrastructure Prerequisites
- vSphere ESXi and vCenter 6.7 installed. For vCenter 6.5 please see cautionary note below:
- A datacenter created with a vSphere host added to it, a datastore exists and has adequate capacity
- The playbook(s) assumes you are running a helper node in the same network to provide all the necessary services such as [DHCP/DNS/HAProxy as LB]. Also, the MAC addresses for the machines should match between helper repo and this. If not using the helper node, the minimum expectation is that the webserver and tftp server (for PXE boot) are running on the same external host, which we will then treat as a helper node.
- The necessary services such as [DNS/LB(Load Balancer] must be up and running before this repo can be used
- Ansible (preferably latest) on the machine where this repo is cloned.
- Before you install Ansible, install the
epel-release
, runyum -y install epel-release
- Before you install Ansible, install the
For vSphere 6.5, the files relating to interaction with VMware/vCenter such as this may need to have
vmware_deploy_ovf
module to includecluster
,resource-pool
parameters and their values set to work correctly.
Installation Steps
Set Global Variables
Pre-populated entries in group_vars/all.yml are ready to be used unless you need to customize further. Any updates described below refer to group_vars/all.yml unless otherwise specified.
- Get the pull secret from here. Save the pull secret as a file called
pullsecret
in the home directory of the user running this automation. Alternatively, modifypull_secret
with the contents of the downloaded pull secret. - Get the vCenter details:
- IP address
- Service account username (can be the same as admin)
- Service account password (can be the same as admin)
- Admin account username
- Admin account password
- Datacenter name (created in the prerequisites mentioned above)
- Datastore name
- Absolute path of the vCenter folder to use (optional). If this field is not populated, its is auto-populated and points to
/${vcenter.datacenter}/vm/${config.cluster_name}
- Specify hardware version for VM compatibility. Defaults to 15.
- Downloadable link to
govc
(vSphere CLI, pre-populated) - OpenShift cluster
- base domain (pre-populated with example.com)
- cluster name (pre-populated with ocp4)
- HTTP URL of the bootstrap.ign file (pre-populated with an example config pointing to helper node)
- For
bootstrap_vms
,master_vms
, andworker_vms
, when using static_ips, you can delete macaddr from the dictionary. VMware will auto-generate your MAC addresses. If you are using DHCP, defining macaddr will allow you to reserve the specified IP addresses on your DHCP server to ensure the OpenShift nodes always get the same IP address. - For network_modifications: Network CIDRs default to sensible ranges. If a conflict is present (these ranges of addresses are assigned elsewhere in the organization), you may select other non-conflicting CIDR ranges by changing "enabled: false" to "enabled: true" and entering the new ranges. The ranges shown in the repository are the ones that are used by default, even if "enabled: false" is left as it is. The machine network is the network on which the VMs are created. Be sure to specify the right machine network if you set enabled: true
- Furnish any proxy details with the section like below.
- If
proxy.enabled
is set toFalse
anything defined under proxy and the proxy setup is ignored - The
cert_content
shown below is only for illustration to show the format - When there is no certificate, leave the variable
cert_content
value empty
proxy: enabled: true http_proxy: http://helper.ocp4.example.com:3129 https_proxy: http://helper.ocp4.example.com:3129 no_proxy: example.com cert_content: | -----BEGIN CERTIFICATE----- <certficate content> -----END CERTIFICATE-----
- If
- When doing the restricted network install and following instructions from restricted.md, furnish details related to the registry with a section like below. If
registry.enabled
is set toFalse
anything defined underregistry
and the registry setup is ignoredregistry: enabled: true product_repo: openshift-release-dev product_release_name: ocp-release product_release_version: 4.4.0-x86_64 username: ansible password: ansible email: user@awesome.org cert_content: host: helper.ocp4.example.com port: 5000 repo: ocp4/openshift4
- NEW: disconnected/restricted install for existing registry
registry:
enabled: true
host: registry.example.com (if not 443, include :<port> here)
repo_release: openshift/release (the actual OpenShift container images)
repo_images: openshift/release-images (the OpenShift release images, i.e. 4.y.z-x86_64)
- If you wish to install without enabling the Kubernetes vSphere Cloud Provider (Useful for mixed installs with both Virtual Nodes and Bare Metal Nodes), change the
provider:
tonone
in all.yaml.
config:
provider: none
base_domain: example.com
...
- If you wish to enable custom NTP servers on your nodes, set
ntp.custom
toTrue
and definentp.ntp_server_list
to fit your requirements.
ntp:
custom: True
ntp_server_list:
- 0.rhel.pool.ntp.org
- 1.rhel.pool.ntp.org
- Network Policy is enabled by default. To use Multitenant or Subnet, change isolationMode
isolationMode: Multitenant
Step #5 needn't exist at the time of running the setup/installation step, so provide an accurate guess of where and at what context path bootstrap.ign will eventually be served
Set Ansible Inventory and Configuration
Now configure ansible.cfg
and staging
inventory file based on your environment before picking one of the 5 different install options listed below.
staging
inventory file
Update the Under the webservers.hosts
entry, use one of two options below :
- localhost : if the
ansible-playbook
is being run on the same host as the webserver that would eventually host bootstrap.ign file - the IP address or FQDN of the machine that would run the webserver.
ansible.cfg
based on your needs
Update the - Running the playbook as a root user
- If the localhost runs the webserver
[defaults] host_key_checking = False
- If the remote host runs the webserver
[defaults] host_key_checking = False remote_user = root ask_pass = True
- If the localhost runs the webserver
- Running the playbook as a non-root user
- If the localhost runs the webserver
[defaults] host_key_checking = False [privilege_escalation] become_ask_pass = True
- If the remote host runs the webserver
[defaults] host_key_checking = False remote_user = root ask_pass = True [privilege_escalation] become_ask_pass = True
- If the localhost runs the webserver
Run Installation Playbook
# Option 1: DHCP + use of OVA template
ansible-playbook -i staging dhcp_ova.yml
# Option 2: DHCP + PXE boot
ansible-playbook -i staging dhcp_pxe.yml
# Option 3: ISO + Static IPs
ansible-playbook -i staging static_ips.yml
# Refer to restricted.md file for more details
# Option 4: DHCP + use of OVA template in a Restricted Network
ansible-playbook -i staging restricted_dhcp_ova.yml
# Option 5: Static IPs + use of ISO images in a Restricted Network
ansible-playbook -i staging restricted_static_ips.yml
# Option 6: Static IPs + use of OVA template
# Note: OpenShift 4.6 or higher required
ansible-playbook -i staging static_ips_ova.yml
# Option 7: Static IPs + use of OVA template in a Restricted Network
# Note: OpenShift 4.6 or higher required
ansible-playbook -i staging restricted_static_ips_ova.yml
Miscellaneous
-
If you are re-running the installation playbook make sure to blow away any existing VMs (in
ocp4
folder) listed below:- bootstrap
- masters
- workers
rhcos-vmware
template (if not using the extra param as shown below)
-
If a template by the name
rhcos-vmware
already exists in vCenter, you want to reuse it and skip the OVA download from Red Hat and upload into vCenter, use the following extra param.-e skip_ova=true
-
If you would rather want to clean all folders
bin
,downloads
,install-dir
and re-download all the artifacts, append the following to the command you chose in the first step-e clean=true
Expected Outcome
- Necessary Linux packages installed for the installation. NOTE: support for Mac client to run this automation has been added but is not guaranteed to be complete
- SSH key-pair generated, with key
~/.ssh/ocp4
and public key~/.ssh/ocp4.pub
- Necessary folders [bin, downloads, downloads/ISOs, install-dir] created
- OpenShift client, install and .ova binaries downloaded to the downloads folder
- Unzipped versions of the binaries installed in the bin folder
- In the install-dir folder:
- append-bootstrap.ign file with the HTTP URL of the boostrap.ign file
- master.ign and worker.ign
- base64 encoded files (append-bootstrap.64, master.64, worker.64) for (append-bootstrap.ign, master.ign, worker.ign) respectively. This step assumes you have base64 installed and in your $PATH
- The bootstrap.ign is copied over to the web server in the designated location
- A folder is created in the vCenter under the mentioned datacenter and the template is imported
- The template file is edited to carry certain default settings and runtime parameters common to all the VMs
- VMs (bootstrap, master0-2, worker0-2) are generated in the designated folder and (in state of) poweredon
Final Check:
If everything goes well you should be able to log into all of the machines using the following command:
# Assuming you are able to resolve bootstrap.ocp4.example.com on this machine
# Replace the bootstrap hostname with any of the master or worker hostnames
ssh -i ~/.ssh/ocp4 core@bootstrap.ocp4.example.com
Once logged in, on bootstrap node run the following command to understand if/how the masters are (being) setup:
journalctl -b -f -u bootkube.service
Once the bootkube.service
is complete, the bootstrap VM can safely be poweredoff
and the VM deleted. Finish by checking on the OpenShift with the following commands:
# In the root folder of this repo run the following commands
export KUBECONFIG=$(pwd)/install-dir/auth/kubeconfig
export PATH=$(pwd)/bin:$PATH
# OpenShift Client Commands
oc whoami
oc get co
Debugging
To check if the proxy information has been picked up:
# On Master
cat /etc/systemd/system/machine-config-daemon-host.service.d/10-default-env.conf
# On Bootstrap
cat /etc/systemd/system.conf.d/10-default-env.conf
To check if the registry information has been picked up:
# On Master or Bootstrap
cat /etc/containers/registries.conf
cat /root/.docker/config.json
To check if your certs have been picked up:
# On Master
cat /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
# On Bootstrap
cat /etc/pki/ca-trust/source/anchors/ca.crt