HashiCorp Packer and VMware vSphere to Build Private Cloud Machine Images

Introduction
Requirements
Configuration
Build
Troubleshoot
Credits

Introduction

This repository provides infrastructure-as-code examples to automate the creation of virtual machine images and their guest operating systems on VMware vSphere using HashiCorp Packer and the Packer Plugin for VMware vSphere (vsphere-iso). All examples are authored in the HashiCorp Configuration Language ("HCL2").

Use of this repository is mentioned in the VMware Validated Solution: Private Cloud Automation for VMware Cloud Foundation authored by the maintainer. Learn more about this solution at vmware.com/go/vvs.

By default, the machine image artifacts are transferred to a vSphere Content Library as an OVF template and the temporary machine image is destroyed. If an item of the same name exists in the target content library, Packer will update the existing item with the new OVF template. This method is extremely useful for vRealize Automation as image mappings do not need to be updated when a virtual machine image update is executed and finalized.

The following builds are available:

Linux Distributions

VMware Photon OS 4
Ubuntu Server 20.04 LTS
Ubuntu Server 18.04 LTS
Red Hat Enterprise Linux 8 Server
Red Hat Enterprise Linux 7 Server
AlmaLinux 8
Rocky Linux 8
CentOS Stream 8
CentOS Linux 8
CentOS Linux 7

Microsoft Windows - Core and Desktop Experience

Microsoft Windows Server 2022 - Standard and Datacenter
Microsoft Windows Server 2019 - Standard and Datacenter
Microsoft Windows Server 2016 - Standard and Datacenter
Microsoft Windows 11 Professional (Experimental)
Microsoft Windows 10 Professional

NOTE: Guest customization is not supported for AlmaLinux and Rocky Linux in vCenter Server 7.0 Update 2.

Requirements

Packer:

HashiCorp Packer 1.7.7 or higher.
HashiCorp Packer Plugin for VMware vSphere (vsphere-iso) 1.0.2 or higher.
Packer Plugin for Windows Updates 0.14.0 or higher - a community plugin for HashiCorp Packer.

Required plugins are automatically downloaded and initialized when using ./build.sh. For dark sites, you may download the plugins and place these same directory as your Packer executable /usr/local/bin or $HOME/.packer.d/plugins.

Operating Systems:

Ubuntu Server 20.04 LTS
macOS Big Sur (Intel)

Operating systems and versions tested with the repository examples.

Additional Software Packages:

The following software packages must be installed on the Packer host:

Git command line tools.
- Ubuntu: apt-get install git
- macOS: brew install git
Ansible 2.9 or higher.
- Ubuntu: apt-get install ansible
- macOS: brew install ansible
A command-line .iso creator. Packer will use one of the following:
- xorriso on Ubuntu: apt-get install xorriso
- mkisofs on Ubuntu: apt-get install mkisofs
- hdiutil on macOS: native
mkpasswd
- Ubuntu: apt-get install whois
- macOS: brew install --cask docker
Coreutils
- macOS: brew install coreutils
HashiCorp Terraform 1.0.10 or higher.
- Ubuntu:
  - sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
  - curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
  - sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
  - sudo apt-get update && sudo apt-get install terraform
- macOS:
  - brew tap hashicorp/tap
  - brew install hashicorp/tap/terraform

Platform:

VMware Cloud Foundation 4.2 or higher, or
VMware vSphere 7.0 Update 2 or higher

Configuration

Step 1 - Download the Release.

Download the latest release.

You may also clone main for the latest pre-release updates.

Example:

git clone https://github.com/vmware-samples/packer-examples-for-vsphere.git

The directory structure of the repository.

├── build.sh
├── config.sh
├── LICENSE
├── NOTICE
├── README.md
├── ansible
│   ├── roles
│   │   └── <role>
│   │       ├── defaults
│   │       │   └── main.yml
│   │       ├── files
│   │       │   └── root-ca.cer.example
│   │       ├── handlers
│   │       │   └── main.yml
│   │       ├── meta
│   │       │   └── main.yml
│   │       ├── tasks
│   │       │   └── main.yml
│   │       │   └── *.yml
│   │       └── vars
│   │           └── main.yml
│   ├── ansible.cfg
│   └── main.yml
├── builds
│   ├── ansible.pkvars.hcl.example
│   ├── build.pkvars.hcl.example
│   ├── common.pkvars.hcl.example
│   ├── proxy.pkvars.hcl.example
│   ├── rhsm.pkvars.hcl.example
│   ├── vsphere.pkvars.hcl.example
│   ├── linux
│   │   └── <distribution-version>
│   │       ├── *.pkr.hcl
│   │       ├── *.auto.pkrvars.hcl
│   │       └── data
│   │           └── ks.pkrtpl.hcl
│   └── windows
│       └── <version>
│           ├── *.pkr.hcl
│           ├── *.auto.pkrvars.hcl
│           └── data
│               └── autounattend.pkrtpl.hcl
├── certificates
│   └── root-ca.cer.example
├── manifests
├── scripts
│   ├── linux
│   │   └── *.sh
│   └── windows
│       └── *.ps1
└── terraform
    │── vsphere-role
    └── vsphere-virtual-machine

The files are distributed in the following directories.

ansible - contains the Ansible roles to initialize and prepare the machine image build.
builds - contains the templates, variables, and configuration files for the machine image build.
scripts - contains the scripts to initialize and prepare the machine image build.
certificates - contains the Trusted Root Authority certificates for Windows build.
manifests - manifests created after the completion of the machine image build.

NOTE: The project is transitioning to use Ansible instead of scripts, where possible.

Step 2 - Download the Guest Operating Systems ISOs

Download the x64 guest operating system .iso images.

Linux Distributions
- VMware Photon OS 4 Server
  - Download the 4.0 GA release of the FULL .iso image. (e.g. photon-4.0-ca7c9e933.iso)
- Ubuntu Server 20.04 LTS
  - Download the latest LIVE release .iso image. (e.g. ubuntu-20.04.2-live-server-amd64.iso)
- Ubuntu Server 18.04 LTS
  - Download the latest legacy NON-LIVE release .iso image. (e.g. ubuntu-18.04.6-server-amd64.iso)
- Red Hat Enterprise Linux 8 Server
  - Download the latest release of the FULL .iso image. (e.g. rhel-8-x86_64-dvd1.iso)
- Red Hat Enterprise Linux 7 Server
  - Download the latest release of the FULL .iso image. (e.g. rhel-server-7-x86_64-dvd1.iso)
- AlmaLinux 8
  - Download the latest release of the FULL .iso image. (e.g. AlmaLinux-8-x86_64-dvd1.iso)
- Rocky Linux 8
  - Download the latest release of the FULL .iso image. (e.g. Rocky-8-x86_64-dvd1.iso)
- CentOS Stream 8
  - Download the latest release of the FULL .iso image. (e.g. CentOS-Stream-8-x86_64-dvd1.iso)
- CentOS Linux 8
  - Download the latest release of the FULL .iso image. (e.g. CentOS-8-x86_64-dvd1.iso)
- CentOS Linux 7
  - Download the latest release of the FULL .iso image. (e.g. CentOS-7-x86_64-DVD.iso)
Microsoft Windows
- Microsoft Windows Server 2022
- Microsoft Windows Server 2019
- Microsoft Windows Server 2016
- Microsoft Windows 11 Professional (Experimental)
- Microsoft Windows 10 Professional
Obtain the checksum type (e.g. sha256, md5, etc.) and checksum value for each guest operating system .iso image. This will be use in the build input variables.

Upload your guest operating system .iso images to the ISO datastore and paths that will be used in your variables.

Example: builds/<type>/<build>/*.auto.pkvars.hcl

common_iso_datastore = "sfo-w01-cl01-ds-nfs01"

Example: config/common.pkvars.hcl

iso_path           = "iso/linux/photon"
iso_file           = "photon-4.0-ca7c9e933.iso"
iso_checksum_type  = "md5"
iso_checksum_value = "d8c4bc561e68afaf7815518f78a5b4ab"

Step 3 - Configure Service Account Privileges in vSphere

Create a custom vSphere role with the required privileges to integrate HashiCorp Packer with VMware vSphere. A service account can be added to the role to ensure that Packer has least privilege access to the infrastructure. Clone the default Read-Only vSphere role and add the following privileges:

Category	Privilege	Reference
Content Library	Add library item	`ContentLibrary.AddLibraryItem`
...	Update Library Item	`ContentLibrary.UpdateLibraryItem`
Datastore	Allocate space	`Datastore.AllocateSpace`
...	Browse datastore	`Datastore.Browse`
...	Low level file operations	`Datastore.Browse`
Network	Assign network	`Network.Assign`
Resource	Assign virtual machine to resource pool	`Resource.AssignVMToPool`
vApp	Export	`vApp.Export`
Virtual Machine	Configuration > Add new disk	`VirtualMachine.Config.AddNewDisk`
...	Configuration > Add or remove device	`VirtualMachine.Config.AddRemoveDevice`
...	Configuration > Advanced configuration	`VirtualMachine.Config.AdvancedConfig`
...	Configuration > Change CPU count	`VirtualMachine.Config.CPUCount`
...	Configuration > Change memory	`VirtualMachine.Config.Memory`
...	Configuration > Change settings	`VirtualMachine.Config.Settings`
...	Configuration > Change Resource	`VirtualMachine.Config.Resource`
...	Configuration > Set annotation	`VirtualMachine.Config.Annotation`
...	Edit Inventory > Create from existing	`VirtualMachine.Inventory.CreateFromExisting`
...	Edit Inventory > Create new	`VirtualMachine.Inventory.Create`
...	Edit Inventory > Remove	`VirtualMachine.Inventory.Delete`
...	Interaction > Configure CD media	`VirtualMachine.Interact.SetCDMedia`
...	Interaction > Configure floppy media	`VirtualMachine.Interact.SetFloppyMedia`
...	Interaction > Connect devices	`VirtualMachine.Interact.DeviceConnection`
...	Interaction > Inject USB HID scan codes	`VirtualMachine.Interact.PutUsbScanCodes`
...	Interaction > Power off	`VirtualMachine.Interact.PowerOff`
...	Interaction > Power on	`VirtualMachine.Interact.PowerOn`
...	Provisioning > Create template from virtual machine	`VirtualMachine.Provisioning.CreateTemplateFromVM`
...	Provisioning > Mark as template	`VirtualMachine.Provisioning.MarkAsTemplate`
...	Provisioning > Mark as virtual machine	`VirtualMachine.Provisioning.MarkAsVM`
...	State > Create snapshot	`VirtualMachine.State.CreateSnapshot`

If you'd like to automate the creation of the custom vSphere role, a Terraform example is included in the project.

Navigate to the directory for the example.

cd terraform/vsphere-role

Duplicate the terraform.tfvars.example file to terraform.tfvars in the directory.

cp terraform.tfvars.example terraform.tfvars

Open the terraform.tfvars file and update the variables according to your environment.
Initialize the current directory and the required Terraform provider for VMware vSphere.

terraform init

Create a Terraform plan and save the output to a file.

terraform plan -out=tfplan

Apply the Terraform plan.

terraform apply tfplan

Once the custom vSphere role is created, assign Global Permissions in vSphere for the service account used for the HashiCorp Packer to VMware vSphere integration. Global permissions are required for the content library. For example:

Log in to the vCenter Server at https://<management_vcenter_server_fqdn>/ui as administrator@vsphere.local.
Select Menu > Administration.
In the left pane, select Access control > Global permissions and click the Add permissions icon.
In the Add permissions dialog box, enter the service account (e.g. svc-packer-vsphere@rainpole.io), select the custom role (e.g. Packer to vSphere Integration Role) and the Propagate to children check box, and click OK.

In an environment with many vCenter Server instances, such as management and workload domains, you may wish to further reduce the scope of access across the infrastructure in vSphere for the service account. For example, if you do not want Packer to have access to your management domain, but only allow access to workload domains:

From the Hosts and clusters inventory, select management domain vCenter Server to restrict scope, and click the Permissions tab.
Select the service account with the custom role assigned and click the Change role icon.
In the Change role dialog box, from the Role drop-down menu, select No Access, select the Propagate to children check box, and click OK.

Step 4 - Configure the Variables

The variables are defined in .pkvars.hcl files.

Copy the Example Variables

Run the config script ./config.sh to copy the .pkvars.hcl.example files to the config directory.

The config folder is the default folder, You may override the default by passing an alternate value as the first argument.

./config.sh foo
./build.sh foo

For example, this is useful for the purposes of running machine image builds for different environment.

San Francisco: us-west-1

./config.sh config/us-west-1
./build.sh config/us-west-1

Los Angeles: us-west-2

./config.sh config/us-west-2
./build.sh config/us-west-2

Build Variables

Edit the config/build.pkvars.hcl file to configure the following:

Credentials for the default account on machine images.

Example: config/build.pkvars.hcl

build_username           = "rainpole"
build_password           = "<plaintext_password>"
build_password_encrypted = "<sha512_encrypted_password>"
build_key                = "<public_key>"

You can also override the build_key value with contents of a file, if required.

For example:

build_key = file("${path.root}/config/ssh/build_id_ecdsa.pub")

Generate a SHA-512 encrypted password for the build_password_encrypted using tools like mkpasswd.

Example: mkpasswd using Docker on macOS:

rainpole@macos>  docker run -it --rm alpine:latestvmwar mkpasswd -m sha512
Password: ***************
[password hash]

Example: mkpasswd on Linux:

rainpole@linux>  mkpasswd -m sha-512
Password: ***************
[password hash]

Generate a public key for the build_password_encrypted for public key authentication.

Example: macOS and Linux.

rainpole@macos> cd .ssh/
rainpole@macos ~/.ssh> ssh-keygen -t ecdsa -b 521 -C "code@rainpole.io"
Generating public/private ecdsa key pair.
Enter file in which to save the key (/Users/rainpole/.ssh/id_ecdsa):
Enter passphrase (empty for no passphrase): **************
Enter same passphrase again: **************
Your identification has been saved in /Users/rainpole/.ssh/id_ecdsa.
Your public key has been saved in /Users/rainpole/.ssh/id_ecdsa.pub.

The content of the public key, build_key, is added the key to the .ssh/authorized_keys file of the build_username on the guest operating system.

WARNING: Replace the default public keys and passwords. By default, both Public Key Authentication and Password Authentication are enabled for Linux distributions. If you wish to disable Password Authentication and only use Public Key Authentication, comment or remove the portion of the associated script in the scripts directory.

Ansible Variables

Edit the config/ansible.pkvars.hcl file to configure the following:

Credentials for the Ansible account on Linux machine images.

Example: config/ansible.pkvars.hcl

ansible_username = "ansible"
ansible_key      = "<public_key>"

NOTE: A random password is generated for the Ansible user.

You can also override the ansible_key value with contents of a file, if required.

For example:

ansible_key = file("${path.root}/config/ssh/ansible_id_ecdsa.pub")

Common Variables

Edit the config/common.pkvars.hcl file to configure the following common variables:

Virtual Machine Settings
Template and Content Library Settings
Removable Media Settings
Boot and Provisioning Settings

Example: config/common.pkvars.hcl

// Virtual Machine Settings
common_vm_version           = 19
common_tools_upgrade_policy = true
common_remove_cdrom         = true

// Template and Content Library Settings
common_template_conversion     = false
common_content_library_name    = "sfo-w01-lib01"
common_content_library_ovf     = true
common_content_library_destroy = true

// Removable Media Settings
common_iso_datastore = "sfo-w01-cl01-ds-nfs01"

// Boot and Provisioning Settings
common_data_source      = "http"
common_http_ip          = null
common_http_port_min    = 8000
common_http_port_max    = 8099
common_ip_wait_timeout  = "20m"
common_shutdown_timeout = "15m"

Data Source Options

http is the default provisioning data source for Linux machine image builds.

You can change the common_data_source from http to disk to build supported Linux machine images without the need to use Packer's HTTP server. This is useful for environments that may not be able to route back to the system from which Packer is running. The cd_content option is used when selecting disk unless the distribution does not support a secondary CD-ROM. For distributions that do not support a secondary CD-ROM the floppy_content option is used.

common_data_source = "disk"

HTTP Binding

If you need to define a specific IPv4 address from your host for Packer's HTTP Server, modify the common_http_ip variable from null to a string value that matches an IP address on your Packer host. For example:

common_http_ip = "172.16.11.254"

Proxy Variables (Optional)

Edit the config/proxy.pkvars.hcl file to configure the following:

SOCKS proxy settings used for connecting to Linux machine images.
Credentials for the proxy server.

Example: config/proxy.pkvars.hcl

communicator_proxy_host     = "proxy.rainpole.io"
communicator_proxy_port     = 1080
communicator_proxy_username = "rainpole"
communicator_proxy_password = "<plaintext_password>"

Red Hat Subscription Manager Variables

Edit the config/redhat.pkvars.hcl file to configure the following:

Credentials for your Red Hat Subscription Manager account.

Example: config/redhat.pkvars.hcl

rhsm_username = "rainpole"
rhsm_password = "<plaintext_password>"

These variables are only used if you are performing a Red Hat Enterprise Linux Server build and are used to register the image with Red Hat Subscription Manager during the build for system updates and package installation. Before the build completes, the machine image is unregistered from Red Hat Subscription Manager.

vSphere Variables

Edit the builds/vsphere.pkvars.hcl file to configure the following:

vSphere Endpoint and Credentials
vSphere Settings

Example: config/vsphere.pkvars.hcl

vsphere_endpoint             = "sfo-w01-vc01.sfo.rainpole.io"
vsphere_username             = "svc-packer-vsphere@rainpole.io"
vsphere_password             = "<plaintext_password>"
vsphere_insecure_connection  = true
vsphere_datacenter           = "sfo-w01-dc01"
vsphere_cluster              = "sfo-w01-cl01"
vsphere_datastore            = "sfo-w01-cl01-ds-vsan01"
vsphere_network              = "sfo-w01-seg-dhcp"
vsphere_folder               = "sfo-w01-fd-templates"

Machine Image Variables

Edit the *.auto.pkvars.hcl file in each builds/<type>/<build> folder to configure the following virtual machine hardware settings, as required:

CPU Sockets (init)
CPU Cores (init)
Memory in MB (init)
Primary Disk in MB (init)
.iso Path (string)
.iso File (string)
.iso Checksum Type (string)
.iso Checksum Value (string)

Note: All variables.auto.pkvars.hcl default to using the VMware Paravirtual SCSI controller and the VMXNET 3 network card device types.

Using Environmental Variables

Some of the variables may include sensitive information and environmental data that you would prefer not to save to clear text files. You can add these to environmental variables using the example below:

export PKR_VAR_ansible_username="<ansible_username>"
export PKR_VAR_ansible_key="<ansible_key>"
export PKR_VAR_build_username="<build_username>"
export PKR_VAR_build_password="<build_password>"
export PKR_VAR_build_password_encrypted="<build_password_encrypted>"
export PKR_VAR_build_key="<build_key>"
export PKR_VAR_communicator_proxy_host = "<communicator_proxy_host>"
export PKR_VAR_communicator_proxy_port = "<communicator_proxy_port>"
export PKR_VAR_communicator_proxy_username = "<communicator_proxy_username>"
export PKR_VAR_communicator_proxy_password = "communicator_proxy_password>"
export PKR_VAR_rhsm_username="<rhsm_password>"
export PKR_VAR_rhsm_password="<rhsm_password>"
export PKR_VAR_vsphere_endpoint="<vsphere_endpoint_fqdn>"
export PKR_VAR_vsphere_username="<vsphere_username>"
export PKR_VAR_vsphere_password="<vsphere_password>"
export PKR_VAR_vsphere_datacenter="<vsphere_datacenter>>"
export PKR_VAR_vsphere_cluster="<vsphere_cluster>"
export PKR_VAR_vsphere_datastore="<vsphere_datastore>>"
export PKR_VAR_vsphere_network="<vsphere_network>"
export PKR_VAR_vsphere_folder="<vsphere_folder>"

Step 5 - Modify the Configurations and Scripts (Optional)

If required, modify the configuration and scripts files, for the Linux distributions and Microsoft Windows.

Linux Distribution Kickstart and Scripts

Username and password variables are passed into the kickstart or cloud-init files for each Linux distribution as Packer template files (.pkrtpl.hcl) to generate these on-demand.

Microsoft Windows Unattended amd Scripts

Variables are passed into the Microsoft Windows unattend files (autounattend.xml) as Packer template files (autounattend.pkrtpl.hcl) to generate these on-demand.

By default, each unattended file set the Product Key to use the KMS client setup keys.

Need help customizing the configuration files?

VMware Photon OS - Read the Photon OS Kickstart Documentation.

Ubuntu Server - Install and run system-config-kickstart on a Ubuntu desktop.

sudo apt-get install system-config-kickstart
ssh -X rainpole@ubuntu-desktop
sudo system-config-kickstart

Red Hat Enterprise Linux (as well as CentOS Linux/Stream, AlmaLinux, and Rocky Linux) - Use the Red Hat Kickstart Generator.
Microsoft Windows - Use the Microsoft Windows Answer File Generator if you need to customize the provided examples further.

Step 6 - Add Certificates

Save a copy of your PEM encoded Root Certificate Authority certificate to the following in .cer format.

/ansible/roles/base/files for Linux machine images.
/certificates for Windows machine images.

These files are copied to the guest operating systems and added the certificate to the Trusted Certificate Authority of the guest operating system. Linux distributions uses the Ansible provisioner, but Windows still uses the shell provisioner at this time.

Build

Start a build by running the build script (./build.sh). The script presents a menu the which simply calls Packer and the respective build(s).

Example: Menu for ./build.sh.

    ____             __                ____        _ __    __
   / __ \____ ______/ /_____  _____   / __ )__  __(_) /___/ /____
  / /_/ / __  / ___/ //_/ _ \/ ___/  / __  / / / / / / __  / ___/
 / ____/ /_/ / /__/ ,< /  __/ /     / /_/ / /_/ / / / /_/ (__  )
/_/    \__,_/\___/_/|_|\___/_/     /_____/\__,_/_/_/\__,_/____/

  Select a HashiCorp Packer build for VMware vSphere:

      Linux Distribution:

     	 1  -  VMware Photon OS 4
     	 2  -  Ubuntu Server 20.04 LTS
     	 3  -  Ubuntu Server 18.04 LTS
     	 4  -  Red Hat Enterprise Linux 8
     	 5  -  Red Hat Enterprise Linux 7
     	 6  -  AlmaLinux 8
     	 7  -  Rocky Linux 8
     	 8  -  CentOS Stream 8
     	 9  -  CentOS Linux 8
     	10  -  CentOS Linux 7

       Microsoft Windows:

     	11  -  Windows Server 2022 - All
     	12  -  Windows Server 2022 - Standard Only
     	13  -  Windows Server 2022 - Datacenter Only
     	14  -  Windows Server 2019 - All
     	15  -  Windows Server 2019 - Standard Only
     	16  -  Windows Server 2019 - Datacenter Only
     	17  -  Windows Server 2016 - All
     	18  -  Windows Server 2016 - Standard Only
     	19  -  Windows Server 2016 - Datacenter Only
     	20  -  Windows 11 Professional (Experimental)
     	21  -  Windows 10 Professional

      Other:

        I   -  Information
        Q   -  Quit

You can also start a build based on a specific source for some of the virtual machine images.

For example, if you simply want to build a Microsoft Windows Server 2022 Standard Core, run the following:

Initialize the plugins:

rainpole@macos packer-examples> packer init builds/windows/windows-server-2022

Build a specific machine image:

rainpole@macos windows-server-2022> packer build -force \
      --only vsphere-iso.windows-server-standard-core \
      -var-file="config/vsphere.pkrvars.hcl" \
      -var-file="config/build.pkrvars.hcl" \
      -var-file="config/common.pkrvars.hcl" \
      builds/windows/windows-server-2022

Build a specific machine image using environmental variables:

rainpole@macos windows-server-2022> packer build -force \
      --only vsphere-iso.windows-server-standard-core \
      -var-file="config/common.pkrvars.hcl" \
      builds/windows/windows-server-2022

Happy building!!!

-- Your friends at github.com/vmware-samples.

Troubleshoot

Read Debugging Packer Builds.

Credits

Maher AlAsfar @vmwarelab

Linux Bash scripting hints.
Owen Reynolds @OVDamn

VMware Tools for Windows installation PowerShell script.

jmw1973/packer-examples-for-vsphere