VCF 9 | Ward Vissers

Config a VCF (vSAN ESA) host the Easy Way

A while ago i created:

V1: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

V2: Config a VCF (vSAN ESA) host the Easy Way

With the release of VCF 9.1 is now time for again a updated version!

What does the script now:

✅ Disable ipv6

✅ Set DNS domain name

✅ Rename local datastore

✅Configure NTP

✅ MTU 9000

✅ Installs the vSAN ESA Hardware Mock VIB &

✅ Workaround to reduce impact of resync traffic in vSAN ESA clusters utilizing a 10G network

✅ Installs the Synology NFS Plug-in for VMware VAAI

✅ Memory Optimalisation Additional Transparent Page Sharing management capabilities and new default settings

✅ AMD Zen4/Zen5 IPMI Thermal Driver for ESX AMD Zen4/Zen5 IPMI Thermal Driver for ESX Fling

✅ Generate new certificate on the ESXi host (for the VCF verification check)

✅ Ask are you running Miniforum MS-A2(AMD) host & Optimalization see
VCF 9.1 – Comprehensive ESX Configuration Workarounds for Lab Deployments (Except the vSAN Compression Algorithm)

✅ Enable Memory Tiering for 9.1 and 9.0 and older (Filter OS Disk)

vSAN ESA Mock and the AMD Zen4/Zen5 IPMI Thermal Driver can be download on the Broadcom Fling page.

(https://support.broadcom.com/group/ecx/free-downloads) section and select Flings (https://support.broadcom.com/group/ecx/productdownloads?subfamily=Flings&freeDownloads=true)

You need to download the vibs separately!
For the installs put the vib’s in the same map as the script
.

You can download the script: HERE

Managing VCF Automation Resource Utilization Using CPU Limits

My VCF 9.1 Lab environment is also cpu constraint.
To free up some cpu resources I shutdown VCF automation regularly.

I installed path release https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/release-notes/patch-releases-9-1-0-x/vcf-operations/9-1-0-0100.html

Booting VCF Automation takes al lot of resources. I see regularly spikes. See screenshot below.

.

Normally VCF automation runs between 8 – 10Ghz Which is normal and fine in a 3 Node MS-A2 setup.

To make my Lab environment a little bit safer i limit VCF automation till 25000mhz.
This keeps my MS-A2 a little cooler and ensures that the node does max out.

What’s New in Private Cloud: VMware VCF 9.1 Enhancements

VMware Cloud Foundation (VCF) 9.1 is here — and it’s one of the most feature‑packed releases in years. This update isn’t just incremental; it’s a strategic modernization of compute, storage, networking, security, and operations across the entire private cloud stack.

Let’s break down the biggest enhancements and why they I think they matter.

Modernizing Infrastructure Economics with vSphere Foundation 9.1

VCF 9.1 brings several powerful updates to the vSphere layer, aimed at improving performance efficiency and reducing operational overhead.

Enhanced NVMe Memory Tiering

Workloads that demand high throughput and low latency benefit from smarter memory tiering. NVMe-based memory tiers now deliver improved performance and flexibility. (And yes — many are hoping Secure Boot support lands here as well.)

Parallel Processing of DRS vMotion

DRS can now process multiple vMotions in parallel, dramatically reducing cluster balancing times. This is especially impactful in large-scale environments with frequent workload mobility.

Live Patching for TPM-Enabled Hosts

Live patching now works even on hosts with TPM enabled — a huge win for security-conscious organizations that previously had to choose between uptime and compliance.

Networking Updates: Scale, Simplicity, and Smarter Automation

VCF 9.1 introduces major networking enhancements that streamline operations and expand connectivity options.

Enhanced Day-2 VM Lifecycle Management

Networking changes for VMs — including NIC updates, IP changes, and security policies — are now easier and more automated.

Existing VLAN Connectivity via Distributed Transit Gateways

You can now bridge existing VLAN-based networks into VCF environments more seamlessly, reducing migration friction and simplifying hybrid designs.

Streamlined Firewalls & Automated Inter-VPC Security

Security policies between VPCs are now automated, reducing manual rule creation and improving consistency across tenants.

Terraform Provider Enhancements

Better support for tenant-level policy and content management means more automation and cleaner IaC workflows.

Simplified Workload Connectivity & Enhanced Network Scale

EVPN-VXLAN Interoperability

VCF 9.1 now supports EVPN-VXLAN interoperability with the physicalnetwork fabric. This is a major step toward fully integrated, fabric-aware cloud networking.

Network Assessment & VPC Planning

New tools and workflows help architects plan VPC layouts, assess network readiness, and avoid misconfigurations before deployment.

Optimize, Modernize & Protect Storage with vSAN in VCF 9.1

Storage gets a significant upgrade in this release, especially for environments focused on efficiency and resilience.

Encryption for vSAN Global Deduplication

Global dedupe is now compatible with data-at-rest encryption — a long-awaited capability for secure, space-efficient storage.

Enhanced Stretched Cluster Capabilities

Improved resilience and smarter failure handling strengthen business continuity for mission-critical workloads.

Automated Storage Policy Management

Policies now adjust automatically based on cluster configuration changes, reducing manual tuning and risk of misalignment.

Strengthening Zero Trust Security & Platform Resilience

Security is a major theme in VCF 9.1, with improvements across the stack.

Data-at-Rest Encryption for Global Dedupe

This ensures encrypted storage without sacrificing dedupe efficiency — a rare combination in enterprise storage.

Quick Patching for vCenter

Faster patch cycles reduce exposure windows and simplify maintenance.

Live Patching for TPM-Enabled Hosts

As mentioned earlier, this is a major operational win for secure environments.

Continuous Compliance & Integrated Cyber Recovery

VCF 9.1 pushes deeper into automated compliance and recovery workflows.

Compliance Monitoring & Desired State Remediation

The platform now continuously checks VCF components against desired state and can automatically remediate drift.

VPC Policy-Based Connectivity

Security and connectivity policies can now be applied consistently across VPCs, improving governance and reducing misconfigurations.

VMware Data Services Manager 9.1: Modern Databases for AI & Cloud

Microsoft SQL Server 2022 Now GA

SQL Server 2022 is now fully supported and generally available through DSM 9.1, enabling automated lifecycle management for modern database workloads — including those powering AI and analytics.

Want to See It in Action?

VMware has published a full VCF 9.1 video podcast series that dives deeper into the new capabilities:

Enough to do in my Homelab Starting with Upgrade and testing the new features!!

Setting Up KubeDoom on Kubernetes: A Beginner’s Guide

I followed William Lam’s article about MS-A2 VCF 9.0 Lab: Configuring vSphere Kubernetes Service (VKS)

I don’t have much experience with Kubernetes but wanted to try some new things.

The only container that I have running is Home Assistant on Docker.

Got to try to get Kubedoom working. So I did with the following steps.
Maybe in the near future I’ll try to add more games: Retro DOS Games on Kubernetes

Finally having a Kubernetes cluster version 1.32, which was required for running KubeDoom.

Download kubectl

mkdir d:\kubectl

Extract the downloaded ZIP file and place both executables (kubectl.exe and kubectl-vsphere.exe) in a folder such as: d:\kubectl

.\kubectl version –client

mkdir .kube

cd .kube

New-Item config -type file

.\kubectl vsphere login –server=https://31.31.0.7 –insecure-skip-tls-verify

kubectl –kubeconfig=kubernetes-cluster-jzvx-kubeconfig.yaml get pods

kubectl –kubeconfig=kubernetes-cluster-jzvx-kubeconfig.yaml apply -f kubedoom.yaml

kubectl –kubeconfig=kubernetes-cluster-jzvx-kubeconfig.yaml -n kubedoom get svc

The password to Kubedoom is idbehold

Download VNC Viewer: https://www.realvnc.com/en/connect/download/viewer

This is funny and cool!

Fixing LCMVROPCONFIG20013 in VCF 9.0.1

ERROR LCMVROPCONFIG20013 while VCF Operations update from 9.0.0 to 9.01

ERROR LCMVROPCONFIG20013

Installed Tree: apt install tree # tree

File Exsist on the Offline Depot

File Remove from the Binary Management

Download the upgrade file again

Download the file again: sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VROPS

Pre-staged the VROPS update again

Update successfully

Upgrade VCF to 9.0.1: Setting Up an Offline Depot on Ubuntu

Upgrade VMware VCF 9.0.1: Essential Setting Up an Offline Depot on Ubuntu Instructions

If you are planning to upgrade to latest release of VMware Cloud Foundation (VCF) 9.0.1 and you what to install/upgrade you need to have a offline depot.

I had some struggle with the VCF Fleet upgrades. I thought the VCF installer Offline Depot was sufficient. I seems not.

William Liam did a nice explaining about the options: VCF Software Depot Structure Deep Dive for Install & Upgrade. Which confirms my struggle.

Sow I build my own offline depot

Prerequisites

First, download the required bundle files from the Broadcom VMware portal. This includes:

vcf-9.0.1.0-offline-depot-metadata.zip (mandatory)
vcf-download-tool-9.0.1.0.24962179.tar.gz (mandatory)

On the Depot Server (Ubuntu)

Sizing for the disk is minimal 100GB I used 200GB thin Provisiond

I did a Ubuntu install on a 200GB disk (Partition without lvm)

Login

Sudo passwd root

Vim /etc/ssh/sshd_config

if the following line exists, possibly commented out (with a # in front):

PermitRootLogin

Then change it to the following, uncommenting if needed (remove the # in front):

PermitRootLogin yes

sudo service ssh restart

now you can login als root

Install Apache and tools

Sudo update

sudo apt install apache2 openssl apache2-utils unzip

Create a certificate config file

nano ~/vcf-openssl.cnf

Paste the following:

[ req ]

default_bits = 2048

prompt = no

default_md = sha256

distinguished_name = dn

x509_extensions = v3_req

[ dn ]

C = US

ST = CA

L = LA

O = TS

OU = IT

CN = flt-depot.wardhomelab.nl

emailAddress = a@b.c

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = digitalSignature, keyEncipherment

extendedKeyUsage = serverAuth

subjectAltName = @alt_names

[ alt_names ]

DNS.1 = flt-depot.wardhomelab.nl

IP.1 = 192.168.150.246

Replace the values with your organization’s details.

Generate the certificate

cd /etc/apache2
sudo mkdir ssl

sudo openssl req -x509 -nodes -days 365 \

-newkey rsa:2048 \

-keyout /etc/apache2/ssl/vcf.key \

-out /etc/apache2/ssl/vcf.crt \

-config ~/vcf-openssl.cnf

Create a basic auth user

Sudo htpasswd -c /etc/apache2/.htpasswd vcfadmin

Configure Apache

sudo nano /etc/apache2/sites-available/default-ssl.conf

Paste the following:

ServerAdmin webmaster@localhost

DocumentRoot /var/www/html

SSLEngine on

SSLCertificateFile /etc/apache2/ssl/vcf.crt

SSLCertificateKeyFile /etc/apache2/ssl/vcf.key

SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1

RequestHeader unset Proxy early

Options Indexes FollowSymLinks

AllowOverride None

AuthType Basic

AuthName “VCF Depot”

AuthUserFile /etc/apache2/.htpasswd

Require valid-user

</Directory>

</VirtualHost>

Enable modules and restart Apache

sudo a2enmod ssl headers

sudo a2ensite default-ssl

sudo systemctl restart apache2

Extract the metadata ZIP

Sudo chmod 777 /home

Upload vcf-9.0.1.0-offline-depot-metadata.zip /home/ with Winscp

Sudo unzip vcf-9.0.1.0-offline-depot-metadata.zip -d /var/www/html

Make sure this file exists:

ls /var/www/html/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json

Remove Index.html

sudo rm -f /var/www/html/index.html

Create your Download token

Create your Download token in de Broadcom portal

Create a token file in /home folder

Upload the VCF download tool

Sudo mkdir /home/vcf-download-tool

Sudo chmod 777 /home/vcf-download-tool

cd /tmp/vcf-download-tool

Upload the tool in /tmp/vcf-download-tool folder

tar -xf vcf-download-tool-9.0.1.0.24962179.tar.gz

sudo reboot

Run the following

Cd /tmp/vcf-download-tool/bin

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –automated-install –type=INSTALL

Download all the appliances

sudo ./vcf-download-tool binaries list –depot-download-token-file=/home/token –vcf-version=9.0.1 –type=INSTALL –sku=VCF

Upgrade

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component SDDC_MANAGER_VCF

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VCENTER

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRSLCM

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VROPS

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component NSX_T_MANAGER

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VCF_OPS_CLOUD_PROXY

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRA

Additional

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRNI

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRLI

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component HCX

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRO

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VIDB

Upgrade Binaries will be visible

On de SDDC manager

Open de VM console

Vim /etc/ssh/sshd_config

if the following line exists, possibly commented out (with a # in front):

PermitRootLogin

Then change it to the following, uncommenting if needed (remove the # in front):

PermitRootLogin yes

systemctl restart sshd

Copy vcf.crt

scp root@ftt-depot.wardhomelab.nl:/etc/apache2/ssl/vcf.crt /tmp/vcf.crt

Import the certificate into the Java truststore

sudo keytool -import -trustcacerts -alias vcfDepotCert1 \

-file /tmp/vcf.crt \

-keystore /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts \

-storepass changeit

Reboot

Now u should connect VCF to you offline depot

You want you upgrade to 9.0.1 ivm Edge Issue

Special thanks to vmtechnics for putting me in the right direction

Addressing Edge Server Issues in VCF 9.0 after a Shutdown

After shutting down the VCF 9.0 environment completely, I had 2 strange things.

I followed the shutdown order for the management domain.

Edge servers lost from NIC 2 and NIC 3 the network configuration
Edge Servers did not start, not correctly

I deployed a third edge server. Two new distributed port groups were created.

These are missing for the edge01a and edge01b servers. These should not be deleted/missing after shutdown.

Create a distributed port group with a VLAN trunk

Set NIC 2 and 3 to this network.

Edge Nodes’ Edge Cluster is running fine again 😊

I found this on the Daniel Kruger Blog: https://sdn-warrior.org/posts/vcf9-ms-a2-special/

This is a known issue with VCF 9.0.0: When an edge is created by the Setup Network Connectivity UI, the system-created dvpg consumed by the edge gets deleted when the edge is powered on after 24 hrs. The port group assigned to the NSX Edge uplink has disappeared, making it impossible to use the network through NSX Edge. You can find the release notes here.

Build my lab nested, but not enough resources to do a full upgrade to 9.0.1.
So I build the lab again, but physically. More about that later!

I love testing things; you learn so much!

MS-A2: How a Ubiquity Flex 2.5G skips the 10G check in VCF 9.X

VCF 9 and 9.0.1 require 10GB nic.

I use for my homelab the MS-A2 with the 2 x 10GbE Intel X710 SFP+ (NIC 1 & 2 from the left). It is silent and fast and have enough power to run VCF 9!

So I bought also the 10GbE SFP+ to Copper Transceiver

But when you connect the MS-A2 to USW Flex 2.5G switch on the switch side is 2.5G.

But on the VMware side is 10GB 😊

No need to do the following: Disable 10GbE NIC Pre-Check in the VCF 9.0 Installer or

Enhancement in VCF 9.0.1 to bypass vSAN ESA HCL & Host Commission 10GbE NIC Check

It working great! No need to buy expensive 10GB switch!

Running VCF 9 in a Nested Lab using vSAN ESA on a single host is that not great?

For the deployment I used a server that has 16 cores en 32 cores total and a lot of ram!

Installation on Nested ESX VM’s

For the nested ESX VMs I am using the generic ESX image. In my lab I’m using 4 nested ESX nodes for vSAN ESA (Minimum is needed is 3).

Ideally add 24 vCPUs to the nested ESX, but VCF Automation is capable of being deployed with 16 vCPUs
Check Expose hardware assisted virtualization to the guest OS under CPU
Add a NVME controller and connect disks to this controller if planning to use vSAN ESA (Don’t forget to remove the old Controller)
Add at least two VMXNET3 nics connected to the same network
Choose VMware ESXi 8.0 or later as the Guest OS version.
Change the IDE CD-ROM to SATA ROM (Failed to locate kickstart on Nested ESXi VM CD-ROM in VCF 9.0)

Config the 4 Host for vSAN ESA

To config the 4 Hosts for vSAN ESA I used my own powercli script where I blogged about here: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script

You can download the script HERE!

Installing the Vib works still great. No problem with ESA Pre-check: ✅

If you have problems check this great blog from William Lam: vSAN ESA Disk & HCL Workaround for VCF 9.0

VCF installer.

At first you have to deploy the VMware Cloud Foundation

The name of the installer is a little off: VCF-SDDC-Manager-Appliance-9.0.0.0.24703748.ova

Download token

If you don’t have a download token from the Broadcom support then it’s a little complicated. I won’t go into this in depth now but here is a nice article if you have a Synology Nas: VCF 9.0 Offline Depot using Synology

(Note: For any vExpert reading this… You need to claim your own domain name because build your profile is not working with a @hotmail.com or @gmail accounts, Noted by Broadcom)

I looks something like this!

10GBE Nic Pre-Check Issue

Next thing to do was: Disable 10GbE NIC Pre-Check in the VCF 9.0 Installer or this should also work: How to change the vmxnet3 link speed of a VM (Not tested)

Deployment

I have deployed the beta in a earlier state. For deployment I used the same json file to start with.

VCF Automation will deployed automatically that was only the change that I changed it

It’s running!

Performance running Nested!!

Above the screenshot from the host with de Nested VCF lab deployment it’s quite cpu intensive (2 x CPU 8cores / 16 Threads)

Optimalization

After the deployment I did three things
– On the vSAN ESA Cluster (Disable Auto Disk Claim) (Keep the warning away)
– On the NSX VM reduce the cpu reservation from 6000 🡪 2000 (It helps but not enough)
– VCF automation VM 24 cores to 16 cores.

NSX en VCF automation are really CPU intensive.

I had deployed also two Edge servers but the server did not like that. Edges are also CPU intensive.

I am thinking about adding MS-01 or MS-A2 for splitting some load.

Category: VCF 9

Config a VCF (vSAN ESA) host the Easy Way

Like this:

Managing VCF Automation Resource Utilization Using CPU Limits

Like this:

What’s New in Private Cloud: VMware VCF 9.1 Enhancements

Like this:

Setting Up KubeDoom on Kubernetes: A Beginner’s Guide

Like this:

Fixing LCMVROPCONFIG20013 in VCF 9.0.1

Like this:

Addressing Edge Server Issues in VCF 9.0 after a Shutdown

Like this:

MS-A2: How a Ubiquity Flex 2.5G skips the 10G check in VCF 9.X

Like this:

Running VCF 9 in a Nested Lab using vSAN ESA on a single host is that not great?

Config the 4 Host for vSAN ESA

VCF installer.

Download token

10GBE Nic Pre-Check Issue

Deployment

It’s running!

Performance running Nested!!

Optimalization

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Prerequisites

On the Depot Server (Ubuntu)

Login

Install Apache and tools

Create a certificate config file

Create a basic auth user

Configure Apache

Enable modules and restart Apache

Extract the metadata ZIP

Remove Index.html

Create your Download token

Upload the VCF download tool

Run the following

Download all the appliances

Upgrade

Additional

On de SDDC manager

Copy vcf.crt

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Config the 4 Host for vSAN ESA

VCF installer.

Download token

10GBE Nic Pre-Check Issue

Deployment

It’s running!

Performance running Nested!!

Optimalization

Share this:

Like this: