VCF 9.1 Enable High Availability for a Small VCF Deployment for VCF Management Services (VCFMS)

I still working on my shutdown en startup script for VCF 9.1

Because it is a small lab you have only 1 control plane and 3 workers.
A option to have second control plane vm would be nice for easier recovery.

Then I read about: William Lam: VCF 9.1 – Enabling High Availability for a Small VCF Management Services (

VCF

) Deployment

Tried it:

Checked operations.

Checked vCenter

I have now also logs running in VCF management services.
Go back to a single control plane is the same way disable HA.

When I have some time i want to test: Leaha’s Blog: VCF 9 Management Services Lab Downsize

VCF 9.1 Critical JV Certificate Caching Bug Can Cause Silent Outage

A field experience with VCF Operations and Fleet Lifecycle Management

The problem I encountered

Over the last few weeks, I encountered a strange issue in VMware Cloud Foundation 9.1. The Build > Software and Build > Lifecycle functionality was no longer working correctly.

Figure 1. Lifecycle functionality failing in the VCF interface.

My first thought was that something had gone wrong with the VCF Services Runtime control plane. I did not immediately know why it had failed, and unfortunately I did not have a VM-level backup of the VCF Services virtual machines available.

After investigating the symptoms, I suspected that I had run into a VCF 9.1 certificate-related bug.

Update: critical certificate caching issue

VCF 9.1 contains a critical issue in which the Java Virtual Machine (JVM) caches an internal security certificate during the initial bootstrap.

Issue details

Although the system correctly rotates the certificate in the background, the application does not dynamically reload the renewed certificate. Because these certificates have a strict 90-day validity period, environments deployed at General Availability (May 12, 2026) can begin experiencing silent outages around August 10, 2026.

Impact

The failure can occur without advance health-check warnings. Once the 90-day certificate threshold is reached, the environment may experience:

• Generic “503 Service Unavailable” errors

• Loss of access to affected UI functionality and blockage of component deployments

• Failure of day-two operations within Fleet Lifecycle Management (Fleet LCM)

What this looked like in my environment

In my case, the symptoms appeared in the VCF Build and Lifecycle areas and initially looked like a failure of the VCF Services Runtime control plane. The certificate-caching issue provides a plausible explanation for this behavior.

Action required

If you are running VCF 9.1, review the relevant Broadcom Knowledge Base articles and apply the documented workaround for the affected component. The source document identifies the following KB topics:

• VCF Operations: Error: “transport failure reason: TLS_error” when loading Lifecycle tab in VCF 9.1

• VCF Automation: Error: “transport failure reason: TLS_error” in VCF Automation UI

Takeaway

A certificate can be successfully rotated on the platform while an application continues using an older certificate cached by its JVM. That makes this issue particularly difficult to recognize: certificate rotation may appear healthy even though application communication eventually starts failing.

For VCF 9.1 environments approaching or exceeding 90 days since deployment, certificate-related TLS errors and unexplained 503 responses should therefore be investigated promptly.

What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario

After three great sessions at VMUG Connect Amsterdam and VMUG Connect Online, and VMUGNL Summer Sessions. I’ve had requests to share the slides for my session “What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario“.

The response was great.

In the slide deck you will find:

✅ The Start of My VCF Story
✅ First Challenge
✅ Why i wanted to build a real Senario
✅ Why i choise going to a build a real VCF scenario
✅ The Road
✅ The Lessions Learned & Next Steps

Building and Optimizing a Minimal 2-Node VMware Cloud Foundation 9 Lab Deploymentfrom Ward Vissers

VCF 9.1: Fixing Root Account Password Expiration Issues

One of the first post-deployment tasks after installing VMware Cloud Foundation (VCF) 9.1 is configuring a password policy to ensure compliance with your organization’s security standards.

While the password policy is successfully applied to most managed accounts, you may notice that the root accounts of the VCF Operations appliance and the VCF Proxy appliance do not follow the configured password expiration policy.

As a result, the expiration date shown in VCF Management remains unchanged, even though a password policy has been configured.

Symptoms

You may observe one or more of the following:

• A password policy is configured successfully in VCF Management.
• Compliance checks complete without errors.
• The root account of the VCF Operations appliance still shows an incorrect or outdated password expiration date.
• The same behavior occurs on the VCF Proxy appliance.

This can be confusing because the password policy appears to be configured correctly, but it is not enforced for these Linux root accounts.

Why Does This Happen?

The password policy configured in VCF does not automatically update the Linux root account password aging settings on the VCF Operations and Proxy appliances.

Instead, these appliances continue to rely on the Linux ‘chage’ configuration to determine when the root password expires.

VMware has documented this behavior and provided a straightforward workaround.
https://knowledge.broadcom.com/external/article/441344/configured-password-policy-is-not-being.html

Resolution

Before making any changes, enable SSH access on the VCF Operations appliance if it is currently disabled.
https://knowledge.broadcom.com/external/article/315976/enabling-ssh-access-in-aria-operations.html

Step 1 – Connect to the VCF Operations Appliance

SSH to the VCF Operations appliance using an administrative account.

ssh admin@<vcf-operations-appliance>

Step 2 – Verify the Current Password Expiration

Run:

sudo chage -l root

Step 3 – Configure the Password Expiration

Configure a 365-day password lifetime:

sudo chage -M 365 root

Step 4 – Verify the Change

Run:

chage -l root

Step 5 – Repeat for the VCF Proxy Appliance

Repeat the same commands on the VCF Proxy appliance, either through SSH (if enabled) or via the VMware console.

Wait for VCF to Update

Notes

The updated password expiration date is not reflected immediately in the VCF Management interface.

VCF periodically refreshes password information, so it may take 10–30 minutes (or longer depending on your environment) before the new expiration date appears.

• This change only affects the Linux root account.
• Verify the setting after appliance upgrades.
• Adjust the password lifetime (90, 180, 365 days, etc.) according to your security policy.

Conclusion

Although VCF 9.1 allows administrators to centrally configure password policies, the Linux root accounts on the VCF Operations and VCF Proxy appliances continue to rely on the local Linux password aging configuration.

Updating the password expiration with the chage command ensures that the root account complies with your organization’s password policy. Once VCF completes its next inventory synchronization, the correct expiration date is displayed in the VCF Management interface.

Config a VCF (vSAN ESA) host the Easy Way

A while ago i created:

V1: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

V2: Config a VCF (vSAN ESA) host the Easy Way

With the release of VCF 9.1 is now time for again a updated version!

What does the script now:

✅ Disable ipv6

✅ Set DNS domain name

✅ Rename local datastore

✅Configure NTP

✅ MTU 9000

✅ Installs the vSAN ESA Hardware Mock VIB &

✅ Workaround to reduce impact of resync traffic in vSAN ESA clusters utilizing a 10G network

✅ Installs the Synology NFS Plug-in for VMware VAAI

✅ Memory Optimalisation Additional Transparent Page Sharing management capabilities and new default settings

✅ AMD Zen4/Zen5 IPMI Thermal Driver for ESX AMD Zen4/Zen5 IPMI Thermal Driver for ESX Fling

✅ Generate new certificate on the ESXi host (for the VCF verification check)

✅ Ask are you running Miniforum MS-A2(AMD) host & Optimalization see
VCF 9.1 – Comprehensive ESX Configuration Workarounds for Lab Deployments (Except the vSAN Compression Algorithm)

✅ Enable Memory Tiering for 9.1 and 9.0 and older (Filter OS Disk)

vSAN ESA Mock and the AMD Zen4/Zen5 IPMI Thermal Driver can be download on the Broadcom Fling page.

(https://support.broadcom.com/group/ecx/free-downloads) section and select Flings (https://support.broadcom.com/group/ecx/productdownloads?subfamily=Flings&freeDownloads=true)

You need to download the vibs separately!
For the installs put the vib’s in the same map as the script
.

You can download the script: HERE

Managing VCF Automation Resource Utilization Using CPU Limits

My VCF 9.1 Lab environment is also cpu constraint.
To free up some cpu resources I shutdown VCF automation regularly.

I installed path release https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/release-notes/patch-releases-9-1-0-x/vcf-operations/9-1-0-0100.html

Booting VCF Automation takes al lot of resources. I see regularly spikes. See screenshot below.

.

Normally VCF automation runs between 8 – 10Ghz Which is normal and fine in a 3 Node MS-A2 setup.

To make my Lab environment a little bit safer i limit VCF automation till 25000mhz.
This keeps my MS-A2 a little cooler and ensures that the node does max out.

VCF 9.1 What fixed the stuck deployment at “Deploy and Configure VCF Management Platform”

Overview

While deploying VMware Cloud Foundation (VCF) 9.1 in a homelab environment, the installation repeatedly failed during the ‘Deploy and Configure VCF Management Platform’ stage. Despite performing nine completely clean installations, the deployment consistently stopped at the same point.

Error Observed

The deployment task failed with the message: ‘Add VM Name Prefix to NSX Firewall Exclusion List’. The failure was identified in /var/log/vmware/vcf/domainmanager/domainmanager.log

Initial Research

Several Broadcom Knowledge Base articles appeared relevant, including KB440449 and KB 441122. Although the symptoms were similar, neither article fully resolved the issue.

VMSP Configuration Review

The original VMSP configuration used a name value matching the prefix of the fleetFqdn. The configuration was modified to use a unique VMSP cluster name. While this appeared promising, the issue persisted.

Additional Troubleshooting

Additional troubleshooting included changing VMSP IP ranges, rebuilding DNS records, validating forward and reverse DNS resolution, and reviewing deployment logs for networking issues.

Root Cause Analysis

The issue was ultimately not caused by the NSX firewall exclusion configuration. Multiple infrastructure issues contributed to deployment instability.

Resolution

1. Configure a single authoritative NTP source, preferably the domain controller.
See the planning and preparation workbook
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/planning-and-preparation.html
2. Verify DNS records and name resolution.
3. Upgraded to a dedicated 10G Switch Ubiquiti UniFi Pro XG 8 PoE ipv 2.5GB Ubiquity Switch
Switch Pro XG 8 PoE - Ubiquiti Store Europe
4. Replace faulty network components.

Conclusion

Although the deployment failure appeared to indicate an NSX firewall exclusion issue, the underlying cause was network instability combined with infrastructure configuration problems. After correcting NTP configuration, validating DNS, upgrading network connectivity, and replacing the defective SFP+ module, the VCF 9.1 deployment completed successfully.

Exploring Minimum 2 Node Homelab at VMUG Connect in Amsterdam

Thrilled to join the @MyVMUG community at VMUG Connect in Amstedam! I’ll be diving into Minimum 2 Node Homelab Setup and sharing practical insights you can use in your environment. Can’t wait to learn from other members too!

Afbeelding met tekst, kleding, person, Menselijk gezichtDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, schermopnameDoor AI gegenereerde inhoud is mogelijk onjuist.

Config a VCF (vSAN ESA) host the Easy Way

A while ago i created: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

Sow it’s time for a new updated version.

What does the script now:

✅ Disable ipv6

✅ Set DNS domain name

✅ Rename local datastore

✅ Configure NTP

✅ Installs the vSAN ESA Hardware Mock VIB

✅ Installs the Synology NFS Plug-in for VMware VAAI

✅ Installs the Realtek Network Driver for ESXi more information here
I use the 10Gtek 1.25/2.5/5/10GBase-T SFP+ RJ-45 CAT.6a Copper Transceiver on the MS-A2

✅ Configure vSAN DOMNetworkSchedulerThrottleComponent

✅ Generate new certificate on the ESXi host (for the VCF verification check)

✅ Ask are you running Miniforum MS-A2(AMD) host & Then Disable apichv

✅ Enable Memory Tiering

You need to download the vibs separately!
For the installs put the vib’s in the same map as the script
You can download the script: HERE

Powercli Script to update From VMware.Powercli to VCF.Powercli

I made a little script that does the following:

Check if VMware or VCF PowerCLI is installed and is outdated
Check if VCF.Powercli is the latest installed
If VMware.Powercli is installed upgrade to VCF.Powercli
If VCF Powercli is installed but not the latest version upgrade to VCF to the latest version

You can download the script on my github repository: https://github.com/WardVissers/VCF-Public/blob/main/Check%20Installed%20VCF-Powercli%20Version.ps1

$installedVCFPowercli   = Get-InstalledModule VCF.PowerCLI -ErrorAction SilentlyContinue
$installedVMwarePowercli   = Get-InstalledModule VMware.PowerCLI -ErrorAction SilentlyContinue
$latestVCFPowercli    = Find-Module VCF.PowerCLI -ErrorAction SilentlyContinue
$latestVMwarePowercli =  Find-Module VMware.PowerCLI -ErrorAction SilentlyContinue
$OldModules = Get-InstalledModule VMware.* -ErrorAction SilentlyContinue
$OldModules += Get-InstalledModule VCF.* -ErrorAction SilentlyContinu | Where-Object {[version]$_.Version -lt $latestVCF.Version}

Function Uninstall-OldPowercliEditons {
$Total = $OldModules.Count
$Index = 0
foreach ($Module in $OldModules) {
    $Index++
    $Percent = ($Index / $Total) * 100

    Write-Progress `
        -Activity "Delete old PowerCLI version" `
        -Status "Uninstall version $($Module.Version) ($Index from $Total)" `
        -PercentComplete $Percent

    try {
        Uninstall-Module -Name $Module.name -AllVersions -Force  # -ErrorAction silentlycontinue -ErrorVariable +err
        Write-Host "🗑 Removed: PowerCLI $($Module.Name) with $($Module.Version)"
    }
    catch {
        Write-Error "❌ Error with deleting PowerCLI $($Module.Version): $_"
    }

    Start-Sleep -Seconds 1
  }
}

if (-not ($installedVMwarePowercli -or $installedVCFPowercli -or $OldModules )) {
    "❌ VCF.PowerCLI is not installed"
    Install-Module VCF.PowerCLI -AllowClobber  -Scope CurrentUser # -SkipPublisherCheck
}
elseif ([version]$installedVCFPowercli.Version -eq [version]$latestVCFPowercli.Version) {
    "✅ VCF PowerCLI is up-to-date ($($installed.Version))"
}
elseif ($installedVMwarePowercli) {
    "⬆ VMware Powercli is installed needed upgrade to VCF Powercli"
    Uninstall-OldPowercliEditons
    Write-Host "Uninstall is succes vol"  -ForegroundColor Yellow
    Install-Module VCF.PowerCLI -AllowClobber  -Scope CurrentUser # -SkipPublisherCheck
    Write-Host "Install VCF Powercli is succes" -ForegroundColor Green
}
else {
    "⬆ VCF Powercli Update beschikbaar $($latestVCFPowercli.Version)"
    Uninstall-OldPowercliEditons
    Write-Host "Uninstall is succes" -ForegroundColor Yellow
    Update-Module -Name VCF.PowerCLI -Force
    Write-Host "Update is succes vol" -ForegroundColor Green
}

$installedVCFPowercli   = Get-InstalledModule VCF.PowerCLI -ErrorAction SilentlyContinue
$installedVMwarePowercli   = Get-InstalledModule VMware.PowerCLI -ErrorAction SilentlyContinue
$latestVCFPowercli    = Find-Module VCF.PowerCLI -ErrorAction SilentlyContinue
$latestVMwarePowercli =  Find-Module VMware.PowerCLI -ErrorAction SilentlyContinue
$OldModules = Get-InstalledModule VMware.* -ErrorAction SilentlyContinue
$OldModules += Get-InstalledModule VCF.* -ErrorAction SilentlyContinu | Where-Object {[version]$_.Version -lt $latestVCF.Version}

Function Uninstall-OldPowercliEditons {
$Total = $OldModules.Count
$Index = 0
foreach ($Module in $OldModules) {
    $Index++
    $Percent = ($Index / $Total) * 100

    Write-Progress `
        -Activity "Delete old PowerCLI version" `
        -Status "Uninstall version $($Module.Version) ($Index from $Total)" `
        -PercentComplete $Percent

    try {
        Uninstall-Module -Name $Module.name -AllVersions -Force  # -ErrorAction silentlycontinue -ErrorVariable +err
        Write-Host "🗑 Removed: PowerCLI $($Module.Name) with $($Module.Version)"
    }
    catch {
        Write-Error "❌ Error with deleting PowerCLI $($Module.Version): $_"
    }

    Start-Sleep -Seconds 1
  }
}

if (-not ($installedVMwarePowercli -or $installedVCFPowercli -or $OldModules )) {
    "❌ VCF.PowerCLI is not installed"
    Install-Module VCF.PowerCLI -AllowClobber  -Scope CurrentUser # -SkipPublisherCheck
}
elseif ([version]$installedVCFPowercli.Version -eq [version]$latestVCFPowercli.Version) {
    "✅ VCF PowerCLI is up-to-date ($($installed.Version))"
}
elseif ($installedVMwarePowercli) {
    "⬆ VMware Powercli is installed needed upgrade to VCF Powercli"
    Uninstall-OldPowercliEditons
    Write-Host "Uninstall is succes vol"  -ForegroundColor Yellow
    Install-Module VCF.PowerCLI -AllowClobber  -Scope CurrentUser # -SkipPublisherCheck
    Write-Host "Install VCF Powercli is succes" -ForegroundColor Green
}
else {
    "⬆ VCF Powercli Update beschikbaar $($latestVCFPowercli.Version)"
    Uninstall-OldPowercliEditons
    Write-Host "Uninstall is succes" -ForegroundColor Yellow
    Update-Module -Name VCF.PowerCLI -Force
    Write-Host "Update is succes vol" -ForegroundColor Green
}

Tag: VM

VCF 9.1 Enable High Availability for a Small VCF Deployment for VCF Management Services (VCFMS)

Like this:

Config a VCF (vSAN ESA) host the Easy Way

Like this:

Managing VCF Automation Resource Utilization Using CPU Limits

Like this:

Exploring Minimum 2 Node Homelab at VMUG Connect in Amsterdam

Like this:

Config a VCF (vSAN ESA) host the Easy Way

Like this:

Powercli Script to update From VMware.Powercli to VCF.Powercli

Like this:

Share this:

Like this:

The problem I encountered

Update: critical certificate caching issue

Issue details

Impact

What this looked like in my environment

Action required

Takeaway

Share this:

Like this:

Share this:

Like this:

Symptoms

Why Does This Happen?

Resolution

Step 1 – Connect to the VCF Operations Appliance

Step 2 – Verify the Current Password Expiration

Step 3 – Configure the Password Expiration

Step 4 – Verify the Change

Step 5 – Repeat for the VCF Proxy Appliance

Wait for VCF to Update

Notes

Conclusion

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Overview

Error Observed

Initial Research

VMSP Configuration Review

Additional Troubleshooting

Root Cause Analysis

Resolution

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: