VCF 9.1 Enable High Availability for a Small VCF Deployment for VCF Management Services (VCFMS)

I still working on my shutdown en startup script for VCF 9.1

Because it is a small lab you have only 1 control plane and 3 workers.
A option to have second control plane vm would be nice for easier recovery.


Then I read about: William Lam: VCF 9.1 – Enabling High Availability for a Small VCF Management Services (

VCF

) Deployment

Tried it:

Checked operations.

Checked vCenter

I have now also logs running in VCF management services.
Go back to a single control plane is the same way disable HA.

When I have some time i want to test: Leaha’s Blog: VCF 9 Management Services Lab Downsize

VCF 9.1 Critical JV Certificate Caching Bug Can Cause Silent Outage

A field experience with VCF Operations and Fleet Lifecycle Management

The problem I encountered

Over the last few weeks, I encountered a strange issue in VMware Cloud Foundation 9.1. The Build > Software and Build > Lifecycle functionality was no longer working correctly.

Figure 1. Lifecycle functionality failing in the VCF interface.

My first thought was that something had gone wrong with the VCF Services Runtime control plane. I did not immediately know why it had failed, and unfortunately I did not have a VM-level backup of the VCF Services virtual machines available.

After investigating the symptoms, I suspected that I had run into a VCF 9.1 certificate-related bug.

Update: critical certificate caching issue

VCF 9.1 contains a critical issue in which the Java Virtual Machine (JVM) caches an internal security certificate during the initial bootstrap.

Issue details

Although the system correctly rotates the certificate in the background, the application does not dynamically reload the renewed certificate. Because these certificates have a strict 90-day validity period, environments deployed at General Availability (May 12, 2026) can begin experiencing silent outages around August 10, 2026.

Impact

The failure can occur without advance health-check warnings. Once the 90-day certificate threshold is reached, the environment may experience:

• Generic “503 Service Unavailable” errors
• Loss of access to affected UI functionality and blockage of component deployments
• Failure of day-two operations within Fleet Lifecycle Management (Fleet LCM)

What this looked like in my environment

In my case, the symptoms appeared in the VCF Build and Lifecycle areas and initially looked like a failure of the VCF Services Runtime control plane. The certificate-caching issue provides a plausible explanation for this behavior.

Action required

 

If you are running VCF 9.1, review the relevant Broadcom Knowledge Base articles and apply the documented workaround for the affected component. The source document identifies the following KB topics:

Takeaway

A certificate can be successfully rotated on the platform while an application continues using an older certificate cached by its JVM. That makes this issue particularly difficult to recognize: certificate rotation may appear healthy even though application communication eventually starts failing.

For VCF 9.1 environments approaching or exceeding 90 days since deployment, certificate-related TLS errors and unexplained 503 responses should therefore be investigated promptly.

VCF LOG Deployment changed in 9.1.0.400

Since VCF 9.1.0.400, a lot has changed in the platform and one of the more noticeable shifts is how log management works. It is no longer a standalone virtual appliance.

Instead, it now runs as a service inside the VCF Management Services Cluster, the same Kubernetes-based runtime that also hosts Fleet Management and the other core management components.

Additionally, in this blog I walk through how to deploy VCF Log Management in 9.1.0.400.

What Changed

In VCF 9.0 and earlier, Log Management (formerly Aria Operations for Logs) was deployed as its own appliance VM or appliance cluster. Additionally, from 9.1.0.400 onward, it is a built-in, containerized core service that runs natively on the VCF Management Services Platform. Deployment, lifecycle management, configuration, access control, and alerting are now all handled centrally through VCF Operations, rather than through a separate product UI.

Prerequisites

• DNS: A DNS-registered FQDN (with both forward and reverse lookup working)
• Capacity in the VCF Services Runtime: Deploying Log Management can trigger an automatic resize of the underlying Kubernetes cluster, so plan for the additional compute ahead of time.
• Sizing decision: The deployment size you choose determines how much compute and storage each node gets.
• Networking: Deployment currently supports VLAN-backed networks for the VCF Management Services; overlay networking is not yet supported for this component.

Resource Requirements for Log Management Deployments.

Size

vCPUs per Node

Memory per Node

Storage per Node

Number of Nodes

Small

8

16 GB

575 GB

1-19

Medium

24

48 GB

575 GB

3-19

Large

32

64 GB

575 GB

3-19

Procedure

1.Log in to the VCF Operations user interface at https://<vcf_operations_fqdn
2.Navigate to Build Lifecycle VCF Management.
3.On the Components tab, select Add Component Log management and configure the parameters for the deployment.


Now click Next to proceed to the Summary page for final review before deployment. Click Finish to start the deployment. As soon as the Deployment is done, you will see Log Management is running under VCF Operations -> Build -> Lifecycle.

Additionally, you can find the logs under VCF Operations -> Operate -> Log..

If you click on Log Sources, you will get an overview of the currently configured integrations, but this does not mean, that the Log collection itself is enabled.

Additionally, navigate to VCF Operations -> Operate -> Administration -> Configurations -> Log Collection to check Status. However, under Status, the Log collection is not enabled.

If you click on the “i” under Status you will see:

This is a little bit strange in my opinion, but yes, you have to switch back to the Integrations page inside VCF Operations Operate Administration Integrations and Edit your VMware Cloud Foundation Account:

From the Account page switch to Domains:

Next, on the Domain Page, start with vCenter to activate Logs Collection by clicking ‘Activate Log Collection’ and SAVE.

Same procedure i.e. NSX:

Additionally, after activiation of Log collection you will see more Logs inside VCF Operations:


.

Next, go to VCF Operations Operate Administration Configuration Log Collection.
Next, click Edit and scroll down.

Additionally, enable (Edit Log collection configuration).

.

What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario

After three great sessions at VMUG Connect Amsterdam and VMUG Connect Online, and VMUGNL Summer Sessions. I’ve had requests to share the slides for my session “What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario“.

The response was great.

In the slide deck you will find:

âś… The Start of My VCF Story
âś… First Challenge
âś… Why i wanted to build a real Senario
âś… Why i choise going to a build a real VCF scenario
âś… The Road
âś… The Lessions Learned & Next Steps

VCF 9.1 How to Safely Shut Down the Services Runtime Cluster Using PowerShell

Broadcom provides an official procedure for gracefully shutting down a VMware Cloud Foundation (VCF) 9.1 Services Runtime cluster. The provided solution uses a Linux shell script (vcf_services_runtime_shutdown.sh), which works well on Linux systems but is less convenient for Windows administrators.

Rather than using Windows Subsystem for Linux (WSL) or a separate Linux machine, I converted the official shell script into a native PowerShell version. This allows the entire shutdown procedure to be executed directly from a Windows workstation while following the same workflow as the original Broadcom script.

Downloads

📦 VCF 9.1 Shutdown VCF Services VMs.ps1
📦 vcf_services_runtime_shutdown.ps1

đź“„ Original Broadcom Knowledge Base vcf_services_runtime_shutdown.sh

Why Use the Official Shutdown Procedure?

• Kubernetes workloads are gracefully terminated.
• Running services are stopped in the correct order.
• Databases remain in a consistent state.
• Cluster metadata is preserved.
• The shutdown process completes without leaving services in an inconsistent state.

Prerequisites

• VMware Cloud Foundation 9.1
• PowerShell 7.x (recommended)
• Latest VCF PowerCLI
• vCenter Administrator credentials
• Network connectivity to the Control Plane node

Before You Begin

• Power off any VCF Automation virtual machines before shutting down the Services Runtime cluster to avoid interference with automation tasks.

Determine the Control Plane Node IP Address

• Open VCF Operations.
• Navigate to Build → Lifecycle → Components.
• Select VCF Services Runtime.
• Scroll to the Nodes section.
• Locate a Control Plane node and note its IP address.

PowerShell Wrapper Script

• Detects the current script location.
• Changes to the correct working directory.
• Calls the shutdown script with the required parameters.
• Simplifies execution.

Download Wrapper Script

➡ Download: Shutdown VCF Services.ps1

# VCF Services shutdown by Ward Vissers

# Variables

switch ((get-host).Name) {

‘Windows PowerShell ISE Host’ { $vcf_folder = $psISE.CurrentFile.FullPath -replace ($psISE.CurrentFile.DisplayName,””) }

‘ConsoleHost‘ { $vcf_folder = $myInvocation.MyCommand.Path -replace ($myInvocation.MyCommand.Name,”“) }

‘Visual Studio Code Host‘{ $vcf_folder = $psEditor.GetEditorContext().CurrentFile.Path | Split-Path }

}

# Go to the VCF folder

Set-Location $vcf_folder

$Credential = Get-Credential –UserName “administrator@vsphere.local” -Message “vCenter Login Creds”

# Shutdown Test Run

.\vcf_services_runtime_shutdown.ps1 –DryRun –NodeIp <your_node_here> –Credential $Credential

# Shutdown Run

.\vcf_services_runtime_shutdown.ps1 –DryRun –NodeIp <your_node_here>  –Credential $Credential

Authenticate Securely

• Use Get-Credential instead of storing passwords in the script.

Perform a Dry Run

• Run the script with -DryRun to validate connectivity, authentication, environment configuration and cluster discovery without shutting down services.

.

Execute the Shutdown

• Connect to vCenter.
• Discover the Services Runtime cluster.
• Validate runtime nodes.
• Gracefully terminate Kubernetes workloads.
• Shut down runtime nodes.
• Verify successful completion.

Expected Result

• All VCF Services Runtime virtual machines are powered off in the correct order and the console reports a successful shutdown.

.

.

Conclusion

The PowerShell implementation follows the same workflow as Broadcom’s Linux script while allowing Windows administrators to perform the shutdown natively from Windows without WSL.

What’s Next?

• Automated startup & shutdown script for VCF 9.1 environment

VCF 9.1: Fixing Root Account Password Expiration Issues

.

One of the first post-deployment tasks after installing VMware Cloud Foundation (VCF) 9.1 is configuring a password policy to ensure compliance with your organization’s security standards.



While the password policy is successfully applied to most managed accounts, you may notice that the root accounts of the VCF Operations appliance and the VCF Proxy appliance do not follow the configured password expiration policy.

As a result, the expiration date shown in VCF Management remains unchanged, even though a password policy has been configured.

Symptoms

You may observe one or more of the following:

• A password policy is configured successfully in VCF Management.
• Compliance checks complete without errors.
• The root account of the VCF Operations appliance still shows an incorrect or outdated password expiration date.
• The same behavior occurs on the VCF Proxy appliance.

This can be confusing because the password policy appears to be configured correctly, but it is not enforced for these Linux root accounts.

Why Does This Happen?

The password policy configured in VCF does not automatically update the Linux root account password aging settings on the VCF Operations and Proxy appliances.

Instead, these appliances continue to rely on the Linux ‘chage’ configuration to determine when the root password expires.

VMware has documented this behavior and provided a straightforward workaround.
https://knowledge.broadcom.com/external/article/441344/configured-password-policy-is-not-being.html

Resolution

Before making any changes, enable SSH access on the VCF Operations appliance if it is currently disabled.
https://knowledge.broadcom.com/external/article/315976/enabling-ssh-access-in-aria-operations.html

Step 1 – Connect to the VCF Operations Appliance

SSH to the VCF Operations appliance using an administrative account.

ssh admin@<vcf-operations-appliance>

Step 2 – Verify the Current Password Expiration

Run:

sudo chage -l root

Step 3 – Configure the Password Expiration

Configure a 365-day password lifetime:

sudo chage -M 365 root

Step 4 – Verify the Change

Run:

chage -l root

Step 5 – Repeat for the VCF Proxy Appliance

Repeat the same commands on the VCF Proxy appliance, either through SSH (if enabled) or via the VMware console.

Wait for VCF to Update

Notes

The updated password expiration date is not reflected immediately in the VCF Management interface.

VCF periodically refreshes password information, so it may take 10–30 minutes (or longer depending on your environment) before the new expiration date appears.

.

• This change only affects the Linux root account.
• Verify the setting after appliance upgrades.
• Adjust the password lifetime (90, 180, 365 days, etc.) according to your security policy.

Conclusion

Although VCF 9.1 allows administrators to centrally configure password policies, the Linux root accounts on the VCF Operations and VCF Proxy appliances continue to rely on the local Linux password aging configuration.

Updating the password expiration with the chage command ensures that the root account complies with your organization’s password policy. Once VCF completes its next inventory synchronization, the correct expiration date is displayed in the VCF Management interface.

VCF 9.0 Automate VMware Cloud Foundation Startup and Shutdown with PowerCLI

Powering your VMware Cloud Foundation “lab” environment on and off shouldn’t be a manual process.

A complete shutdown of a VMware Cloud Foundation (VCF) environment is uncommon, but for some energy savings and some time you does not use your lab often, you want a repeatable, reliable, and automated procedure, manually powering dozens of virtual machines in the correct order is both time-consuming and error-prone.

To solve this problem, I created two lightweight PowerCLI scripts:

• VCF 9.0 Small Startup Script
• VCF 9.0 Small Shutdown Script

.

vCenter and ESX hosts are manual (For vSAN cluster I did not find the correct code yet!)
Let me know if you have any questions or addons

Both scripts are available on GitHub and are designed to automate the startup and shutdown of a VMware Cloud Foundation management domain.

GitHub Repository

.

Note: Both Scripts do not work with VCF 9.1!!!!

.

Why These Scripts?

Although VMware Cloud Foundation automates the deployment and lifecycle of the platform, a full platform shutdown still requires the administrator to respect service dependencies.

For example:

• Domain Controllers must be available before authentication works.
• DNS must be online before many VMware services can resolve hostnames.
• vCenter must be operational before SDDC Manager can communicate with the infrastructure.
• NSX components depend on both networking and vCenter.
• VCF services should only start after the management platform is healthy.

Powering everything on simultaneously often results in services that need additional time—or even manual intervention—to recover.

These scripts automate the entire sequence descripted als following:

Start the Management Domain (VCF 9.0)’

Shut Down the Management Domain (VCF 9.0)

Typical Use Cases

These scripts are useful in many environments, including:

• Home labs
• Demonstration environments
• Disaster Recovery testing
• UPS maintenance
• Complete datacenter power outages
• Scheduled maintenance windows
• Hardware replacements

I personally use them in my VCF lab, where powering the environment up or down manually became repetitive and unnecessarily time-consuming. Automating the sequence not only saves time but also ensures a consistent and predictable startup every time.

Customizing the Scripts

Every VMware Cloud Foundation deployment is different.

The scripts are intentionally straightforward so you can easily adapt them by:

• Changing the startup order
• Adding custom virtual machines
• Removing components you don’t use
• Increasing wait times
• Adding health checks
• Integrating notifications
• Extending the logging

Because everything is written in PowerCLI, modifications are simple and require only basic scripting knowledge.

Future Improvements

Some ideas I’m considering for future releases include:

• Automatic dependency discovery
• Email notifications
• Automatic service validation
• Parallel startup where dependencies allow

Contributions and suggestions from the community are always welcome.

Lessons Learned

During development I discovered:

• VMware Tools are the best indicator that a guest OS is ready.
• Fixed sleep timers are unreliable because boot times vary.
• Starting all VMs simultaneously doesn’t necessarily reduce the total startup time.
• Graceful shutdowns significantly reduce recovery issues.
• Simplicity makes the scripts easier to customize.

Download

You can download the latest version from my public GitHub repository:

You can also find more VMware Cloud Foundation automation projects, PowerCLI scripts, and lab guides on my website:

Conclusion

A VMware Cloud Foundation environment consists of many interconnected services, and those services should be started and stopped in the correct order.

These lightweight PowerCLI scripts automate that process, making startup and shutdown predictable, repeatable, and significantly less error-prone. Whether you’re running a production management domain or a small VCF lab, automating these operational tasks saves time and reduces the risk of mistakes.

If you have ideas for improvements or additional features, feel free to open an issue or submit a pull request on GitHub. Happy automating!

.

Config a VCF (vSAN ESA) host the Easy Way

A while ago i created: 

V1: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

V2: Config a VCF (vSAN ESA) host the Easy Way

With the release of VCF 9.1 is now time for again a updated version!

What does the script now:

âś… Disable ipv6

âś… Set DNS domain name

âś… Rename local datastore

âś…Configure NTP

âś… MTU 9000

✅ Installs the vSAN ESA Hardware Mock VIB &

âś… Workaround to reduce impact of resync traffic in vSAN ESA clusters utilizing a 10G network

✅ Installs the Synology NFS Plug-in for VMware VAAI

âś… Memory Optimalisation Additional Transparent Page Sharing management capabilities and new default settings

 ✅ AMD Zen4/Zen5 IPMI Thermal Driver for ESX AMD Zen4/Zen5 IPMI Thermal Driver for ESX Fling

✅ Generate new certificate on the ESXi host (for the VCF verification check)

âś… Ask are you running Miniforum MS-A2(AMD) host & Optimalization see
VCF 9.1 – Comprehensive ESX Configuration Workarounds for Lab Deployments (Except the vSAN Compression Algorithm)

 ✅ Enable Memory Tiering for 9.1 and 9.0 and older (Filter OS Disk)

vSAN ESA Mock and the AMD Zen4/Zen5 IPMI Thermal Driver can be download on the Broadcom Fling page.

(https://support.broadcom.com/group/ecx/free-downloads) section and select Flings (https://support.broadcom.com/group/ecx/productdownloads?subfamily=Flings&freeDownloads=true)

You need to download the vibs separately!
For the installs put the vib’s in the same map as the script
.

You can download the script: HERE

.

Managing VCF Automation Resource Utilization Using CPU Limits

My VCF 9.1 Lab environment is also cpu constraint.
To free up some cpu resources I shutdown VCF automation regularly.

I installed path release https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/release-notes/patch-releases-9-1-0-x/vcf-operations/9-1-0-0100.html

Booting VCF Automation takes al lot of resources. I see regularly spikes. See screenshot below.

.

.

Normally VCF automation runs between 8 – 10Ghz Which is normal and fine in a 3 Node MS-A2 setup.

To make my Lab environment a little bit safer i limit VCF automation till 25000mhz.
This keeps my MS-A2 a little cooler and ensures that the node does max out. 

VCF 9.1 What fixed the stuck deployment at “Deploy and Configure VCF Management Platform”

Overview

While deploying VMware Cloud Foundation (VCF) 9.1 in a homelab environment, the installation repeatedly failed during the ‘Deploy and Configure VCF Management Platform’ stage. Despite performing nine completely clean installations, the deployment consistently stopped at the same point.

Error Observed

The deployment task failed with the message: ‘Add VM Name Prefix to NSX Firewall Exclusion List’. The failure was identified in /var/log/vmware/vcf/domainmanager/domainmanager.log

Initial Research

Several Broadcom Knowledge Base articles appeared relevant, including KB440449 and KB 441122. Although the symptoms were similar, neither article fully resolved the issue.

VMSP Configuration Review

The original VMSP configuration used a name value matching the prefix of the fleetFqdn. The configuration was modified to use a unique VMSP cluster name. While this appeared promising, the issue persisted.

Additional Troubleshooting

Additional troubleshooting included changing VMSP IP ranges, rebuilding DNS records, validating forward and reverse DNS resolution, and reviewing deployment logs for networking issues.

Root Cause Analysis

The issue was ultimately not caused by the NSX firewall exclusion configuration. Multiple infrastructure issues contributed to deployment instability.

Resolution

1. Configure a single authoritative NTP source, preferably the domain controller.
See the planning and preparation workbook
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/planning-and-preparation.html
2. Verify DNS records and name resolution.
3. Upgraded to a dedicated 10G Switch Ubiquiti UniFi Pro XG 8 PoE ipv 2.5GB Ubiquity Switch
Switch Pro XG 8 PoE - Ubiquiti Store Europe
4. Replace faulty network components.

Conclusion

Although the deployment failure appeared to indicate an NSX firewall exclusion issue, the underlying cause was network instability combined with infrastructure configuration problems. After correcting NTP configuration, validating DNS, upgrading network connectivity, and replacing the defective SFP+ module, the VCF 9.1 deployment completed successfully.

Translate »