VCF 9.1 Critical JV Certificate Caching Bug Can Cause Silent Outage

A field experience with VCF Operations and Fleet Lifecycle Management

The problem I encountered

Over the last few weeks, I encountered a strange issue in VMware Cloud Foundation 9.1. The Build > Software and Build > Lifecycle functionality was no longer working correctly.

Figure 1. Lifecycle functionality failing in the VCF interface.

My first thought was that something had gone wrong with the VCF Services Runtime control plane. I did not immediately know why it had failed, and unfortunately I did not have a VM-level backup of the VCF Services virtual machines available.

After investigating the symptoms, I suspected that I had run into a VCF 9.1 certificate-related bug.

Update: critical certificate caching issue

VCF 9.1 contains a critical issue in which the Java Virtual Machine (JVM) caches an internal security certificate during the initial bootstrap.

Issue details

Although the system correctly rotates the certificate in the background, the application does not dynamically reload the renewed certificate. Because these certificates have a strict 90-day validity period, environments deployed at General Availability (May 12, 2026) can begin experiencing silent outages around August 10, 2026.

Impact

The failure can occur without advance health-check warnings. Once the 90-day certificate threshold is reached, the environment may experience:

• Generic “503 Service Unavailable” errors

• Loss of access to affected UI functionality and blockage of component deployments

• Failure of day-two operations within Fleet Lifecycle Management (Fleet LCM)

What this looked like in my environment

In my case, the symptoms appeared in the VCF Build and Lifecycle areas and initially looked like a failure of the VCF Services Runtime control plane. The certificate-caching issue provides a plausible explanation for this behavior.

Action required

If you are running VCF 9.1, review the relevant Broadcom Knowledge Base articles and apply the documented workaround for the affected component. The source document identifies the following KB topics:

• VCF Operations: Error: “transport failure reason: TLS_error” when loading Lifecycle tab in VCF 9.1

• VCF Automation: Error: “transport failure reason: TLS_error” in VCF Automation UI

Takeaway

A certificate can be successfully rotated on the platform while an application continues using an older certificate cached by its JVM. That makes this issue particularly difficult to recognize: certificate rotation may appear healthy even though application communication eventually starts failing.

For VCF 9.1 environments approaching or exceeding 90 days since deployment, certificate-related TLS errors and unexplained 503 responses should therefore be investigated promptly.

VCF LOG Deployment changed in 9.1.0.400

Since VCF 9.1.0.400, a lot has changed in the platform and one of the more noticeable shifts is how log management works. It is no longer a standalone virtual appliance.

Instead, it now runs as a service inside the VCF Management Services Cluster, the same Kubernetes-based runtime that also hosts Fleet Management and the other core management components.

Additionally, in this blog I walk through how to deploy VCF Log Management in 9.1.0.400.

What Changed

In VCF 9.0 and earlier, Log Management (formerly Aria Operations for Logs) was deployed as its own appliance VM or appliance cluster. Additionally, from 9.1.0.400 onward, it is a built-in, containerized core service that runs natively on the VCF Management Services Platform. Deployment, lifecycle management, configuration, access control, and alerting are now all handled centrally through VCF Operations, rather than through a separate product UI.

Prerequisites

• DNS: A DNS-registered FQDN (with both forward and reverse lookup working)

• Capacity in the VCF Services Runtime: Deploying Log Management can trigger an automatic resize of the underlying Kubernetes cluster, so plan for the additional compute ahead of time.

• Sizing decision: The deployment size you choose determines how much compute and storage each node gets.

• Networking: Deployment currently supports VLAN-backed networks for the VCF Management Services; overlay networking is not yet supported for this component.

Resource Requirements for Log Management Deployments.

Size	vCPUs per Node	Memory per Node	Storage per Node	Number of Nodes
Small	8	16 GB	575 GB	1-19
Medium	24	48 GB	575 GB	3-19
Large	32	64 GB	575 GB	3-19

Procedure

1.Log in to the VCF Operations user interface at https://<vcf_operations_fqdn

2.Navigate to Build Lifecycle VCF Management.

3.On the Components tab, select Add Component Log management and configure the parameters for the deployment.

Now click Next to proceed to the Summary page for final review before deployment. Click Finish to start the deployment. As soon as the Deployment is done, you will see Log Management is running under VCF Operations -> Build -> Lifecycle.

Additionally, you can find the logs under VCF Operations -> Operate -> Log..

If you click on Log Sources, you will get an overview of the currently configured integrations, but this does not mean, that the Log collection itself is enabled.

Additionally, navigate to VCF Operations -> Operate -> Administration -> Configurations -> Log Collection to check Status. However, under Status, the Log collection is not enabled.

If you click on the “i” under Status you will see:

This is a little bit strange in my opinion, but yes, you have to switch back to the Integrations page inside VCF Operations Operate Administration Integrations and Edit your VMware Cloud Foundation Account:

From the Account page switch to Domains:

Next, on the Domain Page, start with vCenter to activate Logs Collection by clicking ‘Activate Log Collection’ and SAVE.

Same procedure i.e. NSX:

Additionally, after activiation of Log collection you will see more Logs inside VCF Operations:

Next, go to VCF Operations Operate Administration Configuration Log Collection.
Next, click Edit and scroll down.

Additionally, enable (Edit Log collection configuration).

.

What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario

After three great sessions at VMUG Connect Amsterdam and VMUG Connect Online, and VMUGNL Summer Sessions. I’ve had requests to share the slides for my session “What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario“.

The response was great.

In the slide deck you will find:

✅ The Start of My VCF Story
✅ First Challenge
✅ Why i wanted to build a real Senario
✅ Why i choise going to a build a real VCF scenario
✅ The Road
✅ The Lessions Learned & Next Steps

Building and Optimizing a Minimal 2-Node VMware Cloud Foundation 9 Lab Deploymentfrom Ward Vissers

VCF 9.1: Fixing Root Account Password Expiration Issues

One of the first post-deployment tasks after installing VMware Cloud Foundation (VCF) 9.1 is configuring a password policy to ensure compliance with your organization’s security standards.

While the password policy is successfully applied to most managed accounts, you may notice that the root accounts of the VCF Operations appliance and the VCF Proxy appliance do not follow the configured password expiration policy.

As a result, the expiration date shown in VCF Management remains unchanged, even though a password policy has been configured.

Symptoms

You may observe one or more of the following:

• A password policy is configured successfully in VCF Management.
• Compliance checks complete without errors.
• The root account of the VCF Operations appliance still shows an incorrect or outdated password expiration date.
• The same behavior occurs on the VCF Proxy appliance.

This can be confusing because the password policy appears to be configured correctly, but it is not enforced for these Linux root accounts.

Why Does This Happen?

The password policy configured in VCF does not automatically update the Linux root account password aging settings on the VCF Operations and Proxy appliances.

Instead, these appliances continue to rely on the Linux ‘chage’ configuration to determine when the root password expires.

VMware has documented this behavior and provided a straightforward workaround.
https://knowledge.broadcom.com/external/article/441344/configured-password-policy-is-not-being.html

Resolution

Before making any changes, enable SSH access on the VCF Operations appliance if it is currently disabled.
https://knowledge.broadcom.com/external/article/315976/enabling-ssh-access-in-aria-operations.html

Step 1 – Connect to the VCF Operations Appliance

SSH to the VCF Operations appliance using an administrative account.

ssh admin@<vcf-operations-appliance>

Step 2 – Verify the Current Password Expiration

Run:

sudo chage -l root

Step 3 – Configure the Password Expiration

Configure a 365-day password lifetime:

sudo chage -M 365 root

Step 4 – Verify the Change

Run:

chage -l root

Step 5 – Repeat for the VCF Proxy Appliance

Repeat the same commands on the VCF Proxy appliance, either through SSH (if enabled) or via the VMware console.

Wait for VCF to Update

Notes

The updated password expiration date is not reflected immediately in the VCF Management interface.

VCF periodically refreshes password information, so it may take 10–30 minutes (or longer depending on your environment) before the new expiration date appears.

• This change only affects the Linux root account.
• Verify the setting after appliance upgrades.
• Adjust the password lifetime (90, 180, 365 days, etc.) according to your security policy.

Conclusion

Although VCF 9.1 allows administrators to centrally configure password policies, the Linux root accounts on the VCF Operations and VCF Proxy appliances continue to rely on the local Linux password aging configuration.

Updating the password expiration with the chage command ensures that the root account complies with your organization’s password policy. Once VCF completes its next inventory synchronization, the correct expiration date is displayed in the VCF Management interface.

VCF 9.1 What fixed the stuck deployment at “Deploy and Configure VCF Management Platform”

Overview

While deploying VMware Cloud Foundation (VCF) 9.1 in a homelab environment, the installation repeatedly failed during the ‘Deploy and Configure VCF Management Platform’ stage. Despite performing nine completely clean installations, the deployment consistently stopped at the same point.

Error Observed

The deployment task failed with the message: ‘Add VM Name Prefix to NSX Firewall Exclusion List’. The failure was identified in /var/log/vmware/vcf/domainmanager/domainmanager.log

Initial Research

Several Broadcom Knowledge Base articles appeared relevant, including KB440449 and KB 441122. Although the symptoms were similar, neither article fully resolved the issue.

VMSP Configuration Review

The original VMSP configuration used a name value matching the prefix of the fleetFqdn. The configuration was modified to use a unique VMSP cluster name. While this appeared promising, the issue persisted.

Additional Troubleshooting

Additional troubleshooting included changing VMSP IP ranges, rebuilding DNS records, validating forward and reverse DNS resolution, and reviewing deployment logs for networking issues.

Root Cause Analysis

The issue was ultimately not caused by the NSX firewall exclusion configuration. Multiple infrastructure issues contributed to deployment instability.

Resolution

1. Configure a single authoritative NTP source, preferably the domain controller.
See the planning and preparation workbook
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/planning-and-preparation.html
2. Verify DNS records and name resolution.
3. Upgraded to a dedicated 10G Switch Ubiquiti UniFi Pro XG 8 PoE ipv 2.5GB Ubiquity Switch
Switch Pro XG 8 PoE - Ubiquiti Store Europe
4. Replace faulty network components.

Conclusion

Although the deployment failure appeared to indicate an NSX firewall exclusion issue, the underlying cause was network instability combined with infrastructure configuration problems. After correcting NTP configuration, validating DNS, upgrading network connectivity, and replacing the defective SFP+ module, the VCF 9.1 deployment completed successfully.

Upgrade VCF to 9.0.1: Setting Up an Offline Depot on Ubuntu

Upgrade VMware VCF 9.0.1: Essential Setting Up an Offline Depot on Ubuntu Instructions

If you are planning to upgrade to latest release of VMware Cloud Foundation (VCF) 9.0.1 and you what to install/upgrade you need to have a offline depot.

I had some struggle with the VCF Fleet upgrades. I thought the VCF installer Offline Depot was sufficient. I seems not.

William Liam did a nice explaining about the options: VCF Software Depot Structure Deep Dive for Install & Upgrade. Which confirms my struggle.

Sow I build my own offline depot

Prerequisites

First, download the required bundle files from the Broadcom VMware portal. This includes:

vcf-9.0.1.0-offline-depot-metadata.zip (mandatory)
vcf-download-tool-9.0.1.0.24962179.tar.gz (mandatory)

On the Depot Server (Ubuntu)

Sizing for the disk is minimal 100GB I used 200GB thin Provisiond

I did a Ubuntu install on a 200GB disk (Partition without lvm)

Login

Sudo passwd root

Vim /etc/ssh/sshd_config

if the following line exists, possibly commented out (with a # in front):

PermitRootLogin

Then change it to the following, uncommenting if needed (remove the # in front):

PermitRootLogin yes

sudo service ssh restart

now you can login als root

Install Apache and tools

Sudo update

sudo apt install apache2 openssl apache2-utils unzip

Create a certificate config file

nano ~/vcf-openssl.cnf

Paste the following:

[ req ]

default_bits = 2048

prompt = no

default_md = sha256

distinguished_name = dn

x509_extensions = v3_req

[ dn ]

C = US

ST = CA

L = LA

O = TS

OU = IT

CN = flt-depot.wardhomelab.nl

emailAddress = a@b.c

[ v3_req ]

basicConstraints = CA:FALSE

keyUsage = digitalSignature, keyEncipherment

extendedKeyUsage = serverAuth

subjectAltName = @alt_names

[ alt_names ]

DNS.1 = flt-depot.wardhomelab.nl

IP.1 = 192.168.150.246

Replace the values with your organization’s details.

Generate the certificate

cd /etc/apache2
sudo mkdir ssl

sudo openssl req -x509 -nodes -days 365 \

-newkey rsa:2048 \

-keyout /etc/apache2/ssl/vcf.key \

-out /etc/apache2/ssl/vcf.crt \

-config ~/vcf-openssl.cnf

Create a basic auth user

Sudo htpasswd -c /etc/apache2/.htpasswd vcfadmin

Configure Apache

sudo nano /etc/apache2/sites-available/default-ssl.conf

Paste the following:

ServerAdmin webmaster@localhost

DocumentRoot /var/www/html

SSLEngine on

SSLCertificateFile /etc/apache2/ssl/vcf.crt

SSLCertificateKeyFile /etc/apache2/ssl/vcf.key

SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1

RequestHeader unset Proxy early

Options Indexes FollowSymLinks

AllowOverride None

AuthType Basic

AuthName “VCF Depot”

AuthUserFile /etc/apache2/.htpasswd

Require valid-user

</Directory>

</VirtualHost>

Enable modules and restart Apache

sudo a2enmod ssl headers

sudo a2ensite default-ssl

sudo systemctl restart apache2

Extract the metadata ZIP

Sudo chmod 777 /home

Upload vcf-9.0.1.0-offline-depot-metadata.zip /home/ with Winscp

Sudo unzip vcf-9.0.1.0-offline-depot-metadata.zip -d /var/www/html

Make sure this file exists:

ls /var/www/html/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json

Remove Index.html

sudo rm -f /var/www/html/index.html

Create your Download token

Create your Download token in de Broadcom portal

Create a token file in /home folder

Upload the VCF download tool

Sudo mkdir /home/vcf-download-tool

Sudo chmod 777 /home/vcf-download-tool

cd /tmp/vcf-download-tool

Upload the tool in /tmp/vcf-download-tool folder

tar -xf vcf-download-tool-9.0.1.0.24962179.tar.gz

sudo reboot

Run the following

Cd /tmp/vcf-download-tool/bin

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –automated-install –type=INSTALL

Download all the appliances

sudo ./vcf-download-tool binaries list –depot-download-token-file=/home/token –vcf-version=9.0.1 –type=INSTALL –sku=VCF

Upgrade

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component SDDC_MANAGER_VCF

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VCENTER

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRSLCM

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VROPS

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component NSX_T_MANAGER

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VCF_OPS_CLOUD_PROXY

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRA

Additional

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRNI

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRLI

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component HCX

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRO

sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VIDB

Upgrade Binaries will be visible

On de SDDC manager

Open de VM console

Vim /etc/ssh/sshd_config

if the following line exists, possibly commented out (with a # in front):

PermitRootLogin

Then change it to the following, uncommenting if needed (remove the # in front):

PermitRootLogin yes

systemctl restart sshd

Copy vcf.crt

scp root@ftt-depot.wardhomelab.nl:/etc/apache2/ssl/vcf.crt /tmp/vcf.crt

Import the certificate into the Java truststore

sudo keytool -import -trustcacerts -alias vcfDepotCert1 \

-file /tmp/vcf.crt \

-keystore /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts \

-storepass changeit

Reboot

Now u should connect VCF to you offline depot

You want you upgrade to 9.0.1 ivm Edge Issue

Special thanks to vmtechnics for putting me in the right direction

Top VCF 9 Updates: Installer, NVME, and More

What are my things I would like to test

VCF 9 installer (VCF 9 Beta i looked good)
NVME Tiering
vSAN ESA Dedub
VCF 9 with Ubiquiti
Kubernetes Service now includes Windows containerization
NSX VPC Support

The VCF Cloud Foundation Installer makes lives a lot easier! More about this coming very soon

The VCF Operations Console is looking good! I used it in the VCF 9 beta

More about this also later!

Easy Script to Create DNS Records in VCF Lab

When you build your VCF Lab environment you want to create your DNS records automatically. I use for DNS a Windows Server.

The Script:

function ConvertTo-DecimalIP {
param ([string]$ip)
$parts = $ip.Split(‘.’) | ForEach-Object { [int]$_ }
return ($parts[0] -shl 24) + ($parts[1] -shl 16) + ($parts[2] -shl 8) + $parts[3]
}

function ConvertTo-DottedIP {
param ([int]$intIP)
$part1 = ($intIP -shr 24) -band 0xFF
$part2 = ($intIP -shr 16) -band 0xFF
$part3 = ($intIP -shr 8) -band 0xFF
$part4 = $intIP -band 0xFF
return “$part1.$part2.$part3.$part4”
}

$zone = “testlab.nl”
$startip = “192.168.200.10”

$dnsrecords = “vcf-m01-cb01″,”vcf-m01-sddcm01″,”vcf-m01-esx01″,”vcf-m01-esx02″,”vcf-m01-esx03″,”vcf-m01-esx04″,”vcf-w01-esx02″,”vcf-w01-esx03″,”vcf-w01-esx04″,”vcf-w01-esx04″,”vcf-m01-nsx01a”,”vcf-m01-nsx01b”,”vcf-m01-nsx01c”,”vcf-m01-nsx01″,”vcf-w01-nsx01a”,”vcf-w01-nsx01b”,”vcf-w01-nsx01c”,”vcf-w01-nsx01″,”vcf-m01-vc01″,”vcf-w01-vc01″
$count = $dnsrecords.count

# Convert start IP to decimal

$decimalIP = ConvertTo-DecimalIP $startIP
$i = 0

# Loop and print incremented IPs

foreach ($dnsrecord in $dnsrecords) {
$i -lt
$count;
$i++
$currentDecimalIP = $decimalIP + $i
$currentIP = ConvertTo-DottedIP $currentDecimalIP
Add-DnsServerResourceRecordA -Name $dnsrecord -ZoneName $zone -AllowUpdateAny -IPv4Address $currentIP -CreatePtr
Write-Output “DNS record $dnsrecord in $zone with $currentIP is created” -ForegroundColor Green

The problem I encountered

Update: critical certificate caching issue

Issue details

Impact

What this looked like in my environment

Action required

Takeaway

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Symptoms

Why Does This Happen?

Resolution

Step 1 – Connect to the VCF Operations Appliance

Step 2 – Verify the Current Password Expiration

Step 3 – Configure the Password Expiration

Step 4 – Verify the Change

Step 5 – Repeat for the VCF Proxy Appliance

Wait for VCF to Update

Notes

Conclusion

Share this:

Like this:

Overview

Error Observed

Initial Research

VMSP Configuration Review

Additional Troubleshooting

Root Cause Analysis

Resolution

Share this:

Like this:

Prerequisites

On the Depot Server (Ubuntu)

Login

Install Apache and tools

Create a certificate config file

Create a basic auth user

Configure Apache

Enable modules and restart Apache

Extract the metadata ZIP

Remove Index.html

Create your Download token

Upload the VCF download tool

Run the following

Download all the appliances

Upgrade

Additional

On de SDDC manager

Copy vcf.crt

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: