A field experience with VCF Operations and Fleet Lifecycle Management

The problem I encountered

Over the last few weeks, I encountered a strange issue in VMware Cloud Foundation 9.1. The Build > Software and Build > Lifecycle functionality was no longer working correctly.

Figure 1. Lifecycle functionality failing in the VCF interface.

My first thought was that something had gone wrong with the VCF Services Runtime control plane. I did not immediately know why it had failed, and unfortunately I did not have a VM-level backup of the VCF Services virtual machines available.

After investigating the symptoms, I suspected that I had run into a VCF 9.1 certificate-related bug.

Update: critical certificate caching issue

VCF 9.1 contains a critical issue in which the Java Virtual Machine (JVM) caches an internal security certificate during the initial bootstrap.

Issue details

Although the system correctly rotates the certificate in the background, the application does not dynamically reload the renewed certificate. Because these certificates have a strict 90-day validity period, environments deployed at General Availability (May 12, 2026) can begin experiencing silent outages around August 10, 2026.

Impact

The failure can occur without advance health-check warnings. Once the 90-day certificate threshold is reached, the environment may experience:

• Generic “503 Service Unavailable” errors

• Loss of access to affected UI functionality and blockage of component deployments

• Failure of day-two operations within Fleet Lifecycle Management (Fleet LCM)

What this looked like in my environment

In my case, the symptoms appeared in the VCF Build and Lifecycle areas and initially looked like a failure of the VCF Services Runtime control plane. The certificate-caching issue provides a plausible explanation for this behavior.

Action required

If you are running VCF 9.1, review the relevant Broadcom Knowledge Base articles and apply the documented workaround for the affected component. The source document identifies the following KB topics:

• VCF Operations: Error: “transport failure reason: TLS_error” when loading Lifecycle tab in VCF 9.1

• VCF Automation: Error: “transport failure reason: TLS_error” in VCF Automation UI

Takeaway

A certificate can be successfully rotated on the platform while an application continues using an older certificate cached by its JVM. That makes this issue particularly difficult to recognize: certificate rotation may appear healthy even though application communication eventually starts failing.

For VCF 9.1 environments approaching or exceeding 90 days since deployment, certificate-related TLS errors and unexplained 503 responses should therefore be investigated promptly.