What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario

After three great sessions at VMUG Connect Amsterdam and VMUG Connect Online, and VMUGNL Summer Sessions. I’ve had requests to share the slides for my session “What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario“.

The response was great.

In the slide deck you will find:

✅ The Start of My VCF Story
✅ First Challenge
✅ Why i wanted to build a real Senario
✅ Why i choise going to a build a real VCF scenario
✅ The Road
✅ The Lessions Learned & Next Steps

VCF 9.1 How to Safely Shut Down the Services Runtime Cluster Using PowerShell

Broadcom provides an official procedure for gracefully shutting down a VMware Cloud Foundation (VCF) 9.1 Services Runtime cluster. The provided solution uses a Linux shell script (vcf_services_runtime_shutdown.sh), which works well on Linux systems but is less convenient for Windows administrators.

Rather than using Windows Subsystem for Linux (WSL) or a separate Linux machine, I converted the official shell script into a native PowerShell version. This allows the entire shutdown procedure to be executed directly from a Windows workstation while following the same workflow as the original Broadcom script.

Downloads

📦 VCF 9.1 Shutdown VCF Services VMs.ps1
📦 vcf_services_runtime_shutdown.ps1

📄 Original Broadcom Knowledge Base vcf_services_runtime_shutdown.sh

Why Use the Official Shutdown Procedure?

 Kubernetes workloads are gracefully terminated.
 Running services are stopped in the correct order.
 Databases remain in a consistent state.
 Cluster metadata is preserved.
 The shutdown process completes without leaving services in an inconsistent state.

Prerequisites

 VMware Cloud Foundation 9.1
 PowerShell 7.x (recommended)
 Latest VCF PowerCLI
 vCenter Administrator credentials
 Network connectivity to the Control Plane node

Before You Begin

 Power off any VCF Automation virtual machines before shutting down the Services Runtime cluster to avoid interference with automation tasks.

Determine the Control Plane Node IP Address

 Open VCF Operations.
 Navigate to Build → Lifecycle → Components.
 Select VCF Services Runtime.
 Scroll to the Nodes section.
 Locate a Control Plane node and note its IP address.

PowerShell Wrapper Script

 Detects the current script location.
 Changes to the correct working directory.
 Calls the shutdown script with the required parameters.
 Simplifies execution.

Download Wrapper Script

Download: Shutdown VCF Services.ps1

# VCF Services shutdown by Ward Vissers

# Variables

switch ((get-host).Name) {

‘Windows PowerShell ISE Host’ { $vcf_folder = $psISE.CurrentFile.FullPath -replace ($psISE.CurrentFile.DisplayName,””) }

ConsoleHost{ $vcf_folder = $myInvocation.MyCommand.Path -replace ($myInvocation.MyCommand.Name,”“) }

‘Visual Studio Code Host‘{ $vcf_folder = $psEditor.GetEditorContext().CurrentFile.Path | Split-Path }

}

# Go to the VCF folder

Set-Location $vcf_folder

$Credential = Get-Credential –UserNameadministrator@vsphere.local” -Message “vCenter Login Creds”

# Shutdown Test Run

.\vcf_services_runtime_shutdown.ps1 –DryRunNodeIp <your_node_here> –Credential $Credential

# Shutdown Run

.\vcf_services_runtime_shutdown.ps1 –DryRunNodeIp <your_node_here>  –Credential $Credential

Authenticate Securely

 Use Get-Credential instead of storing passwords in the script.

Perform a Dry Run

 Run the script with -DryRun to validate connectivity, authentication, environment configuration and cluster discovery without shutting down services.

.

Execute the Shutdown

 Connect to vCenter.
 Discover the Services Runtime cluster.
 Validate runtime nodes.
 Gracefully terminate Kubernetes workloads.
 Shut down runtime nodes.
 Verify successful completion.

Expected Result

 All VCF Services Runtime virtual machines are powered off in the correct order and the console reports a successful shutdown.

.

.

Conclusion

The PowerShell implementation follows the same workflow as Broadcom’s Linux script while allowing Windows administrators to perform the shutdown natively from Windows without WSL.

What’s Next?

 Automated startup & shutdown script for VCF 9.1 environment

VCF 9.1: Fixing Root Account Password Expiration Issues

.

One of the first post-deployment tasks after installing VMware Cloud Foundation (VCF) 9.1 is configuring a password policy to ensure compliance with your organization’s security standards.



While the password policy is successfully applied to most managed accounts, you may notice that the root accounts of the VCF Operations appliance and the VCF Proxy appliance do not follow the configured password expiration policy.


As a result, the expiration date shown in VCF Management remains unchanged, even though a password policy has been configured.

Symptoms

You may observe one or more of the following:

• A password policy is configured successfully in VCF Management.

• Compliance checks complete without errors.

• The root account of the VCF Operations appliance still shows an incorrect or outdated password expiration date.

• The same behavior occurs on the VCF Proxy appliance.


This can be confusing because the password policy appears to be configured correctly, but it is not enforced for these Linux root accounts.

Why Does This Happen?

The password policy configured in VCF does not automatically update the Linux root account password aging settings on the VCF Operations and Proxy appliances.

Instead, these appliances continue to rely on the Linux ‘chage’ configuration to determine when the root password expires.


VMware has documented this behavior and provided a straightforward workaround.

https://knowledge.broadcom.com/external/article/441344/configured-password-policy-is-not-being.html

Resolution

Before making any changes, enable SSH access on the VCF Operations appliance if it is currently disabled.
https://knowledge.broadcom.com/external/article/315976/enabling-ssh-access-in-aria-operations.html

Step 1 – Connect to the VCF Operations Appliance

SSH to the VCF Operations appliance using an administrative account.

ssh admin@<vcf-operations-appliance>

Step 2 – Verify the Current Password Expiration

Run:

sudo chage -l root

Step 3 – Configure the Password Expiration

Configure a 365-day password lifetime:

sudo chage -M 365 root

Step 4 – Verify the Change

Run:

chage -l root

Step 5 – Repeat for the VCF Proxy Appliance

Repeat the same commands on the VCF Proxy appliance, either through SSH (if enabled) or via the VMware console.

Wait for VCF to Update

Notes

The updated password expiration date is not reflected immediately in the VCF Management interface.

VCF periodically refreshes password information, so it may take 10–30 minutes (or longer depending on your environment) before the new expiration date appears.

.

• This change only affects the Linux root account.
• Verify the setting after appliance upgrades.

• Adjust the password lifetime (90, 180, 365 days, etc.) according to your security policy.

Conclusion

Although VCF 9.1 allows administrators to centrally configure password policies, the Linux root accounts on the VCF Operations and VCF Proxy appliances continue to rely on the local Linux password aging configuration.

Updating the password expiration with the chage command ensures that the root account complies with your organization’s password policy. Once VCF completes its next inventory synchronization, the correct expiration date is displayed in the VCF Management interface.

VCF 9.0 Automate VMware Cloud Foundation Startup and Shutdown with PowerCLI

Powering your VMware Cloud Foundation “lab” environment on and off shouldn’t be a manual process.

A complete shutdown of a VMware Cloud Foundation (VCF) environment is uncommon, but for some energy savings and some time you does not use your lab often, you want a repeatable, reliable, and automated procedure, manually powering dozens of virtual machines in the correct order is both time-consuming and error-prone.

To solve this problem, I created two lightweight PowerCLI scripts:

 VCF 9.0 Small Startup Script
 VCF 9.0 Small Shutdown Script

.

vCenter and ESX hosts are manual (For vSAN cluster I did not find the correct code yet!)
Let me know if you have any questions or addons

Both scripts are available on GitHub and are designed to automate the startup and shutdown of a VMware Cloud Foundation management domain.

GitHub Repository

.

Note: Both Scripts do not work with VCF 9.1!!!!

.

Why These Scripts?

Although VMware Cloud Foundation automates the deployment and lifecycle of the platform, a full platform shutdown still requires the administrator to respect service dependencies.

For example:

 Domain Controllers must be available before authentication works.
 DNS must be online before many VMware services can resolve hostnames.
 vCenter must be operational before SDDC Manager can communicate with the infrastructure.
 NSX components depend on both networking and vCenter.
 VCF services should only start after the management platform is healthy.

Powering everything on simultaneously often results in services that need additional time—or even manual intervention—to recover.

These scripts automate the entire sequence descripted als following:

Start the Management Domain (VCF 9.0)

Shut Down the Management Domain (VCF 9.0)

Typical Use Cases

These scripts are useful in many environments, including:

 Home labs
 Demonstration environments
 Disaster Recovery testing
 UPS maintenance
 Complete datacenter power outages
 Scheduled maintenance windows
 Hardware replacements

I personally use them in my VCF lab, where powering the environment up or down manually became repetitive and unnecessarily time-consuming. Automating the sequence not only saves time but also ensures a consistent and predictable startup every time.

Customizing the Scripts

Every VMware Cloud Foundation deployment is different.

The scripts are intentionally straightforward so you can easily adapt them by:

 Changing the startup order
 Adding custom virtual machines
 Removing components you don’t use
 Increasing wait times
 Adding health checks
 Integrating notifications
 Extending the logging

Because everything is written in PowerCLI, modifications are simple and require only basic scripting knowledge.

Future Improvements

Some ideas I’m considering for future releases include:

 Automatic dependency discovery
 Email notifications
 Automatic service validation
 Parallel startup where dependencies allow

Contributions and suggestions from the community are always welcome.

Lessons Learned

During development I discovered:

 VMware Tools are the best indicator that a guest OS is ready.
 Fixed sleep timers are unreliable because boot times vary.
 Starting all VMs simultaneously doesn’t necessarily reduce the total startup time.
 Graceful shutdowns significantly reduce recovery issues.
 Simplicity makes the scripts easier to customize.

Download

You can download the latest version from my public GitHub repository:

You can also find more VMware Cloud Foundation automation projects, PowerCLI scripts, and lab guides on my website:

Conclusion

A VMware Cloud Foundation environment consists of many interconnected services, and those services should be started and stopped in the correct order.

These lightweight PowerCLI scripts automate that process, making startup and shutdown predictable, repeatable, and significantly less error-prone. Whether you’re running a production management domain or a small VCF lab, automating these operational tasks saves time and reduces the risk of mistakes.

If you have ideas for improvements or additional features, feel free to open an issue or submit a pull request on GitHub. Happy automating!

.

Config a VCF (vSAN ESA) host the Easy Way

A while ago i created: 

V1: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

V2: Config a VCF (vSAN ESA) host the Easy Way

With the release of VCF 9.1 is now time for again a updated version!

What does the script now:

Disable ipv6

Set DNS domain name

Rename local datastore

Configure NTP

MTU 9000

Installs the vSAN ESA Hardware Mock VIB &

Workaround to reduce impact of resync traffic in vSAN ESA clusters utilizing a 10G network

Installs the Synology NFS Plug-in for VMware VAAI

Memory Optimalisation Additional Transparent Page Sharing management capabilities and new default settings

  AMD Zen4/Zen5 IPMI Thermal Driver for ESX AMD Zen4/Zen5 IPMI Thermal Driver for ESX Fling

 Generate new certificate on the ESXi host (for the VCF verification check)

Ask are you running Miniforum MS-A2(AMD) host & Optimalization see
VCF 9.1 – Comprehensive ESX Configuration Workarounds for Lab Deployments (Except the vSAN Compression Algorithm)

  Enable Memory Tiering for 9.1 and 9.0 and older (Filter OS Disk)

vSAN ESA Mock and the AMD Zen4/Zen5 IPMI Thermal Driver can be download on the Broadcom Fling page.

(https://support.broadcom.com/group/ecx/free-downloads) section and select Flings (https://support.broadcom.com/group/ecx/productdownloads?subfamily=Flings&freeDownloads=true)

You need to download the vibs separately!
For the installs put the vib’s in the same map as the script
.

You can download the script: HERE

.

Managing VCF Automation Resource Utilization Using CPU Limits

My VCF 9.1 Lab environment is also cpu constraint.
To free up some cpu resources I shutdown VCF automation regularly.

I installed path release https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/release-notes/patch-releases-9-1-0-x/vcf-operations/9-1-0-0100.html

Booting VCF Automation takes al lot of resources. I see regularly spikes. See screenshot below.

.

.

Normally VCF automation runs between 8 – 10Ghz Which is normal and fine in a 3 Node MS-A2 setup.

To make my Lab environment a little bit safer i limit VCF automation till 25000mhz.
This keeps my MS-A2 a little cooler and ensures that the node does max out. 

VCF 9.1 What fixed the stuck deployment at “Deploy and Configure VCF Management Platform”

Overview

While deploying VMware Cloud Foundation (VCF) 9.1 in a homelab environment, the installation repeatedly failed during the ‘Deploy and Configure VCF Management Platform’ stage. Despite performing nine completely clean installations, the deployment consistently stopped at the same point.

Error Observed

The deployment task failed with the message: ‘Add VM Name Prefix to NSX Firewall Exclusion List’. The failure was identified in /var/log/vmware/vcf/domainmanager/domainmanager.log

Initial Research

Several Broadcom Knowledge Base articles appeared relevant, including KB440449 and KB 441122. Although the symptoms were similar, neither article fully resolved the issue.

VMSP Configuration Review

The original VMSP configuration used a name value matching the prefix of the fleetFqdn. The configuration was modified to use a unique VMSP cluster name. While this appeared promising, the issue persisted.

Additional Troubleshooting

Additional troubleshooting included changing VMSP IP ranges, rebuilding DNS records, validating forward and reverse DNS resolution, and reviewing deployment logs for networking issues.

Root Cause Analysis

The issue was ultimately not caused by the NSX firewall exclusion configuration. Multiple infrastructure issues contributed to deployment instability.

Resolution

1. Configure a single authoritative NTP source, preferably the domain controller.
See the planning and preparation workbook
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/planning-and-preparation.html
2. Verify DNS records and name resolution.
3. Upgraded to a dedicated 10G Switch Ubiquiti UniFi Pro XG 8 PoE ipv 2.5GB Ubiquity Switch
Switch Pro XG 8 PoE - Ubiquiti Store Europe
4. Replace faulty network components.

Conclusion

Although the deployment failure appeared to indicate an NSX firewall exclusion issue, the underlying cause was network instability combined with infrastructure configuration problems. After correcting NTP configuration, validating DNS, upgrading network connectivity, and replacing the defective SFP+ module, the VCF 9.1 deployment completed successfully.

What’s New in Private Cloud: VMware VCF 9.1 Enhancements

VMware Cloud Foundation (VCF) 9.1 is here — and it’s one of the most feature‑packed releases in years. This update isn’t just incremental; it’s a strategic modernization of compute, storage, networking, security, and operations across the entire private cloud stack.


.

Let’s break down the biggest enhancements and why they I think they matter.

Modernizing Infrastructure Economics with vSphere Foundation 9.1

VCF 9.1 brings several powerful updates to the vSphere layer, aimed at improving performance efficiency and reducing operational overhead.

Enhanced NVMe Memory Tiering

Workloads that demand high throughput and low latency benefit from smarter memory tiering. NVMe-based memory tiers now deliver improved performance and flexibility. (And yes — many are hoping Secure Boot support lands here as well.)


Parallel Processing of DRS vMotion

DRS can now process multiple vMotions in parallel, dramatically reducing cluster balancing times. This is especially impactful in large-scale environments with frequent workload mobility.

Live Patching for TPM-Enabled Hosts

Live patching now works even on hosts with TPM enabled — a huge win for security-conscious organizations that previously had to choose between uptime and compliance.

Networking Updates: Scale, Simplicity, and Smarter Automation

VCF 9.1 introduces major networking enhancements that streamline operations and expand connectivity options.

Enhanced Day-2 VM Lifecycle Management

Networking changes for VMs — including NIC updates, IP changes, and security policies — are now easier and more automated.

Existing VLAN Connectivity via Distributed Transit Gateways

You can now bridge existing VLAN-based networks into VCF environments more seamlessly, reducing migration friction and simplifying hybrid designs.

Streamlined Firewalls & Automated Inter-VPC Security

Security policies between VPCs are now automated, reducing manual rule creation and improving consistency across tenants.

Terraform Provider Enhancements

Better support for tenant-level policy and content management means more automation and cleaner IaC workflows.

Simplified Workload Connectivity & Enhanced Network Scale

EVPN-VXLAN Interoperability

VCF 9.1 now supports EVPN-VXLAN interoperability with the physicalnetwork fabric. This is a major step toward fully integrated, fabric-aware cloud networking.

Network Assessment & VPC Planning

New tools and workflows help architects plan VPC layouts, assess network readiness, and avoid misconfigurations before deployment.

Optimize, Modernize & Protect Storage with vSAN in VCF 9.1

Storage gets a significant upgrade in this release, especially for environments focused on efficiency and resilience.

Encryption for vSAN Global Deduplication

Global dedupe is now compatible with data-at-rest encryption — a long-awaited capability for secure, space-efficient storage.

Enhanced Stretched Cluster Capabilities

Improved resilience and smarter failure handling strengthen business continuity for mission-critical workloads.

Automated Storage Policy Management

Policies now adjust automatically based on cluster configuration changes, reducing manual tuning and risk of misalignment.

Strengthening Zero Trust Security & Platform Resilience

Security is a major theme in VCF 9.1, with improvements across the stack.

Data-at-Rest Encryption for Global Dedupe

This ensures encrypted storage without sacrificing dedupe efficiency — a rare combination in enterprise storage.

Quick Patching for vCenter

Faster patch cycles reduce exposure windows and simplify maintenance.

Live Patching for TPM-Enabled Hosts

As mentioned earlier, this is a major operational win for secure environments.

Continuous Compliance & Integrated Cyber Recovery

VCF 9.1 pushes deeper into automated compliance and recovery workflows.

Compliance Monitoring & Desired State Remediation

The platform now continuously checks VCF components against desired state and can automatically remediate drift.

VPC Policy-Based Connectivity

Security and connectivity policies can now be applied consistently across VPCs, improving governance and reducing misconfigurations.

VMware Data Services Manager 9.1: Modern Databases for AI & Cloud

Microsoft SQL Server 2022 Now GA

SQL Server 2022 is now fully supported and generally available through DSM 9.1, enabling automated lifecycle management for modern database workloads — including those powering AI and analytics.

Want to See It in Action?

VMware has published a full VCF 9.1 video podcast series that dives deeper into the new capabilities:

Enough to do in my Homelab Starting with Upgrade and testing the new features!!

Mitigating Secure Boot Risks in 2026: A Comprehensive Guide

In June 2026 Secure boot certs start to going to expire for physical en virtual machines Servers en Clients. PS not only Windows but also Linux!!

PS. Make sure Client en Servers all installed with latest updates!!

Made a little Risk Assessment:

The expiration and replacement of Microsoft Secure Boot certificates pose a high risk to IT environments. If not properly managed, systems may fail to boot, updates may fail, and security risks may increase. This is particularly critical in automated and virtualized environments.

Key risks:

 Systems failing to boot after updates
 Incompatibility during OS or hypervisor upgrades
 Increased security risks due to outdated certificates

Recommended actions:

1.Update firmware and Secure Boot certificates
2.Test all workloads in a lab environment
3.Update golden images and automation pipelines

A phased rollout and proper validation are essential to prevent disruptions.

1. Scope

This document describes the risks, impact, and mitigations related to the expiration of Microsoft Secure Boot certificates in enterprise environments.

2. Affected Components

 Systems with UEFI firmware (Servers, Desktops, Virtual Machines)
 Microsoft UEFI CA certificates
 Operating Systems (Servers, Clients) (Windows, Linux)
 Automation tools like (Packer, MDT, SCCM)

3. Risk Analysis

Key risks:

 Incompatibility during upgrades
 Security vulnerabilities caused by outdated trust stores
 Errors in automation pipelines
 Firmware incompatibility

4. Risk Matrix

 Upgrade Issues: High
 Security Exposure: High
 Automation Failures: Medium
 Firmware Issues: High

5. Mitigations

 Update firmware on all systems
 Apply Microsoft Secure Boot updates
 Verify Event ID 1808
 Rebuild images with updated certificates
 Perform a phased rollout

6. Validation & Testing

 Test OS boot scenarios
 Validate Secure Boot status
 Verify automation pipelines

7. Conclusion

Changes to Secure Boot certificates must be treated as critical infrastructure updates. Proper preparation, testing, and phased implementation are essential to avoid disruptions.

.Microsoft has released patch’s for the following OS.

Windows 11 (23H2/24H2/25H2)
Windows Server 2016/2019/2022/2025.

VMware is creating a “Fix or Update” for this

* I did not test versions with extended support like Windows 2012 R2 and Windows 10.

Get your list with:
Get-VM | Where-Object { $_.ExtensionData.Config.Firmware -eq “efi” -and

$_.ExtensionData.Config.BootOptions.EfiSecureBootEnabled } | Select-Object Name,

   @{N=”OS”;E={$_.ExtensionData.Guest.GuestFullName}},  PowerState

There is a updated coming from VMware by Broadcom: Check this article:     @{N=”OS”;E={$_.ExtensionData.Guest.GuestFullName}},  PowerState

https://knowledge.broadcom.com/external/article/423893

Extra Info

Microsoft Info:

I hope that most People have Read: Windows Secure Boot certificate expiration and CA updates

and Secure Boot playbook for certificates expiring in 2026

Redhat:
Secure Boot Certificate Changes in 2026: Guidance for RHEL Environments

Broadcom:
Secure Boot Certificate Expirations and Update Failures in VMware Virtual Machines

Manual Update of the Secure Boot Platform Key in Virtual Machines





Simplified Fix Secure Boot Script for Easy VM Updates

I recently created 3 version of a FixSecureBoot script — a lightweight alternative inspired by the excellent work of haz-ard-9, the author of FixSecureBootBulk.ps1. Their script is powerful and absolutely the right choice if you rely on BitLocker or need a fully automated, safety‑first workflow.

However, at roughly 3,000 lines of code, the original script is understandably complex. It includes many checks and safeguards, which are great for production environments but made it harder for me to fully understand what was happening under the hood. I wanted something simpler, easier to read, and tailored to my own workflow.

So I took the time to study the original script, copied only the parts I needed, and built a much more compact version that gives me exactly the result I want — which show the verification step that every thing is correct updated.

What My Script Does

Here’s the full sequence of actions my simplified script performs:

1.Shuts down the VM
2.Creates a snapshot
3.Enables UEFI Setup Mode
4.Clears VMRAM (for older VMs)
5.Upgrades virtual hardware if the VM is below version 21 (vSphere 8)
6.Starts the VM and waits for VMware Tools
7.Checks that the guest OS is fully online
8.Downloads the required certificates (only once)
9.Uploads the two certificates to the VM if not exist
10.Installs the new boot certificates
11.Shuts down the VM and clears Setup Mode
12.Boots the VM and sets AvailableUpdates to 0x5944 (certs ready for install)
13.Reboots until AvailableUpdates becomes 0x4100 (may require multiple reboots)
14.Reboots and runs Secure-Boot-Update again
15.Reboots and runs Secure-Boot-Update again, then checks for Event ID 1808 (if found, everything is good)

I’ve tested this workflow successfully on:

 Windows 11 (23H2, 24H2, 25H2)
 Windows Server 2016, 2019, 2022, and 2025

Downloads
** link the links for downloading the original files from Microsoft Github page.

WindowsOEMDevicesPK.der
microsoft corporation kek 2k ca 2023.der

Rename microsoft corporation kek 2k ca 2023.der to kek2023.der

.

Current Limitation: Packer Integration

CLEAN PACKER BUILD IS NOT SAFE!! AFTER THE BUILD YOU NEED TO RUN FIX SECURE BOOT!!

I don’t yet have a complete fix for integrating this into a full Packer build. For now, I simply pre‑stage the certificates:

Build file:

provisioner “file” {

source = “./setup/SecurebootCert/”

destination = “C:/Windows/Temp”

}

I have build three versions:

Fix_Secure_Boot_Manual.ps1
Fix_Secure_Boot_Single.ps1
Fix_Secure_Boot_Multi.ps1

If you want a script that’s easier to read, easier to modify, and still gets the job done (as long as you’re not using BitLocker), this simplified version might be exactly what you need.

Let me know if you want me to share the script itself or write a follow‑up post about how it works internally.
.

.

Template Check if al ready Updated

If VM check if al ready updated

.

Run Script full from Template
.

Translate »