VCF 9.1 How to Safely Shut Down the Services Runtime Cluster Using PowerShell

Broadcom provides an official procedure for gracefully shutting down a VMware Cloud Foundation (VCF) 9.1 Services Runtime cluster. The provided solution uses a Linux shell script (vcf_services_runtime_shutdown.sh), which works well on Linux systems but is less convenient for Windows administrators.

Rather than using Windows Subsystem for Linux (WSL) or a separate Linux machine, I converted the official shell script into a native PowerShell version. This allows the entire shutdown procedure to be executed directly from a Windows workstation while following the same workflow as the original Broadcom script.

Downloads

📦 VCF 9.1 Shutdown VCF Services VMs.ps1
📦 vcf_services_runtime_shutdown.ps1

📄 Original Broadcom Knowledge Base vcf_services_runtime_shutdown.sh

Why Use the Official Shutdown Procedure?

 Kubernetes workloads are gracefully terminated.
 Running services are stopped in the correct order.
 Databases remain in a consistent state.
 Cluster metadata is preserved.
 The shutdown process completes without leaving services in an inconsistent state.

Prerequisites

 VMware Cloud Foundation 9.1
 PowerShell 7.x (recommended)
 Latest VCF PowerCLI
 vCenter Administrator credentials
 Network connectivity to the Control Plane node

Before You Begin

 Power off any VCF Automation virtual machines before shutting down the Services Runtime cluster to avoid interference with automation tasks.

Determine the Control Plane Node IP Address

 Open VCF Operations.
 Navigate to Build → Lifecycle → Components.
 Select VCF Services Runtime.
 Scroll to the Nodes section.
 Locate a Control Plane node and note its IP address.

PowerShell Wrapper Script

 Detects the current script location.
 Changes to the correct working directory.
 Calls the shutdown script with the required parameters.
 Simplifies execution.

Download Wrapper Script

Download: Shutdown VCF Services.ps1

# VCF Services shutdown by Ward Vissers

# Variables

switch ((get-host).Name) {

‘Windows PowerShell ISE Host’ { $vcf_folder = $psISE.CurrentFile.FullPath -replace ($psISE.CurrentFile.DisplayName,””) }

ConsoleHost{ $vcf_folder = $myInvocation.MyCommand.Path -replace ($myInvocation.MyCommand.Name,”“) }

‘Visual Studio Code Host‘{ $vcf_folder = $psEditor.GetEditorContext().CurrentFile.Path | Split-Path }

}

# Go to the VCF folder

Set-Location $vcf_folder

$Credential = Get-Credential –UserNameadministrator@vsphere.local” -Message “vCenter Login Creds”

# Shutdown Test Run

.\vcf_services_runtime_shutdown.ps1 –DryRunNodeIp <your_node_here> –Credential $Credential

# Shutdown Run

.\vcf_services_runtime_shutdown.ps1 –DryRunNodeIp <your_node_here>  –Credential $Credential

Authenticate Securely

 Use Get-Credential instead of storing passwords in the script.

Perform a Dry Run

 Run the script with -DryRun to validate connectivity, authentication, environment configuration and cluster discovery without shutting down services.

.

Execute the Shutdown

 Connect to vCenter.
 Discover the Services Runtime cluster.
 Validate runtime nodes.
 Gracefully terminate Kubernetes workloads.
 Shut down runtime nodes.
 Verify successful completion.

Expected Result

 All VCF Services Runtime virtual machines are powered off in the correct order and the console reports a successful shutdown.

.

.

Conclusion

The PowerShell implementation follows the same workflow as Broadcom’s Linux script while allowing Windows administrators to perform the shutdown natively from Windows without WSL.

What’s Next?

 Automated startup & shutdown script for VCF 9.1 environment

Simplified Fix Secure Boot Script for Easy VM Updates

I recently created 3 version of a FixSecureBoot script — a lightweight alternative inspired by the excellent work of haz-ard-9, the author of FixSecureBootBulk.ps1. Their script is powerful and absolutely the right choice if you rely on BitLocker or need a fully automated, safety‑first workflow.

However, at roughly 3,000 lines of code, the original script is understandably complex. It includes many checks and safeguards, which are great for production environments but made it harder for me to fully understand what was happening under the hood. I wanted something simpler, easier to read, and tailored to my own workflow.

So I took the time to study the original script, copied only the parts I needed, and built a much more compact version that gives me exactly the result I want — which show the verification step that every thing is correct updated.

What My Script Does

Here’s the full sequence of actions my simplified script performs:

1.Shuts down the VM
2.Creates a snapshot
3.Enables UEFI Setup Mode
4.Clears VMRAM (for older VMs)
5.Upgrades virtual hardware if the VM is below version 21 (vSphere 8)
6.Starts the VM and waits for VMware Tools
7.Checks that the guest OS is fully online
8.Downloads the required certificates (only once)
9.Uploads the two certificates to the VM if not exist
10.Installs the new boot certificates
11.Shuts down the VM and clears Setup Mode
12.Boots the VM and sets AvailableUpdates to 0x5944 (certs ready for install)
13.Reboots until AvailableUpdates becomes 0x4100 (may require multiple reboots)
14.Reboots and runs Secure-Boot-Update again
15.Reboots and runs Secure-Boot-Update again, then checks for Event ID 1808 (if found, everything is good)

I’ve tested this workflow successfully on:

 Windows 11 (23H2, 24H2, 25H2)
 Windows Server 2016, 2019, 2022, and 2025

Downloads
** link the links for downloading the original files from Microsoft Github page.

WindowsOEMDevicesPK.der
microsoft corporation kek 2k ca 2023.der

Rename microsoft corporation kek 2k ca 2023.der to kek2023.der

.

Current Limitation: Packer Integration

CLEAN PACKER BUILD IS NOT SAFE!! AFTER THE BUILD YOU NEED TO RUN FIX SECURE BOOT!!

I don’t yet have a complete fix for integrating this into a full Packer build. For now, I simply pre‑stage the certificates:

Build file:

provisioner “file” {

source = “./setup/SecurebootCert/”

destination = “C:/Windows/Temp”

}

I have build three versions:

Fix_Secure_Boot_Manual.ps1
Fix_Secure_Boot_Single.ps1
Fix_Secure_Boot_Multi.ps1

If you want a script that’s easier to read, easier to modify, and still gets the job done (as long as you’re not using BitLocker), this simplified version might be exactly what you need.

Let me know if you want me to share the script itself or write a follow‑up post about how it works internally.
.

.

Template Check if al ready Updated

If VM check if al ready updated

.

Run Script full from Template
.

Setting Up KubeDoom on Kubernetes: A Beginner’s Guide

I followed William Lam’s article about MS-A2 VCF 9.0 Lab: Configuring vSphere Kubernetes Service (VKS)

I don’t have much experience with Kubernetes but wanted to try some new things.

The only container that I have running is Home Assistant on Docker.

Got to try to get Kubedoom working. So I did with the following steps.
Maybe in the near future I’ll try to add more games: Retro DOS Games on Kubernetes

Finally having a Kubernetes cluster version 1.32, which was required for running KubeDoom.

Afbeelding met tekst, schermopnameDoor AI gegenereerde inhoud is mogelijk onjuist.

Download kubectl

Afbeelding met tekst, schermopname, brief, ontwerpDoor AI gegenereerde inhoud is mogelijk onjuist.

mkdir d:\kubectl

Extract the downloaded ZIP file and place both executables (kubectl.exe and kubectl-vsphere.exe) in a folder such as: d:\kubectl

.\kubectl version –client

mkdir .kube

cd .kube

New-Item config -type file

.\kubectl vsphere login –server=https://31.31.0.7 –insecure-skip-tls-verify

Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

kubectl –kubeconfig=kubernetes-cluster-jzvx-kubeconfig.yaml get pods

kubectl –kubeconfig=kubernetes-cluster-jzvx-kubeconfig.yaml apply -f kubedoom.yaml

Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

kubectl –kubeconfig=kubernetes-cluster-jzvx-kubeconfig.yaml -n kubedoom get svc

Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

The password to Kubedoom is idbehold

Download VNC Viewer: https://www.realvnc.com/en/connect/download/viewer

Afbeelding met tekst, schermopname, pc-game, MultimediasoftwareDoor AI gegenereerde inhoud is mogelijk onjuist.

This is funny and cool!

Omnissa and the 100Gb Core vSAN license

When you design a new VDI vSAN you have take licensing into account

Some links about the licensing:

Broadcom describes the vSAN licensing in VVF for VDI, https://ftpdocs.broadcom.com/cadocs/0/contentimages/VVF_VDI_SPD_November2024.pdf

You can find more information also here: Omnissa Horizon combined offering with VMware vSphere Foundation for VDI – FAQ (6000381)

So for this test purpose I have 3 x 3.2TB Storage Drives for test.
Sow I have 9.7 TB storage available in a Single node vSAN Cluster

The 100GB “vSAN” Evaluation License comes from the host license! (Licensed for 50years)

Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.


With a 2 x 32 core server I have a total of 64 cores.

With 64 Cores and 100Gb/PerCore license I should not extend more then the raw of 6.4TB.

Wit current raw capacity of 9.6TB I get a error

Alarm is gone after removing 1 Disk from the 9.6 TB vSAN Storage (Left 1 Disk as spare). Storage is now 6.4 TB.

So the conclusion is DON’T go above the RAW Core capacity!!!

For design is the following handy:

Horizon Sizing: The Spreadsheet I Made So I’d Stop Yelling at My Monitor

The Hidden Delta Disk Nobody Cared About

Running VCF 9 in a Nested Lab using vSAN ESA on a single host is that not great?

For the deployment I used a server that has 16 cores en 32 cores total and a lot of ram!

Installation on Nested ESX VM’s

For the nested ESX VMs I am using the generic ESX image. In my lab I’m using 4 nested ESX nodes for vSAN ESA (Minimum is needed is 3).

  • Ideally add 24 vCPUs to the nested ESX, but VCF Automation is capable of being deployed with 16 vCPUs
  • Check Expose hardware assisted virtualization to the guest OS under CPU
  • Add a NVME controller and connect disks to this controller if planning to use vSAN ESA (Don’t forget to remove the old Controller)
  • Add at least two VMXNET3 nics connected to the same network
  • Choose VMware ESXi 8.0 or later as the Guest OS version.
  • Change the IDE CD-ROM to SATA ROM (Failed to locate kickstart on Nested ESXi VM CD-ROM in VCF 9.0)

Afbeelding met tekst, schermopname, nummer, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Config the 4 Host for vSAN ESA

To config the 4 Hosts for vSAN ESA I used my own powercli script where I blogged about here: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script

You can download the script HERE!

Installing the Vib works still great. No problem with ESA  Pre-check: ✅

If you have problems check this great blog from William Lam: vSAN ESA Disk & HCL Workaround for VCF 9.0

VCF installer.

At first you have to deploy the VMware Cloud Foundation

The name of the installer is a little off: VCF-SDDC-Manager-Appliance-9.0.0.0.24703748.ova

Download token

If you don’t have a download token from the Broadcom support then it’s a little complicated. I won’t go into this in depth now but here is a nice article if you have a Synology Nas: VCF 9.0 Offline Depot using Synology

(Note: For any vExpert reading this… You need to claim your own domain name because build your profile is not working with a @hotmail.com or @gmail accounts, Noted by Broadcom)

Afbeelding met tekst, schermopname, Parallel, ontwerpDoor AI gegenereerde inhoud is mogelijk onjuist.

I looks something like this!
Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

10GBE Nic Pre-Check Issue

Next thing to do was: Disable 10GbE NIC Pre-Check in the VCF 9.0 Installer or this should also work: How to change the vmxnet3 link speed of a VM (Not tested)

Afbeelding met tekst, elektronica, schermopname, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

Deployment

I have deployed the beta in a earlier state. For deployment I used the same json file to start with.

VCF Automation will deployed automatically that was only the change that I changed it

Afbeelding met tekst, schermopname, nummer, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, schermopname, Website, ontwerpDoor AI gegenereerde inhoud is mogelijk onjuist.

It’s running!
Afbeelding met tekst, schermopname, scherm, nummerDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, lijn, nummer, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Performance running Nested!!

Afbeelding met tekst, lijn, Perceel, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Above the screenshot from the host with de Nested VCF lab deployment it’s quite cpu intensive (2 x CPU 8cores / 16 Threads)

Optimalization

After the deployment I did three things
– On the vSAN ESA Cluster (Disable Auto Disk Claim) (Keep the warning away)
– On the NSX VM reduce the cpu reservation from 6000 🡪 2000 (It helps but not enough)
– VCF automation VM 24 cores to 16 cores.

NSX en VCF automation are really CPU intensive.

I had deployed also two Edge servers but the server did not like that. Edges are also CPU intensive.

I am thinking about adding MS-01 or MS-A2 for splitting some load.

Deploying Windows VMs with Terraform and Packer

A whale ago, I started using Packer for creating Windows Templates for using in my home lab.
I blogged about it here: Windows Server 2025 “Preview” deployment with Packer

So now I want to deploy a template with Terraform, which was created earlier by Packer.

First, we need to ensure we have the Terraform downloaded and the vSphere provider Initialized.

  1. Install Terraform: You can download it from the official Terraform website and follow the installation instructions for your operating system.

I made my life a little easier to create a PowerShell script that downloads the latest version: Download Terraform.ps1

  1. Initialize Your Terraform Configuration: Create a directory for your Terraform configuration files. Inside this directory, create a `main.tf` file (or any `.tf` file) where you will define your provider and other configurations.
    Define the vSphere Provider in Your main.tf: Add the following block to your Terraform configuration file to specify the use of the vSphere provider:

terraform {

  required_providers {

    vsphere = {

      source = “HashiCorp/vsphere”

      version = “> 2.11”

    }

  }

}

provider “vsphere” {

  user           = vcenter_username

  password       = vcenter_password

  vsphere_server = vcenter_server

  # If you have a self-signed cert

  allow_unverified_ssl = true

}

Replace ”vcenter_username”, ” vcenter_password” , ”vcenter_server” with your actual vSphere credentials and server address.

  1. Basic Terraform Commands

Terraform.ps1

# Go to the Terraform download folder

$terraformfolder = ‘d:\automation\terraform\’

Set-Location $terraformfolder

# Download Terraform plugins

.\terraform.exe init

# Test Run

.\terraform.exe plan

# Run Terraform

.\terraform.exe apply

# Clean Up what you created

.\terraform.exe Destroy

Initialize the Terraform Working Directory: Run the following command in your terminal from the directory where your Terraform configuration file is located:

terraform init

Afbeelding met tekst, schermopname, Lettertype, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

This command will download the vSphere provider and initialize your Terraform working directory. It sets up and downloads the necessary provider plugins for Terraform to interact with vSphere.

Verify the Installation: After running `terraform init`, you should see output indicating that the vSphere provider has been successfully installed. You can now proceed to create Terraform configurations to manage your vSphere resources.

Deploy a Template VM within vSphere

In this example, we are deploying a VM running Windows Server 2022.

Terraform Module Structure

  1. variables.tf: Define the variables for the module.
  2. main.tf: Contains the main configuration for the VM.
  3. outputs.tf: Define the outputs for the module.

Variables.tf

variable “vsphere_user” {

  default     = “<your_vcenter_username_here>”

   description = “vSphere username to use to connect to the environment – Default: administrator@vsphere.local”

}

variable “vsphere_password” {

  default     = “<your_vcenter_password_here>”

  description = “vSphere vCenter password”

}

variable “vsphere_server” {

  default = “<your_vcenter_here>”

  description = “vSphere vCenter server”

}

variable “datacenter” {

  default = “<your_datacenter_here>”

  description = “vSphere datacenter”

}

variable “cluster” {

  default = “<your_cluster_here>”

  description = “vSphere cluster”

}

variable “network” {

  default = “<vcenter_network>”

  description = “vSphere network”

}

variable “datastore” {

  default = “<your_destination_datastore_>”

  description = “vSphere datastore”

}

variable “template” {

  default = “<your_template_name_here>”

  description = “Template VM”

}

variable “customization_specifications” {

  default = “<vcenter_customization_specifications>”

  description = “Customization Spec”

}

When you run “.\terraform plan” you do a test run to check your coding

Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, schermopname, Multimediasoftware, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

When you run “.\terraform apply” build your virtual machine

Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, schermopname, Lettertype, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

Based on: https://vminfrastructure.com/2025/03/11/deploying-a-vm-using-terraform/

It works but it need some tweaking and fixing.

Adding a TPM2.0 security device does not work right now
Server 2025 customization does not work

How to Obtain 3 Years of VMware Licenses with Certification

By passing either of the new VCP-VCF level certification exam(s), anyone maintaining an active VMUG Advantage membership can receive 3 years worth of extensive VMware Cloud Foundation licensing for home lab use!

Afbeelding met tekst, computer, schermopname, WebsiteDoor AI gegenereerde inhoud is mogelijk onjuist.

The VMUG Advantage program has offered affordable home lab VMware licensing packages for years, but did cover most over the entire product portfolio.

Last year Broadcom made a change into this.

Option 1: Get vSphere Standard Edition 32 cores for 1 year: Pass one of the following VCP certification exams

  • VCP-VVF (admin/architect)
  • VCP-VCF (admin/architect)

Option 2:  Get VMware Cloud Foundation (VCF) 128 cores for 3 years: Purchase & Maintain VMUG Advantage, pass the following VCP certification exam.

  • VCP-VCF (admin/architect)

A VMUG Advantage membership was complimentary for vExperts in 2025

The membership is $210 otherwise, and does include a voucher for a 50% discounted VCP-VCF exam

With the requirements in place, head to the Broadcom “VCP Certification Non-Production Licenses” portal and request licenses.

Afbeelding met tekst, software, Computerpictogram, WebpaginaDoor AI gegenereerde inhoud is mogelijk onjuist.

Essential Insights on Windows Server 2025

Essential Insights on Windows Server 2025

  1. Free Windows Server 2025 Security Advice Book read here and download here
  2. Windows Server 2025 is Certified on VMware vSphere
  3. Windows Server 2025 known issues and notifications
  4. New & Updated Security Tools
  5. Windows Server 2022 to 2025: Active Directory Upgrade Guide

How to Remove Inaccessible vSAN Objects in vSphere: Step-by-Step Guide

This post is about how to remove such an inaccessible object within vSAN.

Afbeelding met tekst, schermopname, nummer, LettertypeAutomatisch gegenereerde beschrijving

Open an SSH session to the vCenter and enter the command rvc localhost in the command line.

Navigate to the destinated vSAN cluster where you want to remove the inaccessible objects using cd and utilize ls to list in each step like this one:

Verify the state of vSAN objects using the command vsan.check_state -r . This check involves three steps:

  • Checking for inaccessible vSAN objects
  • Verifying invalid or inaccessible VMs, and
  • Checking for VMs for which VC/hostd/vmx are out of sync

During this check, as you can see in the next screenshot, there are four inaccessible objects with the same UUID as those listed in Virtual Objects within the vSphere Client.

Afbeelding met tekst, schermopname, LettertypeAutomatisch gegenereerde beschrijving

To remove them, open an SSH session to any ESXi in the cluster and use the following command /usr/lib/vmware/osfs/bin/objtool delete -u <UUID> -f replacing UUID with the one you want to remove. Afbeelding met tekst, schermopname, LettertypeAutomatisch gegenereerde beschrijving

After you remove all inaccessible objects and run the (vsan.checkstate -r .) once again, you should no longer see any inaccessible objects. Afbeelding met tekst, schermopname, software, LettertypeAutomatisch gegenereerde beschrijving

My HomeLab anno 2024

My home lab is manly used for testing new stuff released by VMware (70%) en Microsoft (20%) other stuff (10%)

For the Base I use my Home PC

Intel i5 12600k
128 GB Memory
2 x 2TB Samsung 980 and 990 Pro SSD.
Windows 11 Pro
VMware Workstation Pro

On my Home PC running
Server 2022 (Eval for DC)
ESXi801 (16 GB) (NSX Demo Cluster)
ESXi802 (16 GB) (NSX Demo Cluster)
ESXi803 (64 GB) (General Cluster) )
ESXi804 (64 GB) (General Cluster)
ESXi805 (24 GB) (Single Node vSAN Cluster)
ESXi806 (16 GB) (4 Node vSAN Cluster)
ESXi807 (16 GB) (4 Node vSAN Cluster)
ESXi808 (16 GB) (4 Node vSAN Cluster)
ESXi809 (16 GB) (4 Node vSAN Cluster)

ESXi701 (24GB) (General Cluster)
ESXi702 (24GB) (General Cluster)

In general cluster there a running the most VM’s. Also here I am testing Packer and Terraform.

Afbeelding met tekst, schermopname, software, LettertypeAutomatisch gegenereerde beschrijving

For a while I used a 2TB Samsung SSD a Storage for ESXi Server through Truenas
But I wanted a larger storage for all my VM’s.

After reading on William Liam blog Synology DS723+ in Homelab and Synology NFS VAAI Plug-in support for vSphere 8.0

So I did a nice upgrade. Afbeelding met tekst, schermopname, software, nummerAutomatisch gegenereerde beschrijving

I used not the original Synology Parts. Following parts works fine.
Kingston 16 GB DDR4-3200 notebook memory
WD Red SN700, 500 GB SSD
WD Red Pro, 8 TB

* For Read-Write caching you need 2 SSD devices.

For mouting the NFS shared I created a little powercli script.

https://github.com/WardVissers/VMware-Powercli-Public/blob/main/Add%20NFS%20DataStore%20Github.ps1
Translate »