ESX | Ward Vissers

VCF 9.0 Automate VMware Cloud Foundation Startup and Shutdown with PowerCLI

Powering your VMware Cloud Foundation “lab” environment on and off shouldn’t be a manual process.

A complete shutdown of a VMware Cloud Foundation (VCF) environment is uncommon, but for some energy savings and some time you does not use your lab often, you want a repeatable, reliable, and automated procedure, manually powering dozens of virtual machines in the correct order is both time-consuming and error-prone.

To solve this problem, I created two lightweight PowerCLI scripts:

• VCF 9.0 Small Startup Script

• VCF 9.0 Small Shutdown Script

vCenter and ESX hosts are manual (For vSAN cluster I did not find the correct code yet!)
Let me know if you have any questions or addons

Both scripts are available on GitHub and are designed to automate the startup and shutdown of a VMware Cloud Foundation management domain.

GitHub Repository

• VCF 9.0 Small Startup Script

• VCF 9.0 Small Shutdown Script

Note: Both Scripts do not work with VCF 9.1!!!!

Why These Scripts?

Although VMware Cloud Foundation automates the deployment and lifecycle of the platform, a full platform shutdown still requires the administrator to respect service dependencies.

For example:

• Domain Controllers must be available before authentication works.

• DNS must be online before many VMware services can resolve hostnames.

• vCenter must be operational before SDDC Manager can communicate with the infrastructure.

• NSX components depend on both networking and vCenter.

• VCF services should only start after the management platform is healthy.

Powering everything on simultaneously often results in services that need additional time—or even manual intervention—to recover.

These scripts automate the entire sequence descripted als following:

Start the Management Domain (VCF 9.0)’

Shut Down the Management Domain (VCF 9.0)

Typical Use Cases

These scripts are useful in many environments, including:

• Home labs

• Demonstration environments

• Disaster Recovery testing

• UPS maintenance

• Complete datacenter power outages

• Scheduled maintenance windows

• Hardware replacements

I personally use them in my VCF lab, where powering the environment up or down manually became repetitive and unnecessarily time-consuming. Automating the sequence not only saves time but also ensures a consistent and predictable startup every time.

Customizing the Scripts

Every VMware Cloud Foundation deployment is different.

The scripts are intentionally straightforward so you can easily adapt them by:

• Changing the startup order

• Adding custom virtual machines

• Removing components you don’t use

• Increasing wait times

• Adding health checks

• Integrating notifications

• Extending the logging

Because everything is written in PowerCLI, modifications are simple and require only basic scripting knowledge.

Future Improvements

Some ideas I’m considering for future releases include:

• Automatic dependency discovery

• Email notifications

• Automatic service validation

• Parallel startup where dependencies allow

Contributions and suggestions from the community are always welcome.

Lessons Learned

During development I discovered:

• VMware Tools are the best indicator that a guest OS is ready.

• Fixed sleep timers are unreliable because boot times vary.

• Starting all VMs simultaneously doesn’t necessarily reduce the total startup time.

• Graceful shutdowns significantly reduce recovery issues.

• Simplicity makes the scripts easier to customize.

Download

You can download the latest version from my public GitHub repository:

• VCF 9.0 Small Startup Script

• VCF 9.0 Small Shutdown Script

You can also find more VMware Cloud Foundation automation projects, PowerCLI scripts, and lab guides on my website:

• WardVissers.nl

Conclusion

A VMware Cloud Foundation environment consists of many interconnected services, and those services should be started and stopped in the correct order.

These lightweight PowerCLI scripts automate that process, making startup and shutdown predictable, repeatable, and significantly less error-prone. Whether you’re running a production management domain or a small VCF lab, automating these operational tasks saves time and reduces the risk of mistakes.

If you have ideas for improvements or additional features, feel free to open an issue or submit a pull request on GitHub. Happy automating!

Config a VCF (vSAN ESA) host the Easy Way

A while ago i created:

V1: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

V2: Config a VCF (vSAN ESA) host the Easy Way

With the release of VCF 9.1 is now time for again a updated version!

What does the script now:

✅ Disable ipv6

✅ Set DNS domain name

✅ Rename local datastore

✅Configure NTP

✅ MTU 9000

✅ Installs the vSAN ESA Hardware Mock VIB &

✅ Workaround to reduce impact of resync traffic in vSAN ESA clusters utilizing a 10G network

✅ Installs the Synology NFS Plug-in for VMware VAAI

✅ Memory Optimalisation Additional Transparent Page Sharing management capabilities and new default settings

✅ AMD Zen4/Zen5 IPMI Thermal Driver for ESX AMD Zen4/Zen5 IPMI Thermal Driver for ESX Fling

✅ Generate new certificate on the ESXi host (for the VCF verification check)

✅ Ask are you running Miniforum MS-A2(AMD) host & Optimalization see
VCF 9.1 – Comprehensive ESX Configuration Workarounds for Lab Deployments (Except the vSAN Compression Algorithm)

✅ Enable Memory Tiering for 9.1 and 9.0 and older (Filter OS Disk)

vSAN ESA Mock and the AMD Zen4/Zen5 IPMI Thermal Driver can be download on the Broadcom Fling page.

(https://support.broadcom.com/group/ecx/free-downloads) section and select Flings (https://support.broadcom.com/group/ecx/productdownloads?subfamily=Flings&freeDownloads=true)

You need to download the vibs separately!
For the installs put the vib’s in the same map as the script
.

You can download the script: HERE

Config a VCF (vSAN ESA) host the Easy Way

A while ago i created: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

Sow it’s time for a new updated version.

What does the script now:

✅ Disable ipv6

✅ Set DNS domain name

✅ Rename local datastore

✅ Configure NTP

✅ Installs the vSAN ESA Hardware Mock VIB

✅ Installs the Synology NFS Plug-in for VMware VAAI

✅ Installs the Realtek Network Driver for ESXi more information here
I use the 10Gtek 1.25/2.5/5/10GBase-T SFP+ RJ-45 CAT.6a Copper Transceiver on the MS-A2

✅ Configure vSAN DOMNetworkSchedulerThrottleComponent

✅ Generate new certificate on the ESXi host (for the VCF verification check)

✅ Ask are you running Miniforum MS-A2(AMD) host & Then Disable apichv

✅ Enable Memory Tiering

You need to download the vibs separately!
For the installs put the vib’s in the same map as the script
You can download the script: HERE

How to update standalone ESX(i) server

Sometimes I write stuff for myself and publish it as a archive piece, so as this one.

For 8.0U3

esxcli software sources profile list -d https://dl.broadcom.com/<Your_Broadcom_Download_Token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml | grep -i ESXi-8.0U3

Afbeelding met schermopname, tekstDoor AI gegenereerde inhoud is mogelijk onjuist.

For ESX(i) 9

esxcli software sources profile list -d https://dl.broadcom.com/<Your_Broadcom_Download_Token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml | grep -i ESXi-9

If you got a memory error run the following commands

If you do not get a list of versions run the following commands :

esxcli system settings advanced set -o /VisorFS/VisorFSPristineTardisk -i 0
cp /usr/lib/vmware/esxcli-software /usr/lib/vmware/esxcli-software.bak
sed -i ‘s/mem=300/mem=500/g’ /usr/lib/vmware/esxcli-software.bak
mv /usr/lib/vmware/esxcli-software.bak /usr/lib/vmware/esxcli-software -f
esxcli system settings advanced set -o /VisorFS/VisorFSPristineTardisk -i 1

Source: https://williamlam.com/2024/03/quick-tip-using-esxcli-to-upgrade-esxi-8-x-throws-memoryerror-or-got-no-data-from-process.html

Install latest Update (ESXi-8.0U3g-24859861-standard)

esxcli software profile update -p ESXi-8.0U3g-24859861-standard -d https://dl.broadcom.com/<Your_Broadcom_Download_Token>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml

Running VCF 9 in a Nested Lab using vSAN ESA on a single host is that not great?

For the deployment I used a server that has 16 cores en 32 cores total and a lot of ram!

Installation on Nested ESX VM’s

For the nested ESX VMs I am using the generic ESX image. In my lab I’m using 4 nested ESX nodes for vSAN ESA (Minimum is needed is 3).

Ideally add 24 vCPUs to the nested ESX, but VCF Automation is capable of being deployed with 16 vCPUs
Check Expose hardware assisted virtualization to the guest OS under CPU
Add a NVME controller and connect disks to this controller if planning to use vSAN ESA (Don’t forget to remove the old Controller)
Add at least two VMXNET3 nics connected to the same network
Choose VMware ESXi 8.0 or later as the Guest OS version.
Change the IDE CD-ROM to SATA ROM (Failed to locate kickstart on Nested ESXi VM CD-ROM in VCF 9.0)

Config the 4 Host for vSAN ESA

To config the 4 Hosts for vSAN ESA I used my own powercli script where I blogged about here: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script

You can download the script HERE!

Installing the Vib works still great. No problem with ESA Pre-check: ✅

If you have problems check this great blog from William Lam: vSAN ESA Disk & HCL Workaround for VCF 9.0

VCF installer.

At first you have to deploy the VMware Cloud Foundation

The name of the installer is a little off: VCF-SDDC-Manager-Appliance-9.0.0.0.24703748.ova

Download token

If you don’t have a download token from the Broadcom support then it’s a little complicated. I won’t go into this in depth now but here is a nice article if you have a Synology Nas: VCF 9.0 Offline Depot using Synology

(Note: For any vExpert reading this… You need to claim your own domain name because build your profile is not working with a @hotmail.com or @gmail accounts, Noted by Broadcom)

I looks something like this!

10GBE Nic Pre-Check Issue

Next thing to do was: Disable 10GbE NIC Pre-Check in the VCF 9.0 Installer or this should also work: How to change the vmxnet3 link speed of a VM (Not tested)

Deployment

I have deployed the beta in a earlier state. For deployment I used the same json file to start with.

VCF Automation will deployed automatically that was only the change that I changed it

It’s running!

Performance running Nested!!

Above the screenshot from the host with de Nested VCF lab deployment it’s quite cpu intensive (2 x CPU 8cores / 16 Threads)

Optimalization

After the deployment I did three things
– On the vSAN ESA Cluster (Disable Auto Disk Claim) (Keep the warning away)
– On the NSX VM reduce the cpu reservation from 6000 🡪 2000 (It helps but not enough)
– VCF automation VM 24 cores to 16 cores.

NSX en VCF automation are really CPU intensive.

I had deployed also two Edge servers but the server did not like that. Edges are also CPU intensive.

I am thinking about adding MS-01 or MS-A2 for splitting some load.

Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

William Lam created the vSAN ESA HCL hardware mock VIB for Nested ESXi.
It works great for vSAN ESA or for VCF vSAN ESA Nested nodes.

You can download the needed vib here

A whale ago I found a easy way to the deployment of a VMware SDDC based on vSAN ESA: VCF automated lab deployment with vSAN ESA

It works great. I a few hours you have a working VCF management domain on vSAN ESA.

Sow what about the script

For the Workload domain domain I created de nested ESXi vm’s manually .
And I did some testing for VCF 9 which I created the nested ESXi host also manually.

Configure by hand takes a lot of time. So I did some scripting and I used some code
from: VCF automated lab deployment with vSAN ESA

So I created a script that does the following:

Login on DHCP address en configure a Fixt IP address based on DNS name
Disable IPv6
Rename the local datastore to a uniek name with the name of the host
Configure NTP
Install the vSAN ESA VIB mock en restart vSAN mgmt.
Generate new certificate for the host with the correct domain name in it.
Enable KB372309 (10GB ethernet for vSAN ESA)

It works great for ESXi Nested nodes (ESXi 8.0u3b and ESXi9.X beta)

You can find the script HERE!!

Easy Script to Create DNS Records in VCF Lab

When you build your VCF Lab environment you want to create your DNS records automatically. I use for DNS a Windows Server.

The Script:

function ConvertTo-DecimalIP {
param ([string]$ip)
$parts = $ip.Split(‘.’) | ForEach-Object { [int]$_ }
return ($parts[0] -shl 24) + ($parts[1] -shl 16) + ($parts[2] -shl 8) + $parts[3]
}

function ConvertTo-DottedIP {
param ([int]$intIP)
$part1 = ($intIP -shr 24) -band 0xFF
$part2 = ($intIP -shr 16) -band 0xFF
$part3 = ($intIP -shr 8) -band 0xFF
$part4 = $intIP -band 0xFF
return “$part1.$part2.$part3.$part4”
}

$zone = “testlab.nl”
$startip = “192.168.200.10”

$dnsrecords = “vcf-m01-cb01″,”vcf-m01-sddcm01″,”vcf-m01-esx01″,”vcf-m01-esx02″,”vcf-m01-esx03″,”vcf-m01-esx04″,”vcf-w01-esx02″,”vcf-w01-esx03″,”vcf-w01-esx04″,”vcf-w01-esx04″,”vcf-m01-nsx01a”,”vcf-m01-nsx01b”,”vcf-m01-nsx01c”,”vcf-m01-nsx01″,”vcf-w01-nsx01a”,”vcf-w01-nsx01b”,”vcf-w01-nsx01c”,”vcf-w01-nsx01″,”vcf-m01-vc01″,”vcf-w01-vc01″
$count = $dnsrecords.count

# Convert start IP to decimal

$decimalIP = ConvertTo-DecimalIP $startIP
$i = 0

# Loop and print incremented IPs

foreach ($dnsrecord in $dnsrecords) {
$i -lt
$count;
$i++
$currentDecimalIP = $decimalIP + $i
$currentIP = ConvertTo-DottedIP $currentDecimalIP
Add-DnsServerResourceRecordA -Name $dnsrecord -ZoneName $zone -AllowUpdateAny -IPv4Address $currentIP -CreatePtr
Write-Output “DNS record $dnsrecord in $zone with $currentIP is created” -ForegroundColor Green

How to get Aria Operations (Skyline) Diagnostics working

On the 4^th Oktober VMware Skyline was end of life.

VMware Skyline was great:
• Proactive Issue Identification
• Automated Insights
• Health Scans and Remediation
• Integration with support

VMware by Broadcom are building critical Findings and Self-Help recommendations directly in product starting with VCF (from 5.2) and Aria Operations (from v8.18 July 2024)

Many of the other Skyline features are being planned for inclusion in future releases in Cloud Foundation and Aria Operations. We will see what the future will bring.
But for now how do you get this working.

First Step:

Update Aria Operations to 8.18.2 (Lastest)

Second Steps:

1. vCenter (Don’t for get enable vSAN), NSX, VCF, Aria vRA

2. Configure log collection in Aria Logs for the following components:

• Configure vCenter server integration in Aria for Logs

• Configure log forwarding on vCenter server, ESXi hosts(automatically in Aria FOR logs), and SDDC manager

3. Integrating VMware Aria Operations for Logs and VMware Aria Operations

4. Connect Skyline Health Diagnostics (SHD)

5. In Aria LoginSight check the vRops integration checkboxes

Bij default Enable launch in context can be disabled when configured at first.

After upgrading and checking the settings its finally working 😊 (It can take some time).

How to Remove Inaccessible vSAN Objects in vSphere: Step-by-Step Guide

This post is about how to remove such an inaccessible object within vSAN.

Afbeelding met tekst, schermopname, nummer, LettertypeAutomatisch gegenereerde beschrijving

Open an SSH session to the vCenter and enter the command rvc localhost in the command line.

Navigate to the destinated vSAN cluster where you want to remove the inaccessible objects using cd and utilize ls to list in each step like this one:

Verify the state of vSAN objects using the command vsan.check_state -r . This check involves three steps:

Checking for inaccessible vSAN objects
Verifying invalid or inaccessible VMs, and
Checking for VMs for which VC/hostd/vmx are out of sync

During this check, as you can see in the next screenshot, there are four inaccessible objects with the same UUID as those listed in Virtual Objects within the vSphere Client.

Afbeelding met tekst, schermopname, LettertypeAutomatisch gegenereerde beschrijving

To remove them, open an SSH session to any ESXi in the cluster and use the following command /usr/lib/vmware/osfs/bin/objtool delete -u <UUID> -f replacing UUID with the one you want to remove. Afbeelding met tekst, schermopname, LettertypeAutomatisch gegenereerde beschrijving

After you remove all inaccessible objects and run the (vsan.checkstate -r .) once again, you should no longer see any inaccessible objects. Afbeelding met tekst, schermopname, software, LettertypeAutomatisch gegenereerde beschrijving

Tag: ESX

VCF 9.0 Automate VMware Cloud Foundation Startup and Shutdown with PowerCLI

Like this:

Config a VCF (vSAN ESA) host the Easy Way

Like this:

Config a VCF (vSAN ESA) host the Easy Way

Like this:

How to update standalone ESX(i) server

Like this:

Running VCF 9 in a Nested Lab using vSAN ESA on a single host is that not great?

Config the 4 Host for vSAN ESA

VCF installer.

Download token

10GBE Nic Pre-Check Issue

Deployment

It’s running!

Performance running Nested!!

Optimalization

Like this:

Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

Like this:

How to Remove Inaccessible vSAN Objects in vSphere: Step-by-Step Guide

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Config the 4 Host for vSAN ESA

VCF installer.

Download token

10GBE Nic Pre-Check Issue

Deployment

It’s running!

Performance running Nested!!

Optimalization

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: