Running VCF 9 in a Nested Lab using vSAN ESA on a single host is that not great?

For the deployment I used a server that has 16 cores en 32 cores total and a lot of ram!

Installation on Nested ESX VM’s

For the nested ESX VMs I am using the generic ESX image. In my lab I’m using 4 nested ESX nodes for vSAN ESA (Minimum is needed is 3).

  • Ideally add 24 vCPUs to the nested ESX, but VCF Automation is capable of being deployed with 16 vCPUs
  • Check Expose hardware assisted virtualization to the guest OS under CPU
  • Add a NVME controller and connect disks to this controller if planning to use vSAN ESA (Don’t forget to remove the old Controller)
  • Add at least two VMXNET3 nics connected to the same network
  • Choose VMware ESXi 8.0 or later as the Guest OS version.
  • Change the IDE CD-ROM to SATA ROM (Failed to locate kickstart on Nested ESXi VM CD-ROM in VCF 9.0)

Afbeelding met tekst, schermopname, nummer, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Config the 4 Host for vSAN ESA

To config the 4 Hosts for vSAN ESA I used my own powercli script where I blogged about here: Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script

You can download the script HERE!

Installing the Vib works still great. No problem with ESA  Pre-check: ✅

If you have problems check this great blog from William Lam: vSAN ESA Disk & HCL Workaround for VCF 9.0

VCF installer.

At first you have to deploy the VMware Cloud Foundation

The name of the installer is a little off: VCF-SDDC-Manager-Appliance-9.0.0.0.24703748.ova

Download token

If you don’t have a download token from the Broadcom support then it’s a little complicated. I won’t go into this in depth now but here is a nice article if you have a Synology Nas: VCF 9.0 Offline Depot using Synology

(Note: For any vExpert reading this… You need to claim your own domain name because build your profile is not working with a @hotmail.com or @gmail accounts, Noted by Broadcom)

Afbeelding met tekst, schermopname, Parallel, ontwerpDoor AI gegenereerde inhoud is mogelijk onjuist.

I looks something like this!
Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

10GBE Nic Pre-Check Issue

Next thing to do was: Disable 10GbE NIC Pre-Check in the VCF 9.0 Installer or this should also work: How to change the vmxnet3 link speed of a VM (Not tested)

Afbeelding met tekst, elektronica, schermopname, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

Deployment

I have deployed the beta in a earlier state. For deployment I used the same json file to start with.

VCF Automation will deployed automatically that was only the change that I changed it

Afbeelding met tekst, schermopname, nummer, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, schermopname, Website, ontwerpDoor AI gegenereerde inhoud is mogelijk onjuist.

It’s running!
Afbeelding met tekst, schermopname, scherm, nummerDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, lijn, nummer, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Performance running Nested!!

Afbeelding met tekst, lijn, Perceel, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Above the screenshot from the host with de Nested VCF lab deployment it’s quite cpu intensive (2 x CPU 8cores / 16 Threads)

Optimalization

After the deployment I did three things
– On the vSAN ESA Cluster (Disable Auto Disk Claim) (Keep the warning away)
– On the NSX VM reduce the cpu reservation from 6000 🡪 2000 (It helps but not enough)
– VCF automation VM 24 cores to 16 cores.

NSX en VCF automation are really CPU intensive.

I had deployed also two Edge servers but the server did not like that. Edges are also CPU intensive.

I am thinking about adding MS-01 or MS-A2 for splitting some load.

Config vSAN ESA host or VCF ESA vSAN Host the easy way with Config-VSAN-ESA-VCF-Lab-Host Script.

William Lam created the vSAN ESA HCL hardware mock VIB for Nested ESXi.
It works great for vSAN ESA or for VCF vSAN ESA Nested nodes.

You can download the needed vib here

A whale ago I found a easy way to the deployment of a VMware SDDC based on vSAN ESA: VCF automated lab deployment with vSAN ESA

It works great. I a few hours you have a working VCF management domain on vSAN ESA.

Sow what about the script

For the Workload domain domain I created de nested ESXi vm’s manually .
And I did some testing for VCF 9 which I created the nested ESXi host also manually.

Configure by hand takes a lot of time. So I did some scripting and I used some code
from: VCF automated lab deployment with vSAN ESA

So I created a script that does the following:

  • Login on DHCP address en configure a Fixt IP address based on DNS name
  • Disable IPv6
  • Rename the local datastore to a uniek name with the name of the host
  • Configure NTP
  • Install the vSAN ESA VIB mock en restart vSAN mgmt.
  • Generate new certificate for the host with the correct domain name in it.
  • Enable KB372309 (10GB ethernet for vSAN ESA)

It works great for ESXi Nested nodes (ESXi 8.0u3b and ESXi9.X beta)

You can find the script HERE!!

Afbeelding met tekst, schermopname, nummer, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, schermopname, nummer, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Deploying Windows VMs with Terraform and Packer

A whale ago, I started using Packer for creating Windows Templates for using in my home lab.
I blogged about it here: Windows Server 2025 “Preview” deployment with Packer

So now I want to deploy a template with Terraform, which was created earlier by Packer.

First, we need to ensure we have the Terraform downloaded and the vSphere provider Initialized.

  1. Install Terraform: You can download it from the official Terraform website and follow the installation instructions for your operating system.

I made my life a little easier to create a PowerShell script that downloads the latest version: Download Terraform.ps1

  1. Initialize Your Terraform Configuration: Create a directory for your Terraform configuration files. Inside this directory, create a `main.tf` file (or any `.tf` file) where you will define your provider and other configurations.
    Define the vSphere Provider in Your main.tf: Add the following block to your Terraform configuration file to specify the use of the vSphere provider:

terraform {

  required_providers {

    vsphere = {

      source = “HashiCorp/vsphere”

      version = “> 2.11”

    }

  }

}

provider “vsphere” {

  user           = vcenter_username

  password       = vcenter_password

  vsphere_server = vcenter_server

  # If you have a self-signed cert

  allow_unverified_ssl = true

}

Replace ”vcenter_username”, ” vcenter_password” , ”vcenter_server” with your actual vSphere credentials and server address.

  1. Basic Terraform Commands

Terraform.ps1

# Go to the Terraform download folder

$terraformfolder = ‘d:\automation\terraform\’

Set-Location $terraformfolder

# Download Terraform plugins

.\terraform.exe init

# Test Run

.\terraform.exe plan

# Run Terraform

.\terraform.exe apply

# Clean Up what you created

.\terraform.exe Destroy

Initialize the Terraform Working Directory: Run the following command in your terminal from the directory where your Terraform configuration file is located:

terraform init

Afbeelding met tekst, schermopname, Lettertype, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

This command will download the vSphere provider and initialize your Terraform working directory. It sets up and downloads the necessary provider plugins for Terraform to interact with vSphere.

Verify the Installation: After running `terraform init`, you should see output indicating that the vSphere provider has been successfully installed. You can now proceed to create Terraform configurations to manage your vSphere resources.

Deploy a Template VM within vSphere

In this example, we are deploying a VM running Windows Server 2022.

Terraform Module Structure

  1. variables.tf: Define the variables for the module.
  2. main.tf: Contains the main configuration for the VM.
  3. outputs.tf: Define the outputs for the module.

Variables.tf

variable “vsphere_user” {

  default     = “<your_vcenter_username_here>”

   description = “vSphere username to use to connect to the environment – Default: administrator@vsphere.local”

}

variable “vsphere_password” {

  default     = “<your_vcenter_password_here>”

  description = “vSphere vCenter password”

}

variable “vsphere_server” {

  default = “<your_vcenter_here>”

  description = “vSphere vCenter server”

}

variable “datacenter” {

  default = “<your_datacenter_here>”

  description = “vSphere datacenter”

}

variable “cluster” {

  default = “<your_cluster_here>”

  description = “vSphere cluster”

}

variable “network” {

  default = “<vcenter_network>”

  description = “vSphere network”

}

variable “datastore” {

  default = “<your_destination_datastore_>”

  description = “vSphere datastore”

}

variable “template” {

  default = “<your_template_name_here>”

  description = “Template VM”

}

variable “customization_specifications” {

  default = “<vcenter_customization_specifications>”

  description = “Customization Spec”

}

When you run “.\terraform plan” you do a test run to check your coding

Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, schermopname, Multimediasoftware, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

When you run “.\terraform apply” build your virtual machine

Afbeelding met tekst, schermopname, LettertypeDoor AI gegenereerde inhoud is mogelijk onjuist.

Afbeelding met tekst, schermopname, Lettertype, softwareDoor AI gegenereerde inhoud is mogelijk onjuist.

Based on: https://vminfrastructure.com/2025/03/11/deploying-a-vm-using-terraform/

It works but it need some tweaking and fixing.

Adding a TPM2.0 security device does not work right now
Server 2025 customization does not work

How to Obtain 3 Years of VMware Licenses with Certification

By passing either of the new VCP-VCF level certification exam(s), anyone maintaining an active VMUG Advantage membership can receive 3 years worth of extensive VMware Cloud Foundation licensing for home lab use!

Afbeelding met tekst, computer, schermopname, WebsiteDoor AI gegenereerde inhoud is mogelijk onjuist.

The VMUG Advantage program has offered affordable home lab VMware licensing packages for years, but did cover most over the entire product portfolio.

Last year Broadcom made a change into this.

Option 1: Get vSphere Standard Edition 32 cores for 1 year: Pass one of the following VCP certification exams

  • VCP-VVF (admin/architect)
  • VCP-VCF (admin/architect)

Option 2:  Get VMware Cloud Foundation (VCF) 128 cores for 3 years: Purchase & Maintain VMUG Advantage, pass the following VCP certification exam.

  • VCP-VCF (admin/architect)

A VMUG Advantage membership was complimentary for vExperts in 2025

The membership is $210 otherwise, and does include a voucher for a 50% discounted VCP-VCF exam

With the requirements in place, head to the Broadcom “VCP Certification Non-Production Licenses” portal and request licenses.

Afbeelding met tekst, software, Computerpictogram, WebpaginaDoor AI gegenereerde inhoud is mogelijk onjuist.

How to get Aria Operations (Skyline) Diagnostics working

On the 4th Oktober VMware Skyline was end of life.

Afbeelding met tekst, schermopname, elektronica, stroomkringAutomatisch gegenereerde beschrijving

VMware Skyline was great:
• Proactive Issue Identification
• Automated Insights
• Health Scans and Remediation
• Integration with support

VMware by Broadcom are building critical Findings and Self-Help recommendations directly in product starting with VCF (from 5.2) and Aria Operations (from v8.18 July 2024)

Many of the other Skyline features are being planned for inclusion in future  releases in Cloud Foundation and Aria Operations. We will see what the future will bring.
But for now how do you get this working.

First Step:

Update Aria Operations to 8.18.2 (Lastest)

Second Steps:

1. vCenter (Don’t for get enable vSAN), NSX, VCF, Aria vRA

2. Configure log collection in Aria Logs for the following components:

• Configure vCenter server integration in Aria for Logs

• Configure log forwarding on vCenter server, ESXi hosts(automatically in Aria FOR logs), and SDDC manager

3. Integrating VMware Aria Operations for Logs and VMware Aria Operations

4. Connect Skyline Health Diagnostics (SHD)

5. In Aria LoginSight check the vRops integration checkboxes

Bij default Enable launch in context can be disabled when configured at first.

After upgrading and checking the settings its finally working 😊 (It can take some time).
Afbeelding met schermopname, Multimediasoftware, Grafische software, softwareAutomatisch gegenereerde beschrijving

Using Get-vSANClusterHealth for Your Own Custom vSAN Health Reporting

I my made my own vSAN Health report based on Get-vSANinfo

You can find the script on my Github: Link

That script dit not get all info that i wanted. I use is for all my different homelabs.

Funtions: Cluster,Hosts,VMs,vSANVersion,vSanUpgrade,HealthCheckEnabled,TimeOfHclUpdate,StoragePolicy,vSanDiskClaimMode,faultdomaincount,ObjectOutOfcompliance,vSanOverallHealth,vSanOverallHealthDescription,vSanHealthScore,ComponentLimitHealth,OpenIssue,vSanFreeSpaceTB,vSanCapacityTB

Addons:
PerformanceServiceEnabled
PerformanceStatsStoragePolicy
faultdomaincoun
StretchedClusterEnabled
vSanFailureToTolerate (Works only in Second run, Work in Progress)

You schedule the script and send it to your e-mail

VCF MGMT domain

Homelab

Easy Way To Check vCenter Linked Partner Replication Status”

Symptoms

  • You see a error  vCenter Server Replication Status : RED . Replication state is broken for the node: vCenter2.wardvissers.nl
  • You may be missing one or more vCenters from the inventory tree

You can check vCenter Server Management vCenterurl:5480

Possible Causes

  • vCenter services failing
  • Low Disk Space
  • Network/firewall blocking communication
  • vCenter Platform Services Controller misconfiguration: VMware KB 2050273

Let’s Check it

Here are some commands you can run to see what vCenter is doing behind the scenes!

  • /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator
  • /usr/lib/vmware-vmdir/bin/vdcadmintool –> option 6
  • Check the logs
  • Check ping/ssh
  • More documentation here: VMware KB 2127057

How to Remove Inaccessible vSAN Objects in vSphere: Step-by-Step Guide

This post is about how to remove such an inaccessible object within vSAN.

Afbeelding met tekst, schermopname, nummer, LettertypeAutomatisch gegenereerde beschrijving

Open an SSH session to the vCenter and enter the command rvc localhost in the command line.

Navigate to the destinated vSAN cluster where you want to remove the inaccessible objects using cd and utilize ls to list in each step like this one:

Verify the state of vSAN objects using the command vsan.check_state -r . This check involves three steps:

  • Checking for inaccessible vSAN objects
  • Verifying invalid or inaccessible VMs, and
  • Checking for VMs for which VC/hostd/vmx are out of sync

During this check, as you can see in the next screenshot, there are four inaccessible objects with the same UUID as those listed in Virtual Objects within the vSphere Client.

Afbeelding met tekst, schermopname, LettertypeAutomatisch gegenereerde beschrijving

To remove them, open an SSH session to any ESXi in the cluster and use the following command /usr/lib/vmware/osfs/bin/objtool delete -u <UUID> -f replacing UUID with the one you want to remove. Afbeelding met tekst, schermopname, LettertypeAutomatisch gegenereerde beschrijving

After you remove all inaccessible objects and run the (vsan.checkstate -r .) once again, you should no longer see any inaccessible objects. Afbeelding met tekst, schermopname, software, LettertypeAutomatisch gegenereerde beschrijving

‘Ineligible for use by VSAN’ can’t be added to VSAN disk groups

I had the opportunity to test a Dell vSAN node. I had a older unattend install esxi iso.
This installed the ESXi OS on the wrong disk. After a correct install vSAN did not see this this disk ready for use for vSAN. Combining the following articles Dell VXRai vSAN Drives ineligible and identify-and-solve-ineligible-disk-problems-in-virtual-san/
I solved this problem with the following steps:

Step 1: Identify the Disk with vdq -qH

Step 2: Use partedUtil get “/dev/disks/<DISK>” to list all partitions:

partedUtil get “/dev/disks/t10.NVMe____Dell_Ent_NVMe_CM6_MU_3.2TB______________017D7D23E28EE38C”

Step 3: Use This disk has 2 partitions. Use the partedUtil delete “/dev/disks/<DISK>” <PARTITION> command to delete all partitions:

Afbeelding met tekst, Lettertype, schermopnameAutomatisch gegenereerde beschrijving

Step 4:

When all partitions are removed, do a rescan:

~ # esxcli storage core adapter rescan –all

Afbeelding met tekst, schermopname, software, ComputerpictogramAutomatisch gegenereerde beschrijving

Step 5: Claim Unused Disks

Afbeelding met tekst, software, Lettertype, nummerAutomatisch gegenereerde beschrijving

ESXi Unattend Install on Dell BOSS controller

I had the opportunity to test a Dell vSAN node. I had a older unattend install esxi iso.
This installed the ESXi OS on the wrong disk.

I hate to type a very complex password twice.
So automation is the key.
I love de ks.cfg install option

Sow following the following guide did not the trik:
https://www.dell.com/support/kbdoc/en-us/000177584/automating-operating-system-deployment-to-dell-boss-techniques-for-different-operating-systems

VMware ESXi Automated Install

This did not work:
install –overwritevmfs –firstdisk=”DELLBOSS VD”

After doing a manual install:
Afbeelding met tekst, schermopnameAutomatisch gegenereerde beschrijving

What works:

# For Dell Boss Controller “Dell BOSS-N1″

install –overwritevmfs –firstdisk=”Dell BOSS-N1”

Translate »