After three great sessions at VMUG Connect Amsterdam and VMUG Connect Online, and VMUGNL Summer Sessions. I’ve had requests to share the slides for my session “What You Can Learn from a Minimum Resources 2 Node VCF 9 Lab Deployment to a Real Scenario“.
The response was great.
In the slide deck you will find:
✅ The Start of My VCF Story
✅ First Challenge
✅ Why i wanted to build a real Senario
✅ Why i choise going to a build a real VCF scenario
✅ The Road
✅ The Lessions Learned & Next Steps
One of the first post-deployment tasks after installing VMware Cloud Foundation (VCF) 9.1 is configuring a password policy to ensure compliance with your organization’s security standards.
While the password policy is successfully applied to most managed accounts, you may notice that the root accounts of the VCF Operations appliance and the VCF Proxy appliance do not follow the configured password expiration policy.
As a result, the expiration date shown in VCF Management remains unchanged, even though a password policy has been configured.
You may observe one or more of the following:
• A password policy is configured successfully in VCF Management.
• Compliance checks complete without errors.
• The root account of the VCF Operations appliance still shows an incorrect or outdated password expiration date.
• The same behavior occurs on the VCF Proxy appliance.
This can be confusing because the password policy appears to be configured correctly, but it is not enforced for these Linux root accounts.
The password policy configured in VCF does not automatically update the Linux root account password aging settings on the VCF Operations and Proxy appliances.
Instead, these appliances continue to rely on the Linux ‘chage’ configuration to determine when the root password expires.
VMware has documented this behavior and provided a straightforward workaround.
https://knowledge.broadcom.com/external/article/441344/configured-password-policy-is-not-being.html
Before making any changes, enable SSH access on the VCF Operations appliance if it is currently disabled.
https://knowledge.broadcom.com/external/article/315976/enabling-ssh-access-in-aria-operations.html
SSH to the VCF Operations appliance using an administrative account.
ssh admin@<vcf-operations-appliance>
Run:
sudo chage -l root
Configure a 365-day password lifetime:
sudo chage -M 365 root
Run:
chage -l root
Repeat the same commands on the VCF Proxy appliance, either through SSH (if enabled) or via the VMware console.
The updated password expiration date is not reflected immediately in the VCF Management interface.
VCF periodically refreshes password information, so it may take 10–30 minutes (or longer depending on your environment) before the new expiration date appears.
• This change only affects the Linux root account.
• Verify the setting after appliance upgrades.
• Adjust the password lifetime (90, 180, 365 days, etc.) according to your security policy.
Although VCF 9.1 allows administrators to centrally configure password policies, the Linux root accounts on the VCF Operations and VCF Proxy appliances continue to rely on the local Linux password aging configuration.
Updating the password expiration with the chage command ensures that the root account complies with your organization’s password policy. Once VCF completes its next inventory synchronization, the correct expiration date is displayed in the VCF Management interface.
While deploying VMware Cloud Foundation (VCF) 9.1 in a homelab environment, the installation repeatedly failed during the ‘Deploy and Configure VCF Management Platform’ stage. Despite performing nine completely clean installations, the deployment consistently stopped at the same point.
The deployment task failed with the message: ‘Add VM Name Prefix to NSX Firewall Exclusion List’. The failure was identified in /var/log/vmware/vcf/domainmanager/domainmanager.log
Several Broadcom Knowledge Base articles appeared relevant, including KB440449 and KB 441122. Although the symptoms were similar, neither article fully resolved the issue.
The original VMSP configuration used a name value matching the prefix of the fleetFqdn. The configuration was modified to use a unique VMSP cluster name. While this appeared promising, the issue persisted.
Additional troubleshooting included changing VMSP IP ranges, rebuilding DNS records, validating forward and reverse DNS resolution, and reviewing deployment logs for networking issues.
The issue was ultimately not caused by the NSX firewall exclusion configuration. Multiple infrastructure issues contributed to deployment instability.
1. Configure a single authoritative NTP source, preferably the domain controller.
See the planning and preparation workbook
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-1/planning-and-preparation.html
2. Verify DNS records and name resolution.
3. Upgraded to a dedicated 10G Switch Ubiquiti UniFi Pro XG 8 PoE ipv 2.5GB Ubiquity Switch
4. Replace faulty network components.
Conclusion
Although the deployment failure appeared to indicate an NSX firewall exclusion issue, the underlying cause was network instability combined with infrastructure configuration problems. After correcting NTP configuration, validating DNS, upgrading network connectivity, and replacing the defective SFP+ module, the VCF 9.1 deployment completed successfully.
Upgrade VMware VCF 9.0.1: Essential Setting Up an Offline Depot on Ubuntu Instructions
If you are planning to upgrade to latest release of VMware Cloud Foundation (VCF) 9.0.1 and you what to install/upgrade you need to have a offline depot.
I had some struggle with the VCF Fleet upgrades. I thought the VCF installer Offline Depot was sufficient. I seems not.
William Liam did a nice explaining about the options: VCF Software Depot Structure Deep Dive for Install & Upgrade. Which confirms my struggle.
Sow I build my own offline depot
First, download the required bundle files from the Broadcom VMware portal. This includes:
Sizing for the disk is minimal 100GB I used 200GB thin Provisiond
I did a Ubuntu install on a 200GB disk (Partition without lvm)
Login with the user account that you create
Sudo passwd root
Vim /etc/ssh/sshd_config
if the following line exists, possibly commented out (with a # in front):
PermitRootLogin
Then change it to the following, uncommenting if needed (remove the # in front):
PermitRootLogin yes
sudo service ssh restart
now you can login als root
Sudo update
sudo apt install apache2 openssl apache2-utils unzip
nano ~/vcf-openssl.cnf
Paste the following:
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
x509_extensions = v3_req
[ dn ]
C = US
ST = CA
L = LA
O = TS
OU = IT
CN = flt-depot.wardhomelab.nl
emailAddress = a@b.c
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = flt-depot.wardhomelab.nl
IP.1 = 192.168.150.246
Replace the values with your organization’s details.
Generate the certificate
cd /etc/apache2
sudo mkdir ssl
sudo openssl req -x509 -nodes -days 365 \
-newkey rsa:2048 \
-keyout /etc/apache2/ssl/vcf.key \
-out /etc/apache2/ssl/vcf.crt \
-config ~/vcf-openssl.cnf
Sudo htpasswd -c /etc/apache2/.htpasswd vcfadmin
sudo nano /etc/apache2/sites-available/default-ssl.conf
Paste the following:
<VirtualHost *:443>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/vcf.crt
SSLCertificateKeyFile /etc/apache2/ssl/vcf.key
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
RequestHeader unset Proxy early
<Directory /var/www/html>
Options Indexes FollowSymLinks
AllowOverride None
AuthType Basic
AuthName “VCF Depot”
AuthUserFile /etc/apache2/.htpasswd
Require valid-user
</Directory>
</VirtualHost>
sudo a2enmod ssl headers
sudo a2ensite default-ssl
sudo systemctl restart apache2
Sudo chmod 777 /home
Upload vcf-9.0.1.0-offline-depot-metadata.zip /home/ with Winscp
Sudo unzip vcf-9.0.1.0-offline-depot-metadata.zip -d /var/www/html
Make sure this file exists:
ls /var/www/html/PROD/metadata/productVersionCatalog/v1/productVersionCatalog.json
sudo rm -f /var/www/html/index.html
Create your Download token in de Broadcom portal
Create a token file in /home folder
Sudo mkdir /home/vcf-download-tool
Sudo chmod 777 /home/vcf-download-tool
cd /tmp/vcf-download-tool
Upload the tool in /tmp/vcf-download-tool folder
tar -xf vcf-download-tool-9.0.1.0.24962179.tar.gz
sudo reboot
Cd /tmp/vcf-download-tool/bin
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –automated-install –type=INSTALL
sudo ./vcf-download-tool binaries list –depot-download-token-file=/home/token –vcf-version=9.0.1 –type=INSTALL –sku=VCF
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component SDDC_MANAGER_VCF
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VCENTER
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRSLCM
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VROPS
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component NSX_T_MANAGER
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VCF_OPS_CLOUD_PROXY
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRA
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRNI
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRLI
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component HCX
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VRO
sudo ./vcf-download-tool binaries download –depot-download-token-file=/home/token -d /var/www/html –vcf-version=9.0.1 –type UPGRADE –component VIDB
Upgrade Binaries will be visible
Open de VM console
Login with the root user
Vim /etc/ssh/sshd_config
if the following line exists, possibly commented out (with a # in front):
PermitRootLogin
Then change it to the following, uncommenting if needed (remove the # in front):
PermitRootLogin yes
systemctl restart sshd
scp root@ftt-depot.wardhomelab.nl:/etc/apache2/ssl/vcf.crt /tmp/vcf.crt
Import the certificate into the Java truststore
sudo keytool -import -trustcacerts -alias vcfDepotCert1 \
-file /tmp/vcf.crt \
-keystore /usr/lib/jvm/openjdk-java17-headless.x86_64/lib/security/cacerts \
-storepass changeit
Reboot
Now u should connect VCF to you offline depot
You want you upgrade to 9.0.1 ivm Edge Issue
Special thanks to vmtechnics for putting me in the right direction
What are my things I would like to test
The VCF Cloud Foundation Installer makes lives a lot easier! More about this coming very soon
The VCF Operations Console is looking good! I used it in the VCF 9 beta
More about this also later!
For your VCF homelab you wan to keep the resources small with a little bit overhead.
In this post I will talk about how i managed to deploy a VCF Workload Domain with a single NSX Manager, instead of the standard three nsx nodes.
Warning: Use this only in a Homelab!
The trick is to SSH into your SDDC Manager using the vcf user, and the password used during bring-up of the management domain.
When logged in, run su and log in as root using the password used during bring-up.
run: vi /etc/vmware/vcf/domainmanager/application-prod.properties
Hit i in your keyboard to go into insert mode. Go to the end of the file, and append the following:
nsxt.manager.formfactor=medium
nsxt.manager.resources.validation.skip=true
nsxt.manager.cluster.size=1
nsxt.manager.wait.minutes=120
This will make it so that any workload domain you deploy has one NSX Manager, and that it uses a smaller size. Once done, hit ESC in your keyboard, then type :wq and hit enter to save the file. (w = write, q = quit).
Then run systemctl restart domainmanager and you are good to go!
This worked in my nested Cloud Foundation deployment in my lab running 5.2.1.0.
You will still have to fill in the information for the extra nodes in the UI.
When you build your VCF Lab environment you want to create your DNS records automatically. I use for DNS a Windows Server.
The Script:
function ConvertTo-DecimalIP {
param ([string]$ip)
$parts = $ip.Split(‘.’) | ForEach-Object { [int]$_ }
return ($parts[0] -shl 24) + ($parts[1] -shl 16) + ($parts[2] -shl 8) + $parts[3]
}
function ConvertTo-DottedIP {
param ([int]$intIP)
$part1 = ($intIP -shr 24) -band 0xFF
$part2 = ($intIP -shr 16) -band 0xFF
$part3 = ($intIP -shr 8) -band 0xFF
$part4 = $intIP -band 0xFF
return “$part1.$part2.$part3.$part4”
}
$zone = “testlab.nl”
$startip = “192.168.200.10”
$dnsrecords = “vcf-m01-cb01″,”vcf-m01-sddcm01″,”vcf-m01-esx01″,”vcf-m01-esx02″,”vcf-m01-esx03″,”vcf-m01-esx04″,”vcf-w01-esx02″,”vcf-w01-esx03″,”vcf-w01-esx04″,”vcf-w01-esx04″,”vcf-m01-nsx01a”,”vcf-m01-nsx01b”,”vcf-m01-nsx01c”,”vcf-m01-nsx01″,”vcf-w01-nsx01a”,”vcf-w01-nsx01b”,”vcf-w01-nsx01c”,”vcf-w01-nsx01″,”vcf-m01-vc01″,”vcf-w01-vc01″
$count = $dnsrecords.count
# Convert start IP to decimal
$decimalIP = ConvertTo-DecimalIP $startIP
$i = 0
# Loop and print incremented IPs
foreach ($dnsrecord in $dnsrecords) {
$i -lt
$count;
$i++
$currentDecimalIP = $decimalIP + $i
$currentIP = ConvertTo-DottedIP $currentDecimalIP
Add-DnsServerResourceRecordA -Name $dnsrecord -ZoneName $zone -AllowUpdateAny -IPv4Address $currentIP -CreatePtr
Write-Output “DNS record $dnsrecord in $zone with $currentIP is created” -ForegroundColor Green
By passing either of the new VCP-VCF level certification exam(s), anyone maintaining an active VMUG Advantage membership can receive 3 years worth of extensive VMware Cloud Foundation licensing for home lab use!
The VMUG Advantage program has offered affordable home lab VMware licensing packages for years, but did cover most over the entire product portfolio.
Last year Broadcom made a change into this.
Option 1: Get vSphere Standard Edition 32 cores for 1 year: Pass one of the following VCP certification exams
Option 2: Get VMware Cloud Foundation (VCF) 128 cores for 3 years: Purchase & Maintain VMUG Advantage, pass the following VCP certification exam.
A VMUG Advantage membership was complimentary for vExperts in 2025
The membership is $210 otherwise, and does include a voucher for a 50% discounted VCP-VCF exam
With the requirements in place, head to the Broadcom “VCP Certification Non-Production Licenses” portal and request licenses.
On the 4th Oktober VMware Skyline was end of life.
VMware Skyline was great:
• Proactive Issue Identification
• Automated Insights
• Health Scans and Remediation
• Integration with support
VMware by Broadcom are building critical Findings and Self-Help recommendations directly in product starting with VCF (from 5.2) and Aria Operations (from v8.18 July 2024)
Many of the other Skyline features are being planned for inclusion in future releases in Cloud Foundation and Aria Operations. We will see what the future will bring.
But for now how do you get this working.
First Step:
Update Aria Operations to 8.18.2 (Lastest)
Second Steps:
1. vCenter (Don’t for get enable vSAN), NSX, VCF, Aria vRA
2. Configure log collection in Aria Logs for the following components:
• Configure vCenter server integration in Aria for Logs
• Configure log forwarding on vCenter server, ESXi hosts(automatically in Aria FOR logs), and SDDC manager
3. Integrating VMware Aria Operations for Logs and VMware Aria Operations
4. Connect Skyline Health Diagnostics (SHD)
5. In Aria LoginSight check the vRops integration checkboxes
Bij default Enable launch in context can be disabled when configured at first.
After upgrading and checking the settings its finally working 😊 (It can take some time).
Microsoft has identified an issue that affects Windows Server domain controllers (DCs), and has expedited a resolution that can be applied to affected devices. Out-of-band (OOB) updates have been released for some versions of Windows today, March 22, 2024, to addresses this issue related to a memory leak in the Local Security Authority Subsystem Service (LSASS). This occurs when on-premises and cloud-based Active Directory domain controllers service Kerberos authentication requests.
This issue is not expected to impact Home users, as it is only observed in some versions of Windows Server. Domain controllers are not commonly used in personal and home devices.
Updates are available on the Microsoft Update Catalog only. These are cumulative updates, so you do not need to apply any previous update before installing them, and they supersede all previous updates for affected versions. If your organization uses the affected server platforms as DCs and you haven’t deployed the March 2024 security updated yet, we recommend you apply this OOB update instead. For more information and instructions on how to install this update on your device, consult the below resources for your version of Windows:
Note: The OOB release for Windows Server 2019 will be released in near term.
You must be logged in to post a comment.