Even though it’s a financially unwise decision right now, I decided to buy a third Minisforum MS-A2 because of the upcoming VCF 9.1 release and because I wanted to test other services like SSP in my lab. Luckily, I still had some RAM in my drawer from the good old days, when you didn’t have to sacrifice your firstborn or donate a kidney to get 128 GB of RAM. Yes, friends of sophisticated over-engineering, I am a blessed man.

But joking aside, this blog post is actually supposed to be about how I can expand my existing management domain, since I don’t want to reinstall everything—and to be honest, I’m not sure yet if it will all work, because my management domain currently consists of two clusters. The first cluster runs on two AMD MS-A2 servers with memory tiering and hosts my fleet, and the second cluster is a nested cluster on MS-01 servers (yes, the ones with the Intel CPUs). On top of that, I’m also running a nested instance of VCF9 that’s onboarded into my fleet. Of course, not everything can be running at the same time, since I don’t have enough RAM and processing power. So it’ll be interesting to see if the whole thing will work out somehow.

So if you’re able to read this blog, it must have worked out somehow; if not, you’ll never know.

Let’s Get Ready to Rumble

It all starts off quite innocently: first, unpack the new server, install the NVMEs—you know the drill. Then I install a fresh ESX 9.0.2 image and configure the network, DNS, NTP, memory tiering, and the Ryzen workarounds. If you want to read exactly what needs to be done, you can check out this article. I’ve written everything down in detail there.

The goal should be to have a basic ESX9 host that can resolve DNS, has SSH enabled, has NTP working, has over 377 GB of RAM (memory tiering), and has only one network adapter connected. In other words, it should be exactly the same as if I were installing VCF9 from scratch.

Now that these preparations have been made, I can get started on the actual cluster expansion. To do this, I first need to check my network pool settings and, if necessary, adjust the IP ranges for the NFS and vMotion networks.

Network Pools

The network pools are a bit hidden. In VCF 5.x, this was something you configured in the SDDC Manager. In VCF 9, this has now moved to vCenter under Global Inventory -> Hosts -> Network Pools.

Fortunately, I was a bit more generous here and still have some IP addresses available.

The TEP IP pool is managed in NSX and, of course, must have two free IP addresses per host. In my case, it just barely works. The pool can be adjusted here as well.

NSX is significantly more flexible than VCF in vCenter when it comes to the network pool. Once everything has been checked, I can now add the host.

Host onboarding

Onboarding is no longer done via the SDDC as it used to be, but must now also be done via the global inventory in vCenter. To do this, go to Global Inventory List -> Hosts -> Unassigned Hosts -> COMMISSION HOSTS

After that, you’re immediately greeted with a friendly checklist of everything that needs to be done:

- Host for vSAN/vSAN ESA/vSAN Storage workload domain should be vSAN/vSAN ESA/vSAN Storage compliant and certified per the VMware Hardware Compatibility Guide. BIOS, HBA, SSD, HDD, etc. must match the VMware Hardware Compatibility Guide.
- Host has the drivers and firmware versions specified in the VMware Compatibility Guide.
- Host has ESXi installed on it. The host must be preinstalled with supported versions (9.0.2.0.25148076)
- Host is configured with DNS server for forward and reverse lookup and FQDN.
- Hostname should be same as the FQDN.
- Management IP is configured to first NIC port.
- Ensure that the host has a standard switch and the default uplinks with 10Gb speed are configured starting with traditional numbering (e.g., vmnic0) and increasing sequentially.
- Host hardware health status is healthy without any errors.
- All disk partitions on HDD / SSD are deleted.
- Ensure required network pool is created and available before host commissioning.
- Ensure hosts to be used for VSAN workload domain are associated with VSAN enabled network pool.
- Ensure hosts to be used for NFS workload domain are associated with NFS enabled network pool.
- Ensure hosts to be used for VMFS on FC workload domain are associated with NFS or VMOTION only enabled network pool.
- Ensure hosts to be used for vVol FC workload domain are associated with NFS or VMOTION only enabled network pool.
- Ensure hosts to be used for vVol NFS workload domain are associated with NFS and VMOTION only enabled network pool.
- Ensure hosts to be used for vVol iSCSI workload domain are associated with iSCSI and VMOTION only enabled network pool.
- For hosts with a DPU device, enable SR-IOV in the BIOS and in the vSphere Client (if required by your DPU vendor).

Of course, I’ve carefully checked all of this and can confirm it.

After specifying the correct network pool and entering the correct storage type, fqdn, username, and password, the validation process failed with a certificate error—but why? After all, I created a new self-signed certificate during installation.

The problem is—and this is something the pre-check doesn’t tell you—that once you’ve rolled out your own certificates in your domain, the host certificate must be from the same CA. So I have to create a CA request via the ESX GUI, submit it to my Microsoft CA, and implement it on my host. I love certificates—not.

If you’ve never done this before, you can do it in the ESX GUI under Host -> Manage -> Security & Users -> Certificates. After the certificate exchange, the validation process completes successfully. The ESX server does not need to be restarted. Once the commissioning is complete, the host should now appear under “Unassigned Hosts.” That means half the work is already done.

Extend Cluster

Now comes the fun part: actually expanding the cluster. To do this, right-click on the existing cluster -> Add Host -> Add Unassigned Hosts. Here, too, the process differs from VCF 5.x, as this task is not performed in the SDDC.

The process itself is pretty straightforward, and the most important thing is that the uplink assignment on the distributed switch is correct. That’s actually the only potential source of error at this stage.

Here, too, there is a brief validation step after confirmation, and if no errors occur, the process should run fully automatically. The process can be monitored in both SDDC and vCenter. Personally, I find the view in the SDDC a bit clearer than the one in the vCenter Recent Tasks view. You can also check the progress of the host’s NSX configuration in NSX Manager. After about 10 minutes, the whole thing was done and dusted, and my management domain had been successfully expanded to three nodes.

To perform a manual validation, I booted up a test VM connected to an NSX network and ran a quick connectivity check to make sure my TEP network was working properly. Unfortunately, with VCF 9.0.2 and the MS-A2, I can no longer see the TEP tunnel status displayed correctly in NSX. The status simply remains gray—unknown. However, this is purely a visual issue, and other 9.0.2 users with the same hardware are experiencing the same problem.

Conclusion

What can I say? I expected the certificate error—I’ve fallen for that one before. But to make this article a bit more useful, I decided to highlight this error again. Generally speaking, though, I would have expected more problems, since one SDDC Manager (the one from my other nested domain) is unreachable, so I was pleasantly surprised. That’s because things like an inventory scan don’t run error-free unless all vCenters and SDDC Managers are accessible, and I had already envisioned having to get those two components online somehow to complete the expansion.

The most important thing when expanding is careful preparation and ensuring that all pre-checks are carried out. There’s a good reason why the checklist is included in the onboarding dialog. If you’ve followed all the steps, the expansion will indeed go smoothly.

Wow, so here we are again. Another travel blog post and no tech content. Yes, unfortunately I’m a bit behind on my own “roadmap”, but right now I simply don’t have the personal time to write another tech article. I actually still have so much in the works and just can’t seem to get through it all. But that won’t stop me from sharing my impressions of VMUG Connect in Amsterdam here.

Day 0 - Day of travel

My trip started early Monday morning with the Deutsche Bahn, traveling via Mannheim and Düsseldorf to Amsterdam Centraal—a surprisingly uneventful train ride. I was practically right on time, caught all my connections, and everything just went smoothly—crazy. To be honest, I’m not really used to that.

I bought a metro ticket directly online via the Metro app (without having to register—yeah, that’s how it can be done, Deutsche Bahn), since my hotel was a bit out of the way and I had to take the metro about 16 minutes to the RAI and back every morning/evening. Writing this makes me realize that I really need to get my travel expense report done.

Well, anyway, I checked into the hotel and then spent some time exploring Amsterdam. All in all, the weather wasn’t great that day, so I headed back to my hotel pretty quickly and relaxed a bit.

It wasn’t my first visit to Amsterdam, after all, and I wasn’t really in the mood for sightseeing anyway. So you could say it was a completely unspectacular day of arrival.

Day 1 - PreConnect

The first day started off quite leisurely with a nice breakfast at the hotel, and afterward I met up with my colleague Steffen Richter in front of the Rai to pick up our badges and check out the venue. The expo was still a work in progress at that point, and we ran into the first few familiar faces. Speaking of faces, no trip would be complete without a happy face selfie. I know you didn’t ask for it, but here it is.

But let’s get to the official part. The first session I attended was a so-called PreConnect Session VCF: Deployment, Automation & Networking, presented by and featuring John Nicholson. It wasn’t a traditional session with a slide deck and such, but more like an open discussion. Unfortunately, there was an error in the VMUG app that listed me as a co-speaker, but I wasn’t informed beforehand that I had a second session. I spoke briefly with John Nicholson later, and he didn’t know either that we were supposed to have a session together. I was asked about it several times and can only say sorry, but something went wrong with the organization. For me, it remains Schrödinger’s speaker slot. You never really know whether I should have participated or not. :D

After that, I attended the PreConnect session Security: All Things Security. As before, this wasn’t a traditional session but a panel discussion, and what can I say—Chris McCain doesn’t need a microphone. Some people in the audience said he was “American loud,” and that wasn’t meant negatively, but rather as a compliment. The panel discussion was quite entertaining, even if it didn’t really offer anything new to me.

Unfortunately, I missed my colleague Maria Kmita’s session (sorry, I would have loved to see it, but you know I tend to get sidetracked). These PreConnect sessions were something new for me, and I found them really entertaining. Above all, they were twice as long as regular sessions, and we had the chance to ask all sorts of questions—both the obvious and the unexpected.

After that, i headed over to the expo for some casual “networking”—and by “networking,” I actually mean drinking beer and shooting the breeze with friends, old buddies, and new faces. The beer was actually not too bad, by the way. Yeah, I’m still German at heart, and the fact that I didn’t complain is praise enough.

Day 2 - Here we go

Today is actually the first real day of VMUG Connect. It kicks off with the general session at 9 a.m. Fresh and rested (okay, I’m lying), I showed up right on time just before 9 a.m. and listened intently to Duncan Epping as he talked about what’s new. But dude, what kind of session title is that? From Roadmap to Implementation: Key Innovations Redefining Storage and Cyber Resilience for VCF 9! Sounds like a tongue-twister for me. The general session was split into two parts, and Chris McCain also had a segment titled Shrink the Blast Radius: Private Cloud Segmentation Made Easy by Chris McCain. I have to say that DR isn’t really my area of expertise (sorry, Duncan), and the vDefend session didn’t have a ton of new stuff for me either, but Brad Tomkins did a wonderful job moderating the general keynote, and it was definitely entertaining.

After the opening session, I went straight to the Redefining the Private Cloud: The Next Evolution of VCF session without a break. The session offered some exciting insights into upcoming VCF versions. Among other things, it was mentioned that stateful services will soon be possible in NSX without edges. They also demonstrated what the VXLAN/EVPN integration will look like in version 9.1. We have some exciting new features coming our way, especially in NSX.

After that, I took some time to myself to mentally prepare for my VCAP Operations exam, which I was able to take for free thanks to VMUG Advantage. I passed, so I was able to relax and enjoy the rest of the day.

My next session was led by Giovanni Dominoni and Amedeo Simone Luciano and had the promising title Zero-Impact Network Fabric Migration: From Cisco Nexus 9K to ACI with NSX Environments. It provided a practical demonstration of how to approach a network platform migration without any downtime, and it was truly inspiring to see how the two of them implemented this project. This led to some nice discussions after the session.

Since I ended up chatting with so many people again, I wasn’t able to attend any more sessions that day. But let’s be honest—talking to people is just as important as any session, especially when you work for a partner.

The welcome reception took place that evening. It was held at Strandzuid, a cozy waterfront restaurant beautifully situated right next to the Rai Convention Center, from which you could take a pleasant walks around the lake. I would also like to take this opportunity to mention comdivision, which sponsored the entire welcome reception and treated us all to a wonderful evening.

2026 Amsterdam Welcome Reception

2026 Amsterdam Welcome Reception2

2026 Amsterdam v Beer

2026 Amsterdam More Happy Faces

Day 3 - or the day my legs were shaking

This day was a particular focus for me because I gave my first session at an international event—all in English, with a live demo, and on a spotty internet connection. Naturally, I was pretty nervous leading up to the day. Of course, this wasn’t the first session I’ve ever given, but I think I’ll always be nervous.

Once again, I talked about one of my favorite topics, which is VPCs in NSX, or rather VCF9. What can I say? I just love the feature and the possibilities it offers. I was also very surprised by the audience—for one thing, the room was almost completely full, and for another, at least 80% of the people in the session were using NSX.

I wasn’t used to that at all. So I cut the general NSX section short and focused more (though not enough) on the topic of traffic flow. For anyone who’d like my slides, I’m making them available for download here. Unfortunately, due to space constraints, only the PDF version is available, but that should suffice.

After my session, I had a quick meal, and then, unfortunately, I had to head home again. Since you can always count on Deutsche Bahn, my connecting train in Mannheim was so late that I actually ended up getting home earlier because I was able to catch a train that was supposed to have already left. It’s quite a feat to get home faster thanks to a delay.

Conclusion

Compared to the VMUG in St. Louis, I’d say the Amsterdam event had a better venue. The expo was also somehow better organized. To be fair, it’s worth noting that Connect in St. Louis was the first of its kind and certainly just a trial run. As always with these kinds of events, it’s great when the community comes together and you finally get to see all those people in person whom you otherwise only know from LinkedIn, the vExpert Discord, or local VMUG events. For me, that’s actually always the highlight of every event and, in my view, the very heart of VMUG. I also had great conversations with Duncan Epping, Chris McCain, and other VMware leaders, and of course plenty of chats with vExperts, VMware users, partners, and customers. In the end, that’s what makes these events special for me—not the sessions or any slides, but the direct interaction.

Many thanks to everyone I had the pleasure of sharing the event with. ❤️❤️❤️

IPv6 has been around as a standard since 1998—so it feels like forever. I was 16 years old at the time. Since then, we’ve been hearing that IPv4 addresses are running out and IPv6 is the solution. And although IPv6 is now 25 years old (do you feel as old as I do now?), it is still far from being standard. To be honest, in the last 25 years, I’ve had exactly one customer who uses IPv6 in their data center, and that’s only as a dual stack.

According to estimates by Cloudflare from 2023, global acceptance of IPv6 is around 35.9%. ChatGPT puts it at around 45%, but we are still a long way from widespread adoption. PS: My blog can be accessed with IPv6 native.

Nevertheless, I wanted to take a look at the status quo and set up a working IPv6 setup with NSX, but unfortunately it’s not as easy as I had imagined. Perhaps I should briefly mention the framework conditions. I have a consumer DSL connection from Deutsche Telekom. They generously provide me with a public IPv4 and a /56 IPv6 prefix. I already have my OPNsense appliance running in dual stack. I will only discuss this configuration to a limited extent, as it can be different for each operator.

Unfortunately, my IPv4 and IPv6 are dynamic because I have a normal private customer connection and not a business tariff. Thanks for that – not. Unfortunately, this also means that I have to come up with something else for the IPv6 prefixes for NSX. More on that later. I also have a Mikrotik router between OPNSense and the NSX T0 router.

The current setup is deliberately simple. My T0 router is active/standby at this point and I only use one uplink VLAN. The reason for this is simple. The entire setup runs on a small NUC Ultra 7 with only one LAN card. Any HA setup would be excessive here. But before we jump into the setup, we need to talk about a few IPv6 basics.

GUA / ULA / Link Local and other IPv6 basics

GUA

We start with what is probably the simplest prefix range, the GUA or Global Unicast. This prefix is publicly routable on the Internet and starts with 2000::/3.

ULA

The ULA prefix is the Unique Local Address range and is comparable to the private IP ranges of IPv4. These are not routed on the Internet and play an important role in my setup, as I unfortunately get assigned a dynamic prefix by my provider. ULA starts with fc00::/7 The first bit following the prefix indicates, if set, that the address is locally assigned. This splits the address block in two equally sized blocks, fc00::/8 and fd00::/8. fd00::/8 is the range that is actually used for private addressing. NSX uses an fc00::/8 network by default for hotplug networks between T0 and T1 routers. For this reason alone, you should avoid fc00::/8 and use fd00::/8 networks for private addressing, even though the risk of having duplicate IPs is very low.

Link-Local

LL is automatically available on every interface. Used for Neighbor Discovery, Router Advertisements, and often as BGP Next-Hop. Only valid in the local Layer 2. LL starts with fe80::/10.

Multicast

Replaces broadcast in IPv6. Used for neighbor discovery, DAD, and router advertisements. Multicast can be recognized by the following prefix: ff00::/8.

IPv4-mapped IPv6

IPv4 addresses in IPv6 format, e.g., for MP-BGP, when IPv6 routes are transported over an IPv4 session. The prefix is ::ffff:0:0/96 I had planned to use this for my NSX networks, but my OPNSense couldn’t handle it. My IPv6 routes weren’t in the routing table. I couldn’t figure out whether it was due to OPNSense’s FRR implementation or Mikrotik.

Prefix Delegation

My internet provider gives me a GUA /56 prefix via prefix delegation. Unfortunately, as already mentioned, this is dynamic. My OPNSense supports dynamic prefixes, and thanks to /56, I can create up to 256 /64 networks and assign these individual VLAN interfaces to the OPNSense. Unfortunately, NSX does not support this, which is why I ultimately had to resort to NAT66 and ULA.

SLAAC

SLAAC stands for Stateless Address Autoconfiguration and enables a host to automatically generate its IPv6 address from a /64 prefix announced by the router. To do this, the router sends router advertisements containing the prefix and the default gateway. The client then independently forms its complete IPv6 address without the need for a DHCP server to assign the address.

DHCPv6

DHCPv6 provides IPv6 addresses and additional network information centrally via a server. It can either assign the complete address (stateful) or only provide additional information such as DNS (stateless). If DHCP runs in stateless mode, SLAAC is used for IP address assignment.

NAT66

NAT66 is the IPv6 variant of Network Address Translation, in which the source IPv6 address is replaced by another IPv6 address when leaving the network. It is often used when ULA addresses are used internally and the provider assigns a dynamic global prefix. NAT66 works statefully, which means that the router stores connection states for reverse translation. This keeps the internal address design stable, regardless of the external prefix. However, NAT66 contradicts the end-to-end principle of IPv6.

NSX Supportet IPv6 features

• IPv6 and Dual-Stack Overlay Segments
• IPv6 Distributed Routing (Tier-0 / Tier-1)
• IPv6 BGP (MP-BGP)
• IPv6 Distributed Firewall (L3/L4)
• IPv6 Gateway Firewall
• IPv6 Load Balancer VIPs
• NAT66 support
• DHCPv6 (Stateful & Stateless)
• SLAAC support

NSX DAD-Profile

I could make a DAD joke now, but…

Router: “Doctor, it hurts when IP.” Doctor: “Then stop using IPv6.”

Okay, let’s not go there, that’s awful…

Duplicate Address Detection (DAD) in NSX is an IPv6 mechanism that ensures that an IPv6 address is unique within a segment before it is actively used. NSX uses the standardized Neighbor Discovery Protocol to check whether an address is already being used by another VM, thereby preventing address conflicts.

NSX ND-Profile

An ND (Neighbor Discovery) profile in NSX defines parameters such as router advertisement DHCPv6 mode (stateful/stateless) and SLAAC behavior for a segment. The ND profile controls how IPv6 addresses are assigned and which default gateway information is distributed to the workloads.

That was quite a lot

Now that we’ve refreshed the basics, let’s move on to the setup.

Configuring the OPNSense and Mikrotik router

I decided to set up my routing dynamically with BGP and fell into a few traps. Since I can’t do much with my 4,722,366,482,869,645,213,696 public IPv6 addresses (that’s an unimaginable 4.7 sextillion IP addresses) because they are not permanent, I have decided on a ULA setup and will then use NAT66. I have decided to use my ULA networks from the fd11:22:33::/56 prefix range. For one thing, it is still completely unused and relatively easy to read. My DNS servers have long been IPv6-capable in my network and have their own ULA IPs, which are somewhat arbitrary, however. Historically grown, as we say in Germany. To keep everything clear, here is a short table:

The target setup is shown as a graphic.

First, I configure my static IPv6 on my OPNSense on my BGP interface. It is important to select the IPv6 configuration type. This must be set to Static IPv6 and not Track Interface. With Track Interface, I would take a /64 prefix from my GUA IP address range, which unfortunately is not static.

It is also important that I assign the IP address cleanly with /64, otherwise the network cannot be used for SLAAC, and SLAAC is what we want to use on the other end, as all devices and operating systems that are dual-stack capable can do this.

Next, IPv6 must be enabled on the Mikrotik. This requires a reboot and can be configured under IPv6 -> Settings -> Disable IPv6. By default, the checkbox is selected and must be deselected. One might almost think that none of the vendors are really interested in IPv6.

But let’s get to the first problem I had. My lab already had a BGP configuration between the lab router and OPNSense via IPv4. OPNSense Business Edition does not properly support IPv6 route exchange via IPv4. I saw my IPv6 routes in the FRR plugin, but not in the routing table of the OPNSense.

Conversely, it doesn’t work either, so I couldn’t get this to work. OPNSense simply didn’t want to learn IPv6 routes over IPv4. After an hour of troubleshooting, I gave up and configured a normal IPv6 and IPv4 BGP session.
The settings are relatively straightforward. I am listing my settings here as an example. However, there is nothing special about them.

 ;;; OPNSenseIPv6
     name="OPNSenseIPv6" instance=OPNSense 
     remote.address=fd11:22:33:ff02::1/128 .as=65101 
     local.default-address=fd11:22:33:ff02::2 .role=ebgp 
     connect=yes listen=yes routing-table=main as=65102 nexthop-choice=default hold-time=3m 
     keepalive-time=1m afi=ipv6 use-bfd=yes 
     output.redistribute=connected,bgp 

Here are the OPNSense settings, but they are pretty standard. Thanks to my new Hugo Gallery shortcode, it’s now easier to view them.

Ipv6 Nsx Opnsense Routing

Ipv6 Nsx Opnsense Neigbor

Now that that’s done, I can finally get started on the NSX setup.

NSX IPv6 setup - it just works?

Of course not! Anyone who thought it would all be easy is mistaken. What I didn’t really know—but what a quick glance at the documentation revealed—is that NSX has IPv6 forwarding disabled by default. Apparently, no one really wants to use IPv6.

So I quickly jumped into the NSX Global Network config and enabled IPv6 L3 forwarding. For IPv6 to work, the T0 router must receive IPv6 addresses in addition to IPv4 addresses.

Next, I configure my IPv6 routing, and this is where RFC 5549 comes into play. To minimize the number of BGP sessions and IPv4 addresses, you can exchange both IPv4 and IPv6 routes over a single IPv6 BGP session.

Mikrotik now also supports RFC 5549. The Mikrotik configuration is very similar to the OPNSense configuration. The main difference is that both IPv4 and IPv6 address families are enabled in one session.

;;; BGP_VCF09-E01-Edge01
     name="VCF09-E01-Edge1-01" instance=VCF09-E01-Edge1-U1 
     remote.address=fd11:22:33:ff01::2/128 .as=65004 
     local.default-address=fd11:22:33:ff01::1 .role=ebgp 
     connect=yes listen=yes routing-table=main as=65102 hold-time=3m keepalive-time=1m afi=ip,ipv6 
     use-bfd=yes 
     output.redistribute=connected,bgp .default-originate=if-installed 

;;; BGP_VCF09-E01-Edge02
     name="VCF09-E01-Edge2-01" instance=VCF09-E01-Edge1-U1 
     remote.address=fd11:22:33:ff01::3/128 .as=65004 
     local.default-address=fd11:22:33:ff01::1 .role=ebgp 
     connect=yes listen=yes routing-table=main as=65102 hold-time=3m keepalive-time=1m afi=ip,ipv6 
     use-bfd=yes 
     output.redistribute=connected,bgp .default-originate=if-installed 

Next, the routing must be created on the T0 router. It is important to note that an IPv6 session must be created and an IPv6 route filter must be added to the default IPv4 route filter. Otherwise, the routing will not work. The rest of the configuration is standard. I always use BFD in my setups for fast network convergence, but it is not a must.

Ipv6 Nsx Edge Neigbor

Ipv6 Nsx Edge Routefilter

BGP peering should now be up and running. The IPv4 routing table on the Mikrotik looks a little strange. The immediate gateway shows the link local IP address of the NSX Edge Node. The IPv4 address is only how RouteOS represents IPv4 over IPv6 and is not actually used.

Ipv6 Nsx Ipv4 Over Ipv6

Ipv6 Nsx Ipv6

In the traceroute of the Mikrotik switch, you can also clearly see that for IPv4 networks, the next hop is actually the IPv4 address of the active EdgeVM.

[admin@router.lab.home] > tool/traceroute 10.10.0.1
Columns: ADDRESS, LOSS, SENT, LAST, AVG, BEST, WORST, STD-DEV
#  ADDRESS      LOSS  SENT  LAST   AVG  BEST  WORST  STD-DEV
0  10.28.25.12  0%       3  0.6ms  0.5  0.4   0.6    0.1    
1  10.10.0.1    0%       3  0.3ms  0.3  0.2   0.4    0.1  

IPv6 segments, ND and DAD…

Now that the routing part is done, it would have been much faster if I had used static routing, but that would also have been much more boring. Now I can finally create segments. And yes, I know I already created some for my tests, because without segments, there are no networks and therefore no routing test. But just imagine if I hadn’t done that yet.

In my setup, I decided to build a T1 router for my IPv6 segments and a T1 router for my IPv4 networks, but why did I do that? Firstly, for clarity, and secondly, because of the ND profile, which is still needed. The ND profiles can be defined either globally at the T0 level or more specifically at the T1 level. NSX supports 5 modes for ND:

• Disabled - Router advertisement messages are disabled.
• SLAAC with DNS Through RA - The address and DNS information is generated with the router advertisement message.
• SLAAC with DNS Through DHCP - The address is generated with the router advertisement message and the DNS information is generated by the DHCP server.
• DHCP with Address and DNS through DHCP - The address and DNS information is generated by the DHCP server.
• SLAAC with Address and DNS through DHCP - The address and DNS information is generated by the DHCP server. This option is only supported by NSX Edge and not by ESX hosts.

The standard is SLAAC with DNS Through RA, but as I already mentioned, problems can arise with some operating systems or devices. Specifically, some devices or operating systems cannot use Route advertisment for DNS. Alpine Linux is one such case. I do get a valid IPv6 and can ping my DNS server with it, but name resolution does not work, and without that, the internet is not particularly useful. The DNS servers were, of course, configured correctly in the ND profile.

That’s why I create an ND profile with SLAAC with DNS Through DHCP. This has the advantage that SLAAC and RA can generally be used, but DHCPv6 stateless is also used to explicitly assign DNS. The profiles can be found under Networking -> Networking Profiles -> Select profile type -> ND Profile. In the new profile, I only change the mode; otherwise, all settings remain at default. Under Networking -> Tier-1 Gateways -> Additional Settings -> ND Profile, I then select my new profile for my IPv6 T1 router.

All that’s missing is our DAD…

Why did the boy get fired from his keyboard factory job? - Because he was not doing enough shifts.

Again? Really? OK, let’s keep it short: the default DAD profile from NSX is sufficient for my purposes and can be assigned to both T1 and T0 in virtually the same way as the ND profile. And now I’m never going to talk about DAD ag…Why don’t programmers like nature? - Too many bugs.

Okay, continuing in context, I want to finish the article, even though I’ve just spent 30 minutes scrolling through Reddit and reading dad jokes.

Then I create an overlay segment connected to my IPv6 T1 and configure the following IPv6 gateway CIDR: fd11:22:33:ff10::1/64. I also create a segment DHCP server in which I specify the ULA IPs of my DNS servers in the lab and, of course, a DHCP server IP address from the segment. In my case, I simply take the next free IP fd11:22:33:ff10::2/64.

Ipv6 Nsx DHCP Config

Ipv6 Nsx DHCP Profile

Next, I create a VM from my Alpine template and connect it to the new segment, power up the VM, and am disappointed.

Are we there yet?

In order for my Alpine VM to get an IPv6 address at all, I have to edit the file /etc/network/interfaces and add iface eth0 inet6 auto.

auto lo
iface lo inet loopback

auto eth0
#iface eth0 inet dhcp
iface eth0 inet6 auto

One thing is already working after reboot: my VM is getting a ULA IP address via SLAAC and can also ping my DNS server.

inet6 fd11:22:33:ff10:2b17:367c:2ddc:2925/64 scope global dynamic noprefixroute flags 100 
  valid_lft 2591977sec preferred_lft 604777sec
inet6 fe80::250:56ff:fe86:9e5/64 scope link 
  valid_lft forever preferred_lft forever

Ping test:

template:~# ping6 fd::101
PING fd::101 (fd::101): 56 data bytes
64 bytes from fd::101: seq=0 ttl=60 time=1.157 ms
64 bytes from fd::101: seq=1 ttl=60 time=1.190 ms
64 bytes from fd::101: seq=2 ttl=60 time=1.471 ms

However, name resolution still does not work or rather, DHCPv6 is ignored. For this to work with Alpine, a small package needs to be installed. I should have mentioned this earlier, but then the suspense would have been ruined. For this to work, Alpine needs dhcpcd. I will incorporate this into my template in the future. Installation is super easy if you have internet access.

apk update
apk add dhcpcd
rc-update add dhcpcd

Then restart the network service or simply restart the entire VM, which is faster for me—but then again, I’m no Linux expert. Wow, I finally have name resolution, but only if you have an IPv6-capable DNS, of course. In my environment, that’s two Adguard Home instances. But I could also just use my OPNSense. Does the internet work now? Of course not. I get nice IPv6 name resolution and the routing works too, and yes, it’s not because of firewall rules, no, NAT66 is missing.

Because the entire setup is based on ULA IPs due to the dynamic Telekom prefix, and these are not routable on the internet, we now have to resort to NAT again. I would have liked to use NAT NPTv6, but that is not possible with my setup. I can’t track the PPPoE interface, which means that every time I am assigned a new prefix by Telekom, I would have to manually adjust NAT NPTv6 like a caveman. So in the end, it will be stateful NAT from an IPv6 ULA address to an IPv6 GUA IP. The whole thing is comparable to normal SNAT with IPv4. NSX supports NAT66, but since NSX cannot be a prefix delegation client, that doesn’t help me in my situation.

After configuring OPNSense Outbound NAT for IPv6 and the source IP range fd:11:22:33::/56, my internet is now working – what a struggle!

template:~# ping6 google.com
PING google.com (2a00:1450:4001:804::200e): 56 data bytes
64 bytes from 2a00:1450:4001:804::200e: seq=0 ttl=115 time=7.439 ms
64 bytes from 2a00:1450:4001:804::200e: seq=1 ttl=115 time=7.570 ms
64 bytes from 2a00:1450:4001:804::200e: seq=2 ttl=115 time=7.546 ms

VCF9 and IPv6

Does VCF9 support IPv6? The answer is a clear yes and no. The situation is complicated: individual components support dual stack or even native IPv6, but important core components either do not support IPv6 at all or the automatic provisioning does not support IPv6. In the official documentation for VCF9, there are exactly 143 search hits for IPv6, which pretty much sums up the situation. VKS Supervisor, for example, is one of the components that does not support IPv6, and this is what determines the success or failure of my automation of modern workloads in VCF9. My recommendation would therefore be to continue to rely on IPv4 and use IPv6 as a dual stack for VM workloads at most.

Conclusion

Was it stressful? That’s a difficult question. On the one hand, I rediscovered a lot of forgotten knowledge about IPv6, and on the other hand, playing around with routing was really fun. It’s a disappointment that VCF doesn’t yet fully support IPv6. The same applies to VPCs. Since this will be the future standard, it’s a bit strange that IPv6 isn’t supported. You can assign IPv6 in the VPC connection profile, but it doesn’t work in the VPC.

It’s also a pity that I can only test it to a very limited extent. I definitely want to do more tests, including with IPv6 DHCP, and see if I can somehow use my dynamic prefix and assign normal GUA IPs. I also want to try a few things with AVI. As a backup plan, I also have a public VPS server with fixed IPv6 IPs, which I currently use as a Wireguard rendezvous server and as an internet breakout in case Telekom decides to disadvantage Cloudflare services again, but that’s a whole other story.

It should be noted that even after 25 years, IPv6 is still not standard and IPv4 remains very important in data centers. Furthermore, it is somewhat disappointing that something as simple as DNS over RA is not yet supported by every operating system and every end device. I hope you enjoyed the article and feel encouraged to play around with IPv6 yourself.

In November 2025, I was a guest at Nutanix Next in Darmstadt, where I met up with old colleagues and wanted to take a look at what Nutanix had to offer. Visitors were kindly given the opportunity to obtain certification on the spot, which I naturally took advantage of.

Having worked with VMware for over 20 years, I was naturally skeptical about what Nutanix could do, and what can I say, the differences are perhaps less than I thought. You could also say same, same but different—at least when it comes to the core of virtualization. Nutanix is based on KVM, which is rock solid and which I have been running in my home lab for ages in the form of Unraid. When it comes to storage, Nutanix takes a different approach and primarily relies on HCI and its own storage solution.

In any case, NEXT was decisive enough for me to take another closer look at Nutanix, because after all, there is a Community Edition that should cover most of the features. I would have liked to test the “real” Nutanix version, but unfortunately that’s not easily possible. However, I am in talks with one or two Nutanix employees and perhaps a way can be found. That’s why this isn’t a feature comparison, because it would be like comparing apples and oranges if I were to compare the Community Edition of Nutanix with VCF9.

After all, there is a Community Edition of Nutanix, but VMware currently only offers ESXi8 Free. Personally, I would welcome a Community Edition from VMware. I know you can get licenses for VCF through VMUG Advantage and by passing an exam, but it’s not the same. Phew, that was quite a long introduction, but let’s get started.

Getting started

At first, it wasn’t that easy to find the CE Edition. You need a free Nutanix account, and then you can download the CE here.

Of course, the CE version has a few limitations, and of course I had to implement a few workarounds with my hardware to ensure that everything runs smoothly. But we are already used to that here, and it should come as no surprise to readers.

One limitation, for example, is the cluster size; Nutanix CE can only be deployed as a one, three, or four node cluster. Larger clusters or even two node clusters are reserved for the full version. The same applies to stretched clusters. In addition, the CE version only comes with AHV, which I personally don’t find problematic; if I want to use ESX, I deploy VCF or VVF. I first tried Nutanix as a nested deployment, which worked fine, but the storage was really slow. That’s not Nutanix’s fault, though; it’s the same with a nested vSAN cluster.

If you want to install the CE Edition bare metal, you unfortunately have to use a specific version of Rufus, otherwise Nutanix will not boot from a USB stick. I don’t know exactly why this is the case. I tried Mac alternatives because I no longer have a Windows PC, but it didn’t work. In the end, I got the image to run in a Windows 11 VM with USB passthrough and Rufus 3.21. Rufus 3.2.2 or newer do not work.

Another limitation of the CE Edition is that you must be online and have a Nutanix account. This must be entered after creating the cluster. Nutanix CE cannot be used without free online activation.

Hardware

To install Nutanix, you need at least three hard drives: one for booting, one for the CVM (hot tier must be flash), and one for cold data. I used NVMe storage for all three drives. I used three MS-01s with Intel i9 13th generation CPUs, 96 GB RAM, and 1x1TB and 2x2TB storage. Before the question arises, unfortunately I cannot use all 20 cores of the i9. Nutanix is the same as ESX in this respect; only the P+E cores without HT work. That makes a total of 14 vCPUs. I use 2x10Gb/s for the network, but less is also possible. However, you should not expect miracles in terms of performance if the network connection is less than 10Gb/s.

Network

While we’re on the subject of networks, Nutanix uses the 192.168.5.0/24 network internally for communication between the hypervisor and CVM. I haven’t found a way to change this (yet). If you have important external services such as DNS or NTP in this network area, it becomes difficult. Furthermore, the CVM and the hypervisor host must be in the same L2 network, and unfortunately, the CE Edition does not support VLANs for this. I managed to get my AHV host and my CVM to have a management VLAN, but the VLAN of the CVM is not persistent. After each reboot, the VLAN tag disappeared again. The problem seems to be that the VLAN tag settings are not written to cvm_config.json. I tried to do this manually, but I don’t seem to be using the correct syntax. I need to test this further, and if I figure out how to do it, I will update this article. Another thing that may seem unusual to VMware users is that Nutanix initially uses all network adapters after installation, and this cannot be configured during installation.

Installation

The installation is so simple and straightforward that I briefly considered not writing anything at all. But that’s mainly because there’s virtually nothing to configure.

What you see in this screenshot is exactly what you can configure in the CE version. I use my smallest disk as the boot disk (H) and both 2TB NVMes as data or CVM disks. The installer usually selects the appropriate option. If you have different disk sizes (all flash), I would select the largest one as the data disk. Ideally, the cluster should be uniformly equipped, but this is not a must.

CVM - What is that anyway?

I should perhaps add a brief comment on this. One of my biggest criticisms was the complicated structure—why is there such a thing as a CVM, and why can’t it be like vSAN? I had a very good conversation about this topic with Bas Raayman at Next and if anyone can explain it, it’s him.

I hope I can summarize everything briefly. But on the one hand, the issue is historical. Nutanix started with ESXi as its hypervisor, and it was simply not allowed to bring the functionalities into the kernel via its own extensions. On the other hand, it also makes the platform hypervisor agnostic. Among other things, the CVM serves as a storage controller, provides IO paths, contains the cluster logic, and can provide a file server and certainly much more. The big advantage is that it is flexible. Nutanix runs on AHV, VMware, and the cloud. The CVM code base is always the same. Furthermore, VM metadata is stored in the CVM. This means I can easily migrate VMs between hypervisors. But where there is light, there is also shadow. Of course, this comes at the cost of some latency. According to Bas, it’s hardly noticeable, but if I need to squeeze out every last bit of latency, then ESX is probably faster. The second problem, which I consider much more significant, is that users can log in to the CVM at any time and, in the case of the CE Edition, must do so. With a lot of root privileges comes a lot of responsibility. If you shut down the CVM or break it, the storage node will also fail. Overall, however, the advantages outweigh the disadvantages if you want to be hypervisor agnostic. And I must admit that at first I didn’t understand the necessity of CVM and found the whole thing unnecessarily complicated. However, my opinion has actually changed in the meantime and I can now appreciate the charm of this solution.

Regardless, I will continue to favor NFS/iSCSI as the primary storage in my home lab wherever possible. Maybe I’m just a little stubborn and old-fashioned in that regard.

Creating the cluster

After installing all three hosts, I was faced with the problem that my servers were not accessible. Since it is not officially possible to configure a VLAN for management, I configured my switch ports to untagged. However, the reason why the servers were not accessible was not because of this, but because my first adapter on the MS-01 is also used for the integrated IPMI. By default, Nutanix builds an active/standby bond with all adapters and prefers ETH0 as the active adapter. Even if it only has 1 Gb/s. Since my IPMI also does not support VLAN tags and is located in a different LAN than my Nutanix cluster, I had a problem.

The solution is obvious: reconfigure the open vswitch. But how?

Changing the OVS Switch uplinks

Now this is where the CVM comes into play, but how can you access the CVM if you can’t reach it via the network? Thanks to IPMI, I have console access to my server, and I mentioned the 192.168.5.0/24 network earlier. Each CVM has the same internal IP, so you can jump to the CVM via SSH to 192.168.5.2. Each AHV host has 192.168.5.1. Each CVM has two interfaces. The external interface is assigned to the OVS on the default bridge br0, and the internal interface runs via a Linux bridge virbr0. To log in to the AHV host, you need the root user, and the CVM must be addressed with the user nutanix. Both users have the default password nutanix/4u.

After logging in, you can change the uplinks of the default br0 using the following command. In my case, I only want to use the two 10 Gb/s interfaces of my MS-01.

manage_ovs --bridge_name br0 --interfaces eth2,eth3 --bond_mode active-backup update_uplinks

After this change, the host and CVM are now accessible from my admin desktop. This must now also be done with the other two hosts.

Workaround due to identical UUIDs of the hosts

This brings me to the first real problem with the installation, and I noticed the error very late in the process. Unfortunately, all my MS-01 servers are assigned the same UUID. By default, the default source for the UUID is smbios, but all of my MS-01s have the same UUID. This should not happen and is clearly due to my hardware. I cannot say at this point whether this also happens with an MS-A2, but at least all 6 of my MS01s are affected.

The annoying thing is that I only notice this when a VM needs to be migrated. I can create a cluster without any problems, execute the lifecycle, and even Prism Central deployment works fine, but as soon as a VM needs to be migrated, the problems start.

I received the following error message.

Operation failed: internal error: Attempt to migrate guest to the same host 03000200-0400-0500-0006-000700080009: 61

And if you know a little bit about this, you’ll notice that the reason is already stated in the error message. All hosts have the UUID 03000200-0400-0500-0006-000700080009. As a result, Nutanix assumes that the VM should be migrated to the same host where it is running and therefore aborts the process.

If you are logged in as root on the AHV, you can use the following command to display the UUID.

cat /sys/class/dmi/id/product_uuid

If this is the same on all hosts, then congratulations, the following workaround is necessary. First, we generate fresh UUIDs that are unique. This can be done with the following command on the AHV. Three can be generated at once.

uuidgen -r

It’s best to save them, because unfortunately the fix isn’t 100% permanent, but more on that later. Next, the following file must be modified. /etc/libvirt/libvirtd.conf This can be done with nano or vi. Since I am more comfortable with nano, it is always my tool of choice, if available. The fix is quite simple: a line needs to be commented out relatively far down, and the UUID needs to be replaced with a valid one.

# NB This default all-zeros UUID will not work. Replace
# it with the output of the 'uuidgen' command and then
# uncomment this entry
host_uuid = "c39808fa-8255-4c46-84ee-84e3ad5d16f0"
#host_uuid_source = "smbios"

Finally, restart the libvirt service and we are ready to go.

systemctl restart libvirtd

Creating the cluster - now really!

Once network connectivity has been established and our UUIDs are once again unique, the cluster can be created. Since there is no GUI available for us to use at this point, this must also be done manually via the CLI. Don’t worry, it’s quick and painless. To do this, you must log in to any CVM via nutanix User. This should now also be possible via the configured IP from the installer.

The cluster creation process is simple and can be done quickly using the following CLI command.

cluster -s cvm1_ip_addr,cvm2_ip_addr,... create

After a certain amount of time, the cluster status can be checked with this command.

cluster status

Additional settings via the CLI.

Define the cluster name.

ncli cluster edit-params new-name=cluster_name

Configure a name server for the cluster.

ncli cluster add-to-name-servers servers=public_name_server_ip_address

Configure an external IP address for the cluster.

(This parameter is required for a CE cluster.)

ncli cluster set-external-ip-address external-ip-address=cluster_ip_address

Add an NTP server IP address to the list of NTP servers.

ncli cluster add-to-ntp-servers servers=NTP_server_ip_address

After completing all these steps, you should now be able to log in to the Nutanix cluster via the ClusterIP.

With Prism Elements, Nutanix provides a web interface that can be accessed via any host and via the cluster IP port 9440. This means that it is not necessarily required to deploy Prism Central. If you intend to use Nutanix as a pure hypervisor without Nutanix Flow and other services, Prism Elements is entirely sufficient. If you want to use different clusters or Nutanix Flow (which I am very interested in), then Prism Central is a must. It can be easily deployed as a VM in the cluster via Prism Element. I will describe this further down in the article.

When logging in for the first time at the Prism Elemet GUI with the admin user and the password nutanix/4u, this must be changed. And while we’re on the subject of changing passwords, this should also be done for all other accounts. To change the passwords for all accounts, there is a KB article from Nutanix that describes how to do this via the CVM. I strongly recommend changing all default accounts.

Default Paasword changes

The following three commands help to change the password for the root, admin, and nutanix accounts at the AHV host level and on all the hosts in the cluster. Do not modify the command. It will ask for the new password twice and will not display it. This can run from any CVM in the cluster.

Change CVM local account

nutanix

sudo passwd nutanix

Change AHV local accounts

root

echo -e "CHANGING ALL AHV HOST ROOT PASSWORDS.\nPlease input new password: "; read -rs password1; echo "Confirm new password: "; read -rs password2; if [ "$password1" == "$password2" ]; then for host in $(hostips); do echo Host $host; echo $password1 | ssh root@$host "passwd --stdin root"; done; else echo "The passwords do not match"; fi

admin

echo -e "CHANGING ALL AHV HOST ADMIN PASSWORDS.\nPlease input new password: "; read -rs password1; echo "Confirm new password: "; read -rs password2; if [ "$password1" == "$password2" ]; then for host in $(hostips); do echo Host $host; echo $password1 | ssh root@$host "passwd --stdin admin"; done; else echo "The passwords do not match"; fi

nutanix

echo -e "CHANGING ALL AHV HOST NUTANIX PASSWORDS.\nPlease input new password: "; read -rs password1; echo "Confirm new password: "; read -rs password2; if [ "$password1" == "$password2" ]; then for host in $(hostips); do echo Host $host; echo $password1 | ssh root@$host "passwd --stdin nutanix"; done; else echo "The passwords do not match"; fi

Additional settings

I switched my default network switch vs0 to jumbo frames. This can be done via the dropdown menu (in Prism Element) under Settings -> Network Configuration -> Virtual Switch -> vs0 and the Edit icon. Here, the bond and uplinks can also be adjusted.

Nutanix offers two options: Standard or Quick. In Standard mode, the hosts are put into maintenance mode beforehand; in Quick mode, the changes are implemented immediately. If something is wrong with the physical configuration, the workload would be offline as a result. Nutanix recommends Standard mode here, as adjusting the network also affects storage traffic. This always runs via the vs0 switch. If you have more than two adapters, you can also separate the traffic via an additional switch (alternatively, you can configure a bond without failover and separate traffic with two network types).

In addition, under Settings -> SSL Certificate, I replaced the Prism Element certificate with my own CA from my root CA. Here, all IPs and FQDNs of all cluster nodes and the cluster VIP must be entered under SAN. I also switched the UI to dark mode and English. This can be done under Settings -> UI Settings and Language Settings.

Lifecycle

The lifecycle can be managed via Prism Element or Prism Central. Prism Central has its own lifecycle independent of Prism Element. Nutanix CE comes in AOS version 6.9.1. Currently, the maximum upgrade available via Lifecycle is to AOS version 7.3.1.2.

Deploying Nutanix Prism

To deploy Prism Central, I first created a new storage container. A storage container is a logical construct to which I can apply policies and where I can create my vDisks for VMs. It is not a volume or LUN. Each storage container can have different policies and therefore behave differently, but all storage containers run on the same storage pool (in my lab, I only have one storage pool consisting of the resources of the three servers).

To create the storage container, I switch to Storage via the dropdown menu and create a new container called Workload. Compression and an error tolerance of 1N/1D are set by default. This means that 1 node from the cluster can fail. I could also set the advertised capacity or reserve capacity. I leave everything at default here. Unfortunately, erasure coding does not work because I would need at least 4 hosts for that. It’s a sad, because vSAN ESA is a little more flexible here, allowing me to use Raid 5 erasure coding with as few as 3 hosts.

To deploy Prism Central, simply click on the Nutanx Prism Element Dashboard (top left) and select “Deploy New Prism Central.” The dialog box will display all compatible versions, which you can then download. I opted for the latest version. I deployed Prism Central in the small form factor because I want to use Nutanix Flow later, and that doesn’t work with extra-small. Using Create Network, I created a new VLAN network on the default vSwitch vs0. This VLAN must be routed and must be able to communicate with the CVMs and the AHV server. It must also be present on the physical switches. The other settings are very self-explanatory. IP, gateway IP, DNS, NTP, and storage container should be known and available. I decided not to use an HA deployment due to the resources required. Finally, press Deploy and then it’s time for 1-2 coffees, because the installation can take a good 30-40 minutes.

Once Prism Central has been successfully deployed, the cluster still needs to be registered. This is done via Prism Element in the same place where the deployment was started. I will describe the rest of the Prism Central setup and the post-setup steps in a separate article—sometime in the future.

Safely shut down the environment and restart it.

Since I do not have my lab environments running continuously, I shut down my environments regularly, and as with VCF, this must be done in a fixed order.

• Shutting down all workload VMs via Prism Central or Elements.
• Log in to Prism Central via SSH with the nutanix user and enter cluster stop.
• Wait until the message “Success” appears. You can check the status using cluster status. Even if Prism Central is not a clustered installation, this is the safest option.
• Shut down Prism Central with sudo shutdown now.
• Log in to Prism Element and wait until Prism Central VM has shut down.
• Log in via SSH and the nutanix user on any CMV and execute cluster stop.
• Wait until the message “Success” appears. You can check the status using cluster status.
• Log in to all CVMs via SSH and nutanix user and shut them down with sudo shutdown now
• Log in to all AHV servers via SSH and root
• Check with virsh list whether any VMs are still running
• Once all VMs have been stopped, shut down each AHV host with shutdown now

Starting the cluster is done in exactly the opposite order. Instead of cluster stop, cluster start must be entered. Simply booting up the Prism Central VM is not enough; here too, the cluster must be started explicitly with cluster start.

Summary

At first glance, a lot of things are different from VMware. But what I particularly like, and haven’t mentioned in this article yet, is that as a home lab user, you get a really well-rounded and great software package for free. The CE also includes the Kubernetes Engine and Flow, which is essentially Nutanix’s alternative to NSX and VKS. Overall, I really liked how easy it was to install Flow or Move (the HCX alternative) via a kind of app store.

I have to admit that the installation process wasn’t exactly easy. In this article, I’ve only written down the results of several days of trying and searching for the problem. Nutanix can’t do anything about the UUID problem, but I find it difficult to understand why the AHV installation process doesn’t simply support VLANs like ESX does. However, I also know that these are problems with the CE Edition and do not affect Nutanix Fusion.

But it’s not like you’re faced with unsolvable problems. The Nutanix forum is full of solutions and help, you just have to find it. It also doesn’t hurt to be familiar with KVM and Open vSwitch. I have experience with both thanks to my work with Proxmox and Unraid.

The fact that you need three hard drives could also be an obstacle. I wouldn’t have been able to get Nutanix to run on a Nuc. Maybe it’s possible to boot from a USB stick, I’ll have to test that. Fortunately, nested deployment is also an option if you can live with the lower performance of the storage.

In the future, I will write more articles about my small test cluster because I definitely want to try Move and Flow. I hope that anyone who was hoping I would write a scathing review isn’t too disappointed, but to be fair, I can only compare free software with free software, and VMware would currently come up short. I’m curious to see how this topic develops and what else I will write about it.

Today, I would like to address the topic of how Keycloak can be used as a central identity management system with the Advanced Load Balancer. In theory, this is relatively easy to implement, but in practice, there are a few challenges. I would like to describe exactly what these are and also present a solution. As a little bonus, I will also show you how to use a Fido Key (Yubikey) to log in.

I must admit, however, that I haven’t had such a frustrating setup in a long time, and it involved a lot of trial and error. Now, OIDC isn’t exactly one of my favorite topics, and I only had rudimentary experience with Keycloak, but that shouldn’t stop me from trying something new. Right? Right! So let’s get started.

Setup Keycloak

At this point, I made it relatively easy for myself by installing the official Keycloak Docker version 26.5.2 on my Unraid. Keycloak requires a Postgres DB, so I used an existing one—also on Unraid. If you were hoping that I would write instructions on how to install Keycloak here, I’m afraid I have to disappoint you.

Since I want to use OKTA as a provider in my VCF setup later on, my Keycloak needs a valid certificate that can be publicly validated. Here, too, I have made it easy for myself and am using my NGINX proxy manager with Let’s Encrypt support and a certificate from one of my domains. So that I don’t have to store my private IPs in Cloudflare’s DNS, I use SplitDNS at home. This means I can resolve keycloak.evilcorp.info, but you cannot. Since most of my Docker containers only run internally with HTTP, I have been using the proxy method for a long time and it has proven to be robust and reliable for me.

Realm Setup

Now that you have somehow deployed Keycloak on your system, we can start with the actual configuration. To do this, you first need to create a new realm. A realm is a client, you could say. A realm has its own users, groups, and so on. In addition, you can define your own identity provider for each realm. In my case, for the sake of simplicity, I use Microsoft AD (because it’s there anyway), but I could also use LinkedIn, Facebook, GitHub, Google, and many other identity providers. A combination of several is also possible. The only important thing is that the username must be unique within the realm. No two identical usernames can exist.

A new realm can be easily created under Manage realms and only requires a name. I named my realm lab-ad.

LDAP Connection

I deliberately avoided using LDAPs in my lab because my container does not have persistent storage, which means I cannot easily link the RootCA of my domain. This is on my long to-do list, but you can build it properly. An AD is integrated under User federation and not under Identity providers. It sounds strange, but that’s how it is.

Here are my settings at a glance:

General options

UI display name ldap
Vendor Active Directory

Connection and authentication settings

Connection URL ldap://home.lab
Enable StartTLS Off
Use Truststore SPI Never
Connection pooling On
Connection timeout 
Bind type simple
Bind DN CN=keycloak svc,OU=ServiceAccounts,DC=home,DC=lab
Bind credentials ••••••••••

LDAP searching and updating

Edit mode READ_ONLY
Users DN OU=User,DC=home,DC=lab
Relative user creation DN 
Username LDAP attribute sAMAccountName
RDN LDAP attribute cn
UUID LDAP attribute objectGUID
User object classes user
User LDAP filter (&(objectClass=user)(!(objectClass=computer)))
Search scope One Level
Read timeout 
Pagination On
Referral 

Synchronization settings

Import users On
Sync Registrations On
Batch size 
Periodic full sync Off
Periodic changed users sync On
Changed users sync period 60
Remove invalid users during searches On

Kerberos integration

Allow Kerberos authentication Off
Use Kerberos for password authentication Off

Cache settings

Cache policy DEFAULT

Advanced settings

Enable the LDAPv3 password modify extended operation Off
Validate password policy Off
Trust Email Off
Connection trace Off

This configuration connects Keycloak to an AD server at ldap://home.lab and authenticates there with a service account (CN=keycloak svc,OU=ServiceAccounts), allowing connection and read-only access. Only users in OU=User,DC=home,DC=lab are used, sAMAccountNameis used as the login name, computer objects are filtered out, and the users found are imported into the local database, with changes synchronized every 60 seconds and deleted AD users automatically removed.

Next, a group-ldap-mapper must be created in the Mappers tab. To do this, simply click on create new mapper and then select the type group-ldap-mapper.

My mapper is called ad-groups and these are the settings: ad-groups

LDAP Groups DN OU=SecGroups,DC=home,DC=lab
Relative creation DN 
Group Name LDAP Attribute cn
Group Object Classes group
Preserve Group Inheritance Off
Ignore Missing Groups Off
Membership LDAP Attribute member
Membership Attribute Type DN
Membership User LDAP Attribute sAMAccountName
LDAP Filter 
Mode READ_ONLY
User Groups Retrieve Strategy LOAD_GROUPS_BY_MEMBER_ATTRIBUTE
Member-Of LDAP Attribute memberOf
Mapped Group Attributes 
Drop non-existing groups during sync Off
Groups Path /

These settings tell Keycloak where and how LDAP groups are read: It searches for groups under OU=SecGroups,DC=home,DC=lab, recognizes them as group objects with names from cn, and reads membership via the member attribute (DN-based, users via sAMAccountName). Keycloak works read-only, loads groups via the member attribute, does not automatically transfer missing/unmapped groups, and stores the imported groups under / in the Keycloak group tree.

Once LDAP is connected, users should now be found when searching for a user under Users.

Next, I create an Auth Profile in AVI. AVI is a bit special in this regard because the Keycloak client must be created with a very specific name.

Auth Profile AVI

In AVI, navigate to Templates -> Security -> Auth Profile and create a new SAML profile.

Two things are important here: first, that the IDP metadata URL is used, and second, that the service provider is switched to FQDN. The IDP metadata URL can be found in Keycloak under Realm settings -> General -> Endpoints SAML 2.0 Identity Provider Metadata. Simply click on the browser link and copy the URL from the browser into AVI.

Once the profile has been saved, click on the context menu with the three dots in the Auth Profile overview and select Verify. The Verify view displays the Entity ID and the SSO URL—very intuitive—not.

Entity ID: AviController-vcf09-avi.lab.vcf
SSO URL: https://vcf09-avi.lab.vcf/sso/acs/

Keycloak Client

Keycloak Clients are applications and services that can request authentication of a user. You can have multiple clients in one realm, and clients can have different settings. Such as client-specific roles, for example. I’ll come back to that later, because AVI has a few more peculiarities. A client is created in the realm under Clients.

Create client

Client type SAML
Client ID AviController-vcf09-avi.lab.vcf
Name AVI - Test
Description 
Always display in UI Off
Root URL 
Home URL 
Valid redirect URIs https://vcf09-avi.lab.vcf/sso/acs/
Valid post logout redirect URIs https://vcf09-avi.lab.vcf/sso/acs/
IDP-Initiated SSO URL name 
IDP Initiated SSO Relay State 
Master SAML Processing URL 

Those are the basic settings. Next, a few more settings need to be adjusted. To do this, open the client settings—the range of options is overwhelming at first, but fortunately, not much needs to be changed.

The first setting must be made in Signature and Encryption. Here, Sign assertions must be set to On. In addition, the name ID format must be changed to email (every user must have an email address in the AD).

A theme can also be selected under Loging settings. Next, in the Tab Keys, I had to disable the client signature. The problem only occurred when I changed the AVI SSL/TLS Profile and no longer allowed weak ciphers. I haven’t yet figured out exactly where the problem lies. I imported the certificate as described in the AVI documentation, but it only works for me if I allow weak ciphers in AVI. If anyone has any ideas, I would be grateful for any input.

Under Roles, I create a new client role called avi-access. I’ll explain what this does a little later. Under Client Scopes, you have to create a Group Mapper, otherwise AVI won’t know anything about the groups. To do this, it’s important to go to the Dedicated scopes, which are named the same as the client, only with -dedicated.

Here, a new mapper must be configured. To do this, press the Configure new mapper button and select a mapper of type Group list.

Group attribute name groups
Friendly Name AD Groups
SAML Attribute NameFormat Basic
Single Group Attribute On
Full group path Off

The Group Mapper in the “Client Dedicated” area ensures that when logging in, the user’s groups are written to the token/SAML assertion for that specific client.

AVI will need the attribute later for the mapping profile. In the mapping profile, the attribute must have the same name as it was created in Keycloak—in my case, that is “groups”.

The SSO URL must be entered under Advanced.

Fine Grain SAML Endpoint Configuration

This section to configure exact URLs for Assertion Consumer and Single Logout Service.

Logo URL 
Policy URL 
Terms of service URL 
Assertion Consumer Service POST Binding URL https://vcf09-avi.lab.vcf/sso/acs/
Assertion Consumer Service Redirect Binding URL https://vcf09-avi.lab.vcf/sso/acs/
Logout Service POST Binding URL 
Logout Service Redirect Binding URL 
Logout Service SOAP Binding URL 
Logout Service ARTIFACT Binding URL 
Artifact Binding URL 
Artifact Resolution Service 

This completes the initial configuration of the client. However, do not be concerned, as we will need to configure more later on.

Role mapping for AVI

Next, we use the aforementioned client role. This role can only be used in the AVI client in Keycloak. With this, we will prevent a peculiarity of AVI. AVI has a strange habit of creating a user in AVI as soon as authentication is complete, regardless of whether the user has access or not. Currently, all I have to do is create a valid Keycloak session for a user, who is then created in AVI and receives an error message because they are not authorized. The only problem is that this creates a defective SAML cookie and you can no longer log in to AVI, even if you try with a valid user. Great, I can already see the tickets coming in. The solution is to delete the cookie in the browser and end the session in Keycloak. After that, you can log in again. Of course, you now have a user without authorization in AVI, which you must delete manually.

In general, this behavior is not a security issue, but user complaints are inevitable. To prevent this, I am building in a check to see whether the user is authorized to access the AVI. The check is performed at the Keycloak level, and the request is not sent to the AVI at all if there is no authorization for AVI.

To do this, a role mapping must be created in Keycloak. Under Groups -> vcf-avi -> Role Mapping, a role can be assigned to the group in Keycloak. The groups and user membership come from the AD, making it easy to manage centrally. To assign the role, click Assign role and select the appropriate client role avi-access via client roles.

Auth Mapping Profile for AVI

I promise, we’ll soon be able to log in with Keycloak. An Auth Mapping Profile is still required in AVI. This can be created under Templates -> Security -> Auth Mapping Profile. It must be a SAML type profile. The profile is given a descriptive name and requires a rule. I’ll keep it simple in this example and say that all users whose SAML attribute contains vcf-avi become super users. Remember, the SAML attribute contains all Keycloak groups. You could also write a RegEX, because with Contains I have the problem that a group named vcf-avi-test would also match. But I’ll leave that to the people who have to implement it in production.

Finally, the Auth profile and the Mapping Profile just need to be assigned.

AVI Authentication settings

You can do this easily under Administration -> System Settings and then edit them. Under Authentication, you must switch from Local to Remote. The Enable Local User Login checkbox should be selected, otherwise you will not be able to log in locally if SSO fails. Finally, map the Auth Profile with the Mapping Profile, and then we’re done and can run our first test.

First Test

If everything works correctly, after accessing the AVI page, a message should automatically appear stating that you are not logged in, and the Keycloak login screen should appear directly via the login link.

You can log in with either your username or your email address. Because I was smart enough to use my private email address, you can only see the username here.

Interim conclusion

SSO login via Keycloak should work, but unfortunately it is not yet possible to log in with a FIDO key, and there is still no way to prevent the dummy user from logging in to AVI, or rather, from being rejected by AVI, which breaks our SAML cookie. Time to change that. So grab a coffee, because this article could go on for quite a while.

Enable Fido Key

To use a Yubikey or similar device, a few options need to be set in Keycloak. Passwordless login is also supported, but I’m not personally a fan of this. However, the activation process is similar. First, in the Realm Settings under Authentication -> Required actions -> Webauth Register, set Enabled and Set as default action. For passwordless login, the corresponding option Webauthn Register Passwordless must be activated and set as the default action. Without the default action, users will not be asked to create a passkey when they log in for the first time.

Then, under Authentication -> Policies -> WebAuthn Policy or WebAuthn Passwordless Policy, the policy must be activated. To do this, WebAuthn -> Avoid same authenticator registration should be activated, or, in the case of WebAuthn Passwordless Policy, this option and enable passkeys in general.

Flow

Finally, a flow must be created under Authentication -> Flows. I use the built-in browser-based authentication flow as a basis and create a copy via the context menu. In order to use the client role in the flow, the flow must be assigned to the client. To do this, the flow you just created must be assigned under Clients -> AviController-xxx -> Advanced -> Authentication flow overrides. This allows each client to have its own flow and different checks or login methods.

A flow consists of three key elements. There are steps (executions), which are specific actions in the login process, such as deny access. Secondly, there are conditions, which are ways of checking something, such as whether the user has a certain role, and finally there are sub-flows, which can be used to group steps and conditions. In addition, each element has a requirement that determines whether the login continues or is terminated. In short, a flow is a chain of steps, conditions, and sub-flows with rules that control the login process.

• 1. To activate keypasses, WebAuthn must be configured in the flow and set to required.
• 2. Next, a subflow of type generel must be created. This can be done using the small plus sign. It is important that the subflow is in the correct position at the end and is indented correctly. The subflow is used to check role membership.
• 3. The subflow must be dragged and dropped into the correct position. It is sometimes easier to do this by dragging other elements upwards. The subflow must also be set to Conditional. The conditional block is only executed if the previous step is TRUE. In this flow, 2FA must have been successfully completed, otherwise the login will be canceled.
• 4. Next, a condition and a step must be created using the plus sign. Both must be defined as required, otherwise the check will not be performed and, in the event of a missing authorization, Access denied will not be executed.
• 5. The condition must be of type user role, and client roles must be selected under select role. Then I select the avi-access role and select Negate output. Negate Output ensures that the condition is set to TRUE if the user does not have the avi-access role. If the condition is TRUE, the step deny access is executed. If the user has the role, the condition is FALSE and the flow is terminated with the result that I have access.

If everything has been done correctly, the flow should look like the screenshot. “Indentation” is just as important here as it is in YAML.

Now that everything is configured, it should work roughly as shown in this video.

Troubleshooting

Troubleshooting can be very time-consuming, and I had a few problems. I could bet that I logged into AVI at least 300 times in the last two days. So here are a few general tips.

• Delete your cookies. Even if it is open in an anonymous tab, cookie remnants may remain during runtime, and in case of doubt, this is a broken SAML cookie that then prevents even valid logins.
• Always delete the session in Keycloak. Just because you log out of AVI does not mean that the SSO session is also closed.
• When dragging and dropping in Flow, elements may automatically deactivate if they are moved to an invalid position, for example. Keycloak attempts to save immediately after moving. This nearly drove me to distraction. Sometimes it makes more sense to move other elements than the element you actually want to move.
• Enables logging in Keycloak. This is configured in the Realm Settings under Events.
• If parts of the flow were not executed, the position of the sub-flow should be checked to see if it was nested correctly.
• If “access denied” is displayed despite correct role assignment, the negation in the condition was probably forgotten.

Summary

Keycloak or OIDC is a damn powerful toolkit. However, it is anything but simple and, unfortunately, extremely poorly documented for AVI load balancers. I actually wanted to write down the configuration for the VCF client, but the blog article is becoming so extensive that most people probably won’t read it anyway, so I’d rather pick up the whole thing again in a second article.

Ah Sh*t, Here We Go Again - Happy New Year! What was I thinking, letting you guys vote on what my next topic would be? But okay, I guess I only have myself to blame. Anyway, over 60% wanted me to cover VCF9 automation in All Apps mode. As most people know, there are two modes in VCF Automation: the so-called legacy mode, which is nothing more than VCF Automation 8.18, and the new and much more exciting All Apps mode.

The differences in brief: Legacy Mode follows a classic IaaS approach, focusing on the provision of individual VMs. All Apps Mode is application-centric. The aim is to deploy an application as a whole, with VMs, containers, network, and storage being merely consumable resources. VMs are consumed as a service in a “Kubernetes-native” manner. This is why VKS and NSX (with VPCs) are also mandatory components of the automation solution in VCF9.

Prerequisites

But okay, let’s get to the prerequisites for using automation: We need a VCF9 deployment. In my lab, I have deployed a management design (formerly consolidated design) with two vSphere clusters. The second cluster is not necessary, even though the current VCF 9 documentation claims otherwise.

NSX with a T0 router in active/standby mode is required (VPCs with stateful services such as auto snat still need this, as in VCF9.0.1 the transit gateway inherits the HA mode from the T0) and a VKS supervisor cluster.

Phew, that’s quite a lot. The article will not cover VCF9 deployment and NSX VPC setup. You can find the basics of VPCs in my blog here, here, and here.

But here I will show you VCF Automation Deployment, a basic VKS deployment (with VPCs), setting up an organization in Automation, and creating a VM in All Apps Mode.

VCF Automation Deployment

If automation was not provided via the VCF Installer, it can be deployed retrospectively as a Day 2 operation via VCF Operations.

To deploy automation in small form factor, we need 1 FQDN/IP that serves as VIP and 2 additional IP addresses that are used as cluster node IP pool. I use the internal load balancer in my setup.

The actual setup is fairly straightforward. The component is installed via Operations Fleet Manager under Lifecycle.

I won’t go through every single step here, just the most important ones. When creating the certificate, I always create a wildcard certificate in the lab. It is important that the VIP IP address and the FQDN are entered in the SANs. In my lab, the SAN entries would be *.lab.vcf, 10.28.13.12.

The network settings are also relatively self-explanatory. I deploy my automation to the VM MGMT Network, which was automatically created by VCF. I use the servers specified during the VCF installation as DNS and NTP servers. You don’t have to enter these again separately; the Edit button gives you a selection of all DNS and NTP servers known to VCF. This is somewhat counterintuitive.

Now comes the component part, where many (like me) are probably a little confused at first. I don’t find the description here very clear. Since the internal load balancer is used, the FQDN of the component and the cluster VIP must be identical.

The IP address created for the FQDN is entered under Primary VIP. The Internal Cluster CIDR is for internal communication and is 198.18.0.0/15 by default and should not overlap with existing networks. The range is from the reserved network and is not usually used in any network. The Cluster CIDR cannot be changed afterwards.

Additional VIPs are not required in the standard setup and can be left blank. The two additional IP addresses I mentioned at the beginning are entered in the Cluster Node IP Pool. These are also from the VM MGMT Lan. Each automation node is assigned an IP address from this pool. A spare IP is required for upgrades, so the minimum size of the pool is always the number of automation nodes + 1.

If everything has been done correctly, the precheck will run without any problems. The deployment takes some time. With my hardware, it takes just over an hour, but I have seen deployments that take up to 3 hours.

After one or two cups of VCF Admin fuel, aka coffee, the installation should now be successfully complete and we can focus on the VKS deployment.

VKS Deployment with VPCs

First of all, VKS is not my area of expertise, so I’m going to do the simplest VKS deployment possible here. I’m using the NSX load balancer instead of AVI, and I’m using VPCs. I also use the simple installation and not the HA cluster of the supervisor. If you want to read more about VKS, I recommend my colleague Christian, who does interesting things with Antrea, VPCs, and VKS.

To install VKS, you must first create a storage policy. To do this, you must first create a tag and a tag category. The quickest way to do this is via vCenter by clicking on Storage and selecting Assign Tag. Select Add Tag and then Create New Category to create a datastore tag category.

It is important to select One Tag and Datastore here. Once the category has been created, a tag can be created and assigned to the storage.

Then I create the appropriate storage policy under Policies and Profiles -> VM Storage Policies. This is necessary to ensure that the correct storage is selected during VKS deployment. The storage policy is TAG-based and uses the previously created TAG.

Next, we start the VKS deployment via the vCenter Burger Menu and the Supervisor Management menu item. During deployment, a content library is automatically created on the VKS Storage Policy data store. If you want, you can also do this in advance, but since it should be as simple as possible, I use the automatically generated content library.

In my opinion, deployment via VPCs is the simplest method, as it requires the fewest parameters to be set during deployment. A new VPC must be created in advance for the supervisor cluster and a public VPC subnet for management.

Next, the supervisor location is determined. Since this is a normal cluster deployment and not a vSphere with Zones setup, Cluster Deployment must be selected. In addition, a supervisor name is assigned (in lowercase letters) and, since this is to be a single node cluster, Control Plane HA remains disabled. As mentioned, I have two vSphere clusters in my setup and I want my supervisor to be in cluster m02-cl02.

After that, the storage policy is assigned and you can proceed to the management network setup. Normally, a pool of at least 4 IP addresses (1 VIP, 3 supervisor nodes) is sufficient. In a simple deployment, even fewer IP addresses are required, but I have specified a larger range to enable scale-out in the future. The remaining details are fairly self-explanatory.

The next setting is the workload network, and the nice thing about the VPC setup is that most of it is already pre-filled and was specified during the NSX setup of the VPCs. Since I want to keep this example simple, I am using the default project from NSX. It would of course also be possible to put the supervisor in its own NSX project with its own T0.

The private (VPC) CIDRs can be freely assigned, as it is a non-routable network, so overlapping IP ranges are possible and permissible. Nevertheless, you should be careful not to enter any of your routable networks as private CIDRs. The service CIDR is preselected and usually does not need to be adjusted, and again comes from a privately reserved range.

DNS and NTP can be customized for the workers here, but this is not necessary in my case. These are addressed from the VPC with Auto SNAT, as a worker node cannot communicate directly via the private VPC CIDRs.

Finally, the supervisor control plane size is determined. I am using Small (4 vCPUs and 16 GB RAM per supervisor control node) here.

The VKS setup takes quite a bit of time. I think it took me at least 1 to 2 coffees—so a good 20 minutes. Yes, I should probably reconsider my coffee consumption. And while we’re on the subject of consumption – what a transition – the LCI, or Local Consumption Interface, still needs to be installed.

The LCI can be found in the Broadcom Support Portal under the Free Software category. It is a simple YAML file. If you don’t want to search for it or deal with the Broadcom portal, you can copy version 9.0.2 (the current version at the time of publication) here.

apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  name: cci-ns.vmware.com.9.0.2+f943fb89
spec:
  refName: cci-ns.vmware.com
  version: 9.0.2+f943fb89
  template:
    spec:
      fetch:
      - imgpkgBundle:
          image: projects.packages.broadcom.com/vsphere/iaas/lci-service/9.0.2/lci-service:9.0.2-f943fb89
      template:
      - ytt:
          paths:
          - config/
          ignoreUnknownComments: true
      - kbld:
          paths:
          - '-'
          - .imgpkg/images.yml
      deploy:
      - kapp: {}
  valuesSchema:
    openAPIv3:
      type: object
      additionalProperties: false
      description: OpenAPIv3 Schema for Consumption Interface
      properties:
        namespace:
          type: string
          description: Namespace of the component
          default: cci-svc
        podVMSupported:
          type: boolean
          description: This field indicates whether PodVMs are supported on the environment
          default: false
        virtualIP:
          type: string
          description: IP address of the Kubernetes LoadBalancer type service fronting the apiservers.
          default: ""
---
apiVersion: data.packaging.carvel.dev/v1alpha1
kind: PackageMetadata
metadata:
  name: cci-ns.vmware.com
spec:
  displayName: Local Consumption Interface
  shortDescription: Provides the Local Consumption Interface for Namespaces within vSphere Client.
  longDescription: Provides the Local Consumption Interface for Namespaces within vSphere Client.
  providerName: VMware

The LCI in VKS provides vSphere resources (VM Service) locally via the Kubernetes Supervisor. It makes virtual machines “consumable” for Kubernetes, as if they were native K8s objects. Automation uses the LCI for the deployment of VMs.

To install the LCI, it must first be created as a supervisor service. To do this, go to the vCenter burger menu and select Supervisor Management -> Namespaces -> Services -> Add.

After successful registration, another tile should now appear under Services, and we can roll out the service via Local Consumption Interface -> Actions -> Manage Service. The Supervisor Cluster must be selected, and then the dialog must be confirmed with Next until the end. No additional information is required. If everything has worked, the service is active on the Supervisor. This can be checked via Supervisor Management.

Troubleshooting VKS

Unfortunately, you cannot simply log in to the supervisor. Some of you may have noticed that we do not assign a single password. To access the supervisor’s root account, you must log in to vCenter via SSH and switch to the root shell. Then you run a Python script and get the supervisor’s password.

ssh root@vcf09-vcsa.lab.vcf 
shell
/usr/lib/vmware-wcp/decryptK8Pwd.py

The output should look something like this:

root@vcf09-vcsa [ ~ ]# /usr/lib/vmware-wcp/decryptK8Pwd.py
Read key from file

Connected to PSQL

Cluster: domain-c12002:8f68f1b2-a1ac-4fce-a9bf-26c7492b2fcb
IP: 10.29.0.19
PWD: xxxxxxxxxxxxxxxxx

Since the supervisor is in a public VPC network, we should now be able to access the supervisor via SSH. This can be done from the vCenter or directly.

You can display the status of the pods using kubectl get pods -A. These should be in running or completed status.

root@42022d56cee15501c2f40bcedfcb23b5 [ ~ ]# kubectl get pods -A
NAMESPACE                                   NAME                                                              READY   STATUS      RESTARTS         AGE
kube-state-metrics-domain-c12002            kube-state-metrics-5df7f5548d-jr68x                               2/2     Running     2 (4h20m ago)    30h
kube-system                                 cns-storage-quota-extension-8fd45b984-2p4m6                       1/1     Running     1 (4h20m ago)    30h
kube-system                                 coredns-86688d758-2j2gr                                           1/1     Running     9 (4h20m ago)    30h
kube-system                                 docker-registry-42022d56cee15501c2f40bcedfcb23b5                  1/1     Running     1 (4h20m ago)    30h
kube-system                                 etcd-42022d56cee15501c2f40bcedfcb23b5                             1/1     Running     1 (4h20m ago)    30h
kube-system                                 kube-apiserver-42022d56cee15501c2f40bcedfcb23b5                   1/1     Running     3 (4h19m ago)    30h
kube-system                                 kube-controller-manager-42022d56cee15501c2f40bcedfcb23b5          1/1     Running     3 (4h19m ago)    30h
kube-system                                 kube-proxy-rdk2p                                                  1/1     Running     1 (4h20m ago)    30h
kube-system                                 kube-scheduler-42022d56cee15501c2f40bcedfcb23b5                   2/2     Running     5 (4h19m ago)    30h
kube-system                                 kubectl-plugin-vsphere-42022d56cee15501c2f40bcedfcb23b5           1/1     Running     5 (4h19m ago)    30h
kube-system                                 snapshot-controller-576bd689df-ff7jg                              1/1     Running     1 (4h20m ago)    30h
kube-system                                 storage-quota-webhook-6b8c9c45b9-c2pwx                            1/1     Running     5 (4h19m ago)    30h
kube-system                                 supervisor-authz-service-controller-manager-9c666c964-4g4t6       1/1     Running     1 (4h20m ago)    30h
kube-system                                 wcp-authproxy-42022d56cee15501c2f40bcedfcb23b5                    1/1     Running     2 (4h19m ago)    30h
kube-system                                 wcp-fip-42022d56cee15501c2f40bcedfcb23b5                          1/1     Running     1 (4h20m ago)    30h
...
vmware-system-zoneop                        zone-operator-58bc66b7bb-wvpjf                                    1/1     Running     1 (4h20m ago)    30h

With kubectl get nodes, you can see whether the control plane and the agents on the ESX servers are ready.

root@42022d56cee15501c2f40bcedfcb23b5 [ ~ ]# kubectl get nodes
NAME                               STATUS   ROLES                  AGE   VERSION
42022d56cee15501c2f40bcedfcb23b5   Ready    control-plane,master   30h   v1.31.6+vmware.3-fips
vcf09-m02-esx01.lab.vcf            Ready    agent                  30h   v1.31.6-sph-vmware-clustered-infravisor-trunk-85-g71ed1bf
vcf09-m02-esx02.lab.vcf            Ready    agent                  30h   v1.31.6-sph-vmware-clustered-infravisor-trunk-85-g71ed1bf

To obtain a log file for a single pod, you can use the following command:

kubectl logs -n namespace -p POD/name 

An output from the Coredns Pod looks like this, for example:

root@42022d56cee15501c2f40bcedfcb23b5 [ ~ ]# kubectl logs -n kube-system -p POD/coredns-86688d758-2j2gr  
.:53 on 172.26.0.3
[INFO] plugin/reload: Running configuration SHA512 = 0e63e439c10056457d08907176fbd8310056b76cf036ff455749731bc2e3eac256b51dfe7d2da9cddace06530a8353dbf6f9814cf475b3267f0f664f6d6b4241
CoreDNS-1.11.3
linux/amd64, go1.21.8 X:boringcrypto, v1.11.3+vmware.8-fips
[INFO] 172.26.0.3:39797 - 26176 "HINFO IN 234899636062536918.376157124480636184. udp 55 false 512" NXDOMAIN qr,rd,ra 130 0.013725101s
[INFO] 172.26.0.3:55311 - 31871 "AAAA IN 42022d56cee15501c2f40bcedfcb23b5.vmware-system-monitoring.svc.cluster.local. udp 93 false 512" NXDOMAIN qr,aa,rd 186 0.000161466s
[INFO] 172.26.0.3:49051 - 45168 "A IN 42022d56cee15501c2f40bcedfcb23b5.vmware-system-monitoring.svc.cluster.local. udp 93 false 512" NXDOMAIN qr,aa,rd 186 0.0000998s
[INFO] 172.26.0.3:37736 - 62871 "AAAA IN 42022d56cee15501c2f40bcedfcb23b5.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000090623s
[INFO] 172.26.0.3:59085 - 42593 "A IN 42022d56cee15501c2f40bcedfcb23b5.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000124409s
[INFO] 172.26.0.3:47017 - 52562 "AAAA IN 42022d56cee15501c2f40bcedfcb23b5.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.00005269s
[INFO] 172.26.0.3:56547 - 58782 "A IN 42022d56cee15501c2f40bcedfcb23b5.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000023421s
...

If none of this helps, then you have to sacrifice your firstborn to the Kubernetes gods. Experience in the lab has shown that either the network is set up incorrectly or the control plane is too small.

Finally, we have made all the preparations to configure our automation.

Setup Automation - my first tenant

After redeploying my VKS three times due to some typos, I can now log in to the automation GUI. The organization name to log in to Provider Management is system. The username is admin and the password is the one you set during deployment. If you want, you can replace the TLS certificate with your own via VCF Operations. However, this is not a must.

If the Automation start screen looks different after logging in to the provider, the VKS cluster was probably not recognized. An inventory sync can help here. I use Quick Start for setup because it creates a complete organization with useful settings in a very simple way. An organization is the top level in automation and could also be described as a customer or tenant. An organization is implemented as an NSX project, which ensures separation at the network level. Resources, cost control, and policies always apply across the entire organization. But more on that later.

Using the Quick Start Wizard, the first thing I have to do is choose the name of my organization. In my setup, I name the organization SDN Warrior. Next, I have to create a region and assign at least one VKS Supervisor cluster. A region consists of one or more supervisors and provides organizations with a collection of compute, memory, storage, and networking resources. It can extend across multiple vCenter instances. An organization can contain one or more regions.

Next, a storage policy must be selected. Here, I select my VKS policy. This can be adjusted later under Region Quota. That was it, basically. The rest is created completely automatically.

Pretty simple. But we’re not quite done yet, and we should first go through the settings that were created automatically.

If we now click on Organization in the Automation menu, we get a good overview of what has been set up, and it quickly becomes apparent that Quickstart has set up significantly more than we had to configure. A region quota has been configured, which we can use to configure the permitted VM classes. By default, all default classes are permitted. These VM classes are our T-shirt sizes. The existing VM classes can be customized in vCenter under Supervisor Management -> Services. It takes some time before newly created VM classes can be selected in the Quota.

Under Networking in the organization, you can see that the VPC default settings have been used. If there are multiple edge clusters, a specific edge cluster can be selected. In the VPC Connectivity Profile, you can override the ingress and egress QoS settings. By default, these are unlimited. IP ranges cannot be configured in the Quota, but the public VPC network is used by default. However, these IP ranges are only used for external communication. IP ranges within the organization are managed by the organization itself and not by the provider.

By far the most important setting for our lab is the first user. Currently, no user is stored here. We should change that now. Only one user can be created; for everything else, a single sign-on solution is required, but that is worth its own blog article. I give my first user all rights. This should be the break class account. Normally, you will work with users from an identity source. But who cares about security in the lab?

After creating a user for the Organization Portal, we take a quick look at the network settings. Under Networking -> General -> IP Blocks, we can customize or expand the public IP block. IP blocks represent IPs used both inside and outside this local data center, north and south of the provider gateway. IPs within this scope are used for configuring services and networks.

Under Regions, you can customize the storage class or add more. Going through all the settings in detail would go beyond the scope of this blog, and we want to get results first. But I have at least highlighted the most important settings.

Organization Portal

Everything else is done in the Organization Portal. To do this, go to Organizations, select the organization, and click Launch Organization Portal. Alternatively, we log out of the provider portal, enter the name of our organization as the organization, and log in with the user we created earlier.

Project

We see that a project has already been created by default. This is done automatically when creating an organization. We edit this by clicking on Projects and then on the project name. A brief word about what a project is. A project connects users with the resources they are entitled to and the restrictions they are subject to. A project typically defines an application development team, its users, how much and what type of infrastructure they can use, and which catalog items they can access.

I am renaming my project to a-team. All projects must be written in lowercase. I save the project before creating the vSphere namespace. Then I edit the project again and create a namespace. This refers to a vSphere namespace from VKS. It consists of one of the predefined namespace classes and can be customized under Manage & Govern if necessary. In my case, small (best effort) is sufficient, which means that my project will not be allocated more than 10 GB of RAM and 10 GHz of CPU without reservations. Namespace classes can be used to manage resources effectively. In addition, the region, VPC, and zone must be specified. This is not difficult in my lab, as there is only one of each.

DHCP and DNS

Now we have almost everything we need to deploy a VM. However, I want to use DHCP in my lab, so I need to make a small adjustment in NSX. By default, DHCP can be used with automation, but the VPC service profile does not have a DNS server stored. This can be changed quickly. To do this, I log in to NSX and switch to the SDN-Warrior project (click on Default in the upper left corner of NSX to switch projects). The profile can be found under VPCs -> Profiles -> VPC Service Profile -> Default VPC Service Profile. Here, I add my DNS server to the profile.

Once that has been changed, I create a new subnet set in the Organization Portal. Under Build & Deploy -> Services -> Network -> vcf09-management-domain-default-vpc -> SubnetSets, I create a new subnet set with the name vm and set the DHCP server. The subnet set is, so to speak, my blueprint for how I use my network. It determines the type of network the VM will be in (in this case, a private network) and how an IP address will be assigned.

Content library

We also need a content library. I create this via Build & Deploy -> Content Libraries. The region must be specified and the storage policy selected. I can then upload my OVA template from Alpine Linux to the newly created content library. It is important to note that the space required is deducted from the configured quota created in the provider setup. You don’t have to use Alpine Linux here; any other VM template you have created previously will also work. I use Alpine because of its small size and because it is super easy to configure.

Blueprint Design

Now that we have a basic setup, a customized network, and an image in our content library, we can write the blueprint. A blueprint in VCF Automation is a standardized description of how a VM, service, or application should be automatically provisioned. To do this, we click on Build & Deploy -> Blueprint Design -> New from Blank Canvas. The blueprint needs a name, in my case AlpineLinux, and it must be assigned to a project.

The Blueprint designer appears, and the developer now has to familiarize themselves with YAML. To deploy a simple VM, the namespace must first be specified. To do this, we need a resource of type CCI.Supervisor.Namespace. Then another resource is needed for the actual VM. This is of type CCI.Supervisor.Resource. You can drag and drop the two resources into the canvas or type the YAML directly. I have a relatively simple YAML here that creates an Alpine Linux from a VM template. However, it can also be used with other VM templates. There is also a simple input form so that you can specify the VM name. This must be rfc dns compliant, and a simple regex is also included. The blueprint needs to be slightly modified so that it can be used.

formatVersion: 1
inputs:
  textField_6e3db194:
    type: string
    title: VM Name
    description: DNS-kompatibler Name (lowercase, RFC-konform)
    pattern: ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$
resources:
  CCI_Supervisor_Namespace_1:
    type: CCI.Supervisor.Namespace
    properties:
      name: a-team-space-8sjpd
      existing: true
  Virtual_Machine_1:
    type: CCI.Supervisor.Resource
    properties:
      context: ${resource.CCI_Supervisor_Namespace_1.id}
      manifest:
        apiVersion: vmoperator.vmware.com/v1alpha3
        kind: VirtualMachine
        metadata:
          name: ${input.textField_6e3db194}-${env.shortDeploymentId}
        spec:
          className: tiny
          imageName: vmi-bee6d37ed943e7fc7
          powerState: PoweredOn
          storageClass: vks-storage-policy
          network:
            interfaces:
              - name: eth0
                network:
                  name: vm
                  kind: SubnetSet
          bootstrap:
            cloudInit:
              cloudConfig:
                runcmd:
                  - 'disable_vmware_customization: true'
      wait:
        conditions:
          - type: VirtualMachineCreated
            status: 'True'

To make it easier, I’ll split the YAML into sections and describe where changes need to be made. The vSphere namespace must be specified here. This can be found under Built & Deploy -> Services -> Overview. Alternatively, it can also be found in vCenter or under Manage & Govern -> Namespaces.

resources:
  CCI_Supervisor_Namespace_1:
    type: CCI.Supervisor.Namespace
    properties:
      name: a-team-space-8sjpd
      existing: true

• The VM class must be adjusted in the VM block, as I am using a custom class called Tiny. The existing VM classes can be found under Build & Deploy -> Services -> Virtual Machine -> VM Classes. One of the standard classes, for example, is best-effort-xsmall.

• Image Name must be replaced with the image to be used. This can be found under Build & Deploy -> Content Hub -> VM Images, The image identifier must be used.

• The storage class must be adapted to the existing one. This can be found under Build & Deploy -> Services -> Volume -> Storage Classes.

• Finally, the network must be adjusted. If you delete the entire block from Network, the default SubnetSet will be used, on which DHCP is not active. In the previous steps, I created my own SubnetSet with the name vm.

spec:
className: tiny
imageName: vmi-bee6d37ed943e7fc7
powerState: PoweredOn
storageClass: vks-storage-policy
network:
  interfaces:
    - name: eth0
      network:
        name: vm
        kind: SubnetSet

This part does not need to be adjusted, but I would still like to briefly explain why it is included. Without this adjustment, the VM is created and connected to the correct network, but the Alpine VM is always disconnected afterwards. This can be changed via the vCenter, but that defeats the purpose. The user who will later consume the VMs should not have access to the vCenter. I’ve had this problem with various Linux distributions, but I can’t say whether it’s a general Linux issue.

bootstrap:
  cloudInit:
    cloudConfig:
      runcmd:
        - 'disable_vmware_customization: true'

The Bluebrint should now look more or less like this.

Using the Test button, you can validate the YAML to ensure that at least the formatting is correct. Using Deploy, you can run a deployment test. Once this has been successful, you can use Version to publish the current version in the catalog and make it accessible to users. I recorded a short test video to show how it all works in practice. To do this, I used an advanced project user with limited rights to the project.

Bonus Round - deploying a Kubernetes Cluster

Spoilers: this isn’t a perfect blueprint, but it just shows that you can do more than just deploy VMs with VCF Automation in All Apps mode.

formatVersion: 1
inputs:
  textField_6e3db193:
    type: string
    title: clustername
    description: DNS-kompatibler Name (lowercase, RFC-konform)
    pattern: ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$
resources:
  CCI_Supervisor_Namespace_1:
    type: CCI.Supervisor.Namespace
    properties:
      name: a-team-space-8sjpd
      existing: true
  Kubernetes_Cluster_1:
    type: CCI.Supervisor.Resource
    properties:
      context: ${resource.CCI_Supervisor_Namespace_1.id}
      manifest:
        apiVersion: cluster.x-k8s.io/v1beta1
        kind: Cluster
        metadata:
          name: ${input.textField_6e3db193}-${env.shortDeploymentId}
        spec:
          clusterNetwork:
            pods:
              cidrBlocks:
                - 192.168.156.0/20
            services:
              cidrBlocks:
                - 10.96.0.0/12
            serviceDomain: cluster.local
          topology:
            class: builtin-generic-v3.4.0
            classNamespace: vmware-system-vks-public
            version: v1.33.3---vmware.1-fips-vkr.1
            variables:
              - name: vmClass
                value: best-effort-small
              - name: storageClass
                value: vks-storage-policy
            controlPlane:
              replicas: 1
              metadata:
                annotations:
                  run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-7685ae58460e6c079
            workers:
              machineDeployments:
                - class: node-pool
                  name: kubernetes-cluster-seil-np-1xf6
                  replicas: 1

To deploy the cluster, only the namespace needs to be adjusted. The rest should run in every automation instance, as I only use standards here.

 CCI_Supervisor_Namespace_1:
    type: CCI.Supervisor.Namespace
    properties:
      name: a-team-space-8sjpd
      existing: true

Conclusion

VCF 9 Automation is not just a simple update to the familiar automation solution; it is a new product. In this blog, I have covered some super basic tasks, but there is much more that can be done, and I have only scratched the surface. A general rethink of how VMs are managed is needed, because if my VM has been deployed in a vSphere namespace, it is no longer managed via vCenter. This should also make it clear that a simple migration from Automation 8.X to Automation 9 is not easily possible (unless you use Legacy Mode). However, this is the future of the VCF platform. True multitenancy can only be achieved with VCF Automation. A rethink is necessary, and I look forward to exciting new possibilities with Automation.

I currently have two different systems in my home lab for monitoring things. I started everything but never really finished it. I have a Grafana dashboard for power consumption and an Uptime Kuma to check the general availability of systems. As I have to run a VCF Operations anyway so that I can license my Baselab, I thought, why not use Operations for monitoring purposes, since it’s there anyway.

How hard can it be? Spoiler alert: I never thought I’d be building a Docker container, using a proxy, and converting things with Python, but let’s dive down the rabbit hole.

First things first, where do we start?

In VCF 9, the Management Pack Builder is integrated into Operations. Management Pack Builder? Yes, that’s the tool I’ve probably seen the most over the last few days. The MP allows you to integrate external systems or data sources and use them to create metrics and properties. Sounds simple, and it is—in theory. But first you have to find the tool, because it has been well hidden in Operations. You can find the MP under Administration -> Integrations -> Marketplace and then access it via the Create Management Pack button or or Developer Center -> Managment Pack Builder.

My first management pack

I will describe the process using the Unraid API as an example. My UPS itself does not have an API, but my Unraid storage system can read my UPS data via a USB connection. Since Unraid 7.2, there is now also an API that provides me with the data - nice.

In principle, however, it can be said that depending on the API, MP can be super easy or hellish. More on that later. Unraid relies on GraphQL, which is ideal for VCF operations and, after a brief period of familiarization with how to access the data, is very easy to use. The nice thing is that there is exactly one endpoint, namely /graphql, and Unraid has a built-in Apollo server where you can easily click together your queries.

Define sources

First, a source must be defined in MP, which is usually relatively easy. But here’s the first stumbling block: please use variables if you are using anything other than Basic Auth.

The data source serves as a template, which is used to develop the management pack. Once the management pack has been created, there are no more credentials in the data source, provided that you have followed the guidelines and not entered any credentials without variables. If you use basic authentication, variables are used automatically.

The actual configuration is fairly straightforward. you have to select a collector, the API endpoint, and the port. I have disabled certificate verification, but in production it should of course remain enabled. The base path is optional; for Unraid, graphql would be a good choice. You don’t have to specify a leading /, this is done automatically. Since Unraid uses API tokens, I created a variable called API-Toke via Custom Authentication.

Next are the global settings, which are very API-specific, and I will show another variation with the Veeam API later on. Unraid is relatively simple here; content type - application/json is more or less standard for most APIs, and my token must be specified in the HTTP header with x-api-key. The small paste icon allows you to access the defined variables, and everything is automatically entered correctly so that the previously defined variable is used.

The final step is to test the connection. If the test is not successful, the connection cannot be saved. Any query can generally be used. In my example, I display the system version of my Unraid server. Unraid only uses POST queries. The global HTTP headers are added automatically.

To query data, you have to send a query in the body with Unraid. This is again very specific.

If everything works, you will receive a valid response on the right-hand side and you can save the connection. Step 1 is now successfully complete.

Create Requests

Next, the requests must be created. These requests will then be used later to retrieve the data.

In my example, I query the utilization of my UPS via the Unraid API so that I can use the data later in an object. And as you can see here, I didn’t follow my own recommendation for this request and my API token is hardcoded in the request. Please do better. In my defense, that was my first request, and I will adjust it in the next update of the management pack.

Create Objects

Next, the objects must be created. Each object needs at least one identifier, which must be unique, and an object can contain several metrics and/or properties. Metrics and properties are not mandatory. An object can therefore consist of metrics only or properties only. The difference between metrics and properties is quite simple: metrics can be calculated and must be of the decimal data type, while properties are strings. The unique ID is implemented via a property. Each object can only receive data via one request.

When selecting the attributes, it is important to pay close attention to what is selected and always rely on the result of the API. In the end, clearly assignable values are required. Arrays such as upsDevices.* or data.* are not suitable for metrics. In this case, data.upsDevices.* must be selected, as all results always belong to exactly one upsDevice and can therefore ideally represent an object. If there are several UPS systems in upsDevices, several objects are automatically created.

Selecting the metrics and properties is super simple, but at the same time it’s the biggest problem when dealing with APIs. Unfortunately, you can’t convert the results in any other way; it’s either a string or a decimal. It’s the same as when you only have a hammer, then suddenly everything looks like a nail. In my cases with different APIs, almost everything was a stirng. At least with Decimal, you can still specify the most common units, such as %, bit, byte, byte/s, and so on.

Finally, you just need to select a property as the object instance name and an object ID, which must of course be unique. Since I only have one UPS, I can use the UPS name as both the object instance name and the object ID.

Finally, a verification must be performed before the Management Pack can be installed. Relationshops and events are optional, and for my purposes, I do not need them at this time and will therefore leave them out.

Congratulations, we have now created our first objects and metrics definition.

Side quest: Incompatible APIs, or rather, what to do when I need a metric but only get properties.

Well, friends, I had this problem with the OPNSense API. I can create objects perfectly well using the method described, but the API only returns a string with, for example, 6.7 ms for the latency. The Management Pack Builder can only interpret this as a string because of the unit that is attached, which means I can’t use it for calculations or to display a metric history, for example. But there is a solution.

FastAPI + Phyton = beautiful metrics

What is FastAPI? FastAPI was created to build type-safe REST APIs quickly and easily with Python. I use uvicorn as a fast and lightweight web server. I did the development in PyCharm. The tool runs on my system as a docker. In PyCharm, I created a new project and created exactly two files: requirements.txt and my actual proxy.py. The required libraries are listed in requirements.txt.

requirements.txt:

fastapi
uvicorn[standard]
requests

proxy.py:

import os, re
import requests
from fastapi import FastAPI, Request, Response, HTTPException

UPSTREAM = os.environ.get("UPSTREAM_BASE", "https://xxx.xxx")
VERIFY_TLS = os.environ.get("VERIFY_TLS", "false").lower() == "true"
TIMEOUT = float(os.environ.get("TIMEOUT", "10"))

app = FastAPI()

_ms = re.compile(r"^\s*([0-9]+(?:\.[0-9]+)?)\s*ms\s*$", re.I)
_pct = re.compile(r"^\s*([0-9]+(?:\.[0-9]+)?)\s*%\s*$", re.I)

def parse_ms(v):
    m = _ms.match(v or "")
    return float(m.group(1)) if m else None

def parse_pct(v):
    m = _pct.match(v or "")
    return float(m.group(1)) if m else None

@app.get("/api/routing/settings/search_gateway")
def search_gateway(request: Request):
    auth = request.headers.get("authorization")
    if not auth:
        raise HTTPException(status_code=401, detail="Missing Authorization header")

    url = f"{UPSTREAM}/api/routing/settings/search_gateway"
    headers = {"Authorization": auth, "Accept": "application/json"}

    try:
        r = requests.get(url, headers=headers, timeout=TIMEOUT, verify=VERIFY_TLS)
    except requests.RequestException as e:
        raise HTTPException(status_code=502, detail=f"Upstream request failed: {e}")

    if r.status_code >= 400:
        return Response(
            content=r.text,
            status_code=r.status_code,
            media_type=r.headers.get("content-type", "text/plain"),
        )

    try:
        data = r.json()
    except ValueError:
        raise HTTPException(status_code=502, detail="Upstream returned invalid JSON")

    for row in data.get("rows", []):
        if isinstance(row.get("delay"), str):
            v = parse_ms(row["delay"])
            if v is not None:
                row["delay_ms"] = v

        if isinstance(row.get("stddev"), str):
            v = parse_ms(row["stddev"])
            if v is not None:
                row["stddev_ms"] = v

        if isinstance(row.get("loss"), str):
            v = parse_pct(row["loss"])
            if v is not None:
                row["loss_pct"] = v

    return data

The whole thing is started in my development environment with uvicorn proxy:app –host 0.0.0.0 –port 666

Of course, this is only intended for development purposes. Later, I built a Docker container from it that runs on my Unraid server, is backed up, and also starts automatically. It is important to note that authentication is not touched and is simply passed on. In Python, I parse the values for delay, stddev, and loss and remove the units. However, I keep the original values and simply output additional values that do not have units. FastAPI thus serves as an API proxy, and I no longer address my OPNSense directly, but rather the FastAPI Docker container, which then returns a usable API to me.

But back to the actual topic.

Building Dashboards

I now have my object definitions, and in order to actually receive data, I need to create an account under Administration -> Integrations. Unraid (the created management pack) is now also available as an account type, and I need to enter the data for the API here. Host name, collector, port, SSL configuration, and so on.

To view my objects now, you can find them under Inventory -> Integrations. The object type UPSLoad that I created can be found in the Unraid integration. If the API returned more than one UPS here, there would be two objects of the object type UPSload. Within the object, you can find the queried values in the Metric Power Load Percentage. It takes at least 5 minutes for the first values to appear.

You can now work with these values. However, I only have a percentage value and I would prefer to see the watts displayed, because I can imagine more under 300 watts than under 34% usage. But how does that work?

Super Metrics

Super metrics are values that can be calculated using other metrics. You can also use super metrics for super metrics. However, it is important to remember that the source metric must be available before the super metric can be calculated. It is also important that the super metric is assigned to the correct object type. In my case, the object type is UPSLoad.

A Super Metric must also be assigned to a policy. If this is forgotten, there will be no values. I use the default policy that is used for every object here. However, you could also write a specific policy.

The rest is simple math. Since I know the maximum power of the UPS, i.e., how many watts 100% is, I can easily convert the percentage into watts.

After 1-2 intervals have passed, the Super Metric is displayed in the object and can be used. This can be checked via Inventory.

Building the Dashboard

Now that the basics are done and the first objects are appearing in Operations, I can start building a dashboard. A new dashboard can be created under Infrastructure Operations -> Dashboards & Reports. There are quite a few widgets available for displaying various metrics. For watts, a metric chart is useful, as it can show consumption over a certain period of time. Each widget has a refresh counter (which only determines the interval at which the widget is updated, not the object). Another important option is Self Provider. On means the widget is independent of other widgets. It does not take input from other widgets, hence it requires input to be configured in the widget itself.

Under Input Data, you can choose whether you want to have a complete object (or several) as input or just metrics. If I take the object as input, I can also display properties. The actual output of the widget is then configured under Output Data. Input and output must match. If I select an object as input that is not of the same object type as my output metric, no content will be displayed.

The finished widget now looks something like this.

It wasn’t that difficult, and now all the basics are in place to build a nice dashboard. Of course, there’s a lot more you can do, such as symptom definitions that trigger something when certain metrics occur or properties change. For example, I wrote a symptom definition that reduces the health value of a Veeam object if the backup status is not successful. This also triggers an alarm.

But I think I’d rather write about that separately, because it’s a topic in its own right. The same goes for the Health Metric, which automatically assigns operations to each object.

My final dashboard now looks like this.

Side Quest - Bearer Token

The Veeam API uses bearer tokens with limited runtime, which means that an API request must first be created with a username and password to generate a token before a request to retrieve data can be made. Fortunately, the Management Pack Builder also supports this, but it’s all a bit fiddly, so I’ll describe it here. For Veeam to work, authentication in the Data Source Connection must be set to Custom, otherwise we will not be able to create a secure variable. It is also important to enable Use Session Authentication.

From here on, things change slightly, as we now have to perform two requests. It is also important to note that the username and password must be included in the body for Veeam. A variable is also used for the password here. From here on, things change slightly, as we now have to perform two requests. It is also important to note that Veeam also requires special headers. However, as Veeam uses Swagger, you can extract all of this information from there.

After successfully creating a session, we can now see the token under Global Settings in Session Fields. Session Fields are automatic variables that must be used for the Test Connection and also for the Release Session request. We don’t need to change anything in Global Settings, and since I didn’t want to censor another screenshot, I decided not to bother.

A release session request is nothing more than a logout command, after which the data is retrieved. This should be used to ensure that the session is closed properly.

As a final step, a test connection must be performed so that the DataSource can be saved. I simply queried the session information for this, but ultimately it doesn’t matter what is queried here as long as a valid result is returned.

If the data source is set up in this way, the Management Pack is also secure for export, as no credentials are stored. When integrating into Operations, the username and password must then be provided. The session is created, the token is generated, and the session is closed cleanly with each data retrieval cycle. Everything else is exactly the same as previously described for the Unraid API.

Summary

VCF Operations is a powerful tool, albeit a little outdated in some areas, particularly when it comes to handling APIs, where you can still see the product’s roots and age. Nevertheless, with a little work, Operations can be expanded wonderfully to monitor more than just the VMware environment. If you want, you can build a weather report into the dashboard – why not?

Of course, this article only scratches the surface; we haven’t talked about views yet, or how the whole Alarming works, or how to work with colors, and so on. Considering that it’s already almost 3 a.m. and the topic is so extensive, I’ll write a more in-depth article if there’s interest. I hope I’ve been able to help you make even better use of your operations.

Looking at my final dashboard, I’m quite excited about how much you can achieve with it in a short amount of time. I only looked into the topic about a week ago and have been cautiously playing around with it. My thanks also go to Dale Hassinger, who was kind enough to help me. Check out his blog.

And now all that remains for me to say is, happy creating and Merry Christmas!

In my last Dark Site article, I showed how to deploy a Dark Site Edge on a host (with a temporary second host). However, this didn’t sit well with me, as I believe it should also be possible with just one host and without any additional tools. Spoiler alert: it works, and today I’m going to write down how to do it.

Of course, we need a few workarounds here, and I was unable to create the setup with external storage such as NFS. So today I will use vSAN ESA, even though it requires significantly more resources than NFS.

So buckle up, we’re going in.

Hardware modification

Since I want to use vSAN for this deployment, because a greenfield deployment with external storage is not possible with only one node, I first need to expand my hardware. Fortunately, I purchased my NVMes before the price chaos. I need a total of three NVMes. I have a 1 TB NVMe (PCIe4) for memory tiering, a 2 TB NVMe (PCIe3) for local storage and the actual ESX. This could be a much smaller disk, but I had it anyway. And finally, another 2 TB NVMe (PCIe3) for vSAN.

I could have saved myself an NVMe and booted the ESX via a USB stick, because then you can share the memory tiering disk with the system disk, but I simply only have one USB stick left and I need it for firmware updates, plus I don’t particularly like USB sticks. Now you might ask why I have two PCIe3 disks—well, I don’t. However, the MS-01 only uses one slot with PCIe4 by default BIOS settings, and the others run with PCIe3, as otherwise there is a risk of overheating. In addition, the MS-01 will serve another purpose later on, as it will sooner or later become part of my Nutanix cluster, and I will need the disks for that anyway.

Prerequisites

As with my first attempt, a deployed fleet is required. This must consist of at least one VCF Operations, vCenter, NSX Manager, VCF Operations Cloud Proxy, and one Fleet Manager.

Workarounds

What would an SDN-warrior deployment be without workarounds? That’s right, work and no fun.

Workaround 1 (E&P Cores):

Since I use the MS-01, the first thing that comes into play is the workaround for the Big.LITTLE architecture. I have described this here.

Workaround 2 (Memory Tiering):

Memory tiering is also used, as described in the same article like the Big.LITTLE workaround.

Workaround 3 (Single host deployment):

I have described this here. However, with my 9.0.1 setup, I needed both entries in the feature.properties file.

echo "feature.vcf.vgl-29121.single.host.domain=true" >> /home/vcf/feature.properties
echo "feature.vcf.internal.single.host.domain=true" >> /home/vcf/feature.properties

Workaround 4 (Bypass vSAN HCL):

Now we finally come to something new—at least, something new for me. In VCF 9.0.1, you can finally bypass the HCL check for vSAN ESA. Gone are the days of custom JSON vSAN files and VIB installation on ESX.

To do this, simply log in to the installer via SSH and enter the following in the shell.

echo "vsan.esa.sddc.managed.disk.claim=true" >> /etc/vmware/vcf/domainmanager/application-prod.properties
systemctl restart domainmanager

Now that most of the work has been done, we can proceed with the deployment. Just a quick heads-up: there will be a small workaround after the SDDC Manager has been successfully deployed.

Deployment

Now comes the really exciting part. Unfortunately, deployment via the guided GUI is not possible with vSAN and single host, but that’s what JSON is for. There are a few good VCF JSON generators available online, such as this one, or you can simply use my JSON and modify it. As with the first Dark Site setup, I am using my existing Fleet, which means I don’t have to deploy VCF Operations, Automation, Operations for Logs, or even the Fleet Manager.

To obtain the SSL thumbprints, simply log in to the ESX server and use the following command. Adjust the FQDN in the query for each thumbprint.

echo | openssl s_client -connect vcf09-e01-esx01.lab.vcf:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2

Here is the JSON file I used for VCF 9.0.1. Maybe I should create a GitHub repository for things like this.

{
    "sddcId": "dark-site03",
    "vcfInstanceName": "dark-site03",
    "workflowType": "VCF",
    "version": "9.0.1.0",
    "ceipEnabled": false,
    "skipEsxThumbprintValidation": true,
    "dnsSpec": {
        "nameservers": [
            "192.168.11.2",
            " 192.168.100.254"
        ],
        "subdomain": "lab.vcf"
    },
    "ntpServers": [
        "192.168.12.1"
    ],
    "vcenterSpec": {
        "vcenterHostname": "vcf09-e03-vcsa.lab.vcf",
        "rootVcenterPassword": "xxx",
        "vmSize": "small",
        "storageSize": "",
        "dminUserSsoUserName": "administrator@vsphere.local",
        "adminUserSsoPassword": "xxx",
        "ssoDomain": "vsphere.local",
        "useExistingDeployment": false
    },
    "clusterSpec": {
        "clusterName": "dark-site03",
        "datacenterName": "e03"
    },
    "datastoreSpec": 
    {
        "vsanSpec": {
            "failuresToTolerate": 0,
            "vsanDedup": false,
            "esaConfig": {
            "enabled": true
        },
        "datastoreName": "vsanDatastore"
        }
    },
    "nsxtSpec": {
        "nsxtManagerSize": "medium",
        "nsxtManagers": [
            {
                "hostname": "vcf09-e03-nsxa.lab.vcf"
            }
        ],
        "vipFqdn": "vcf09-e03-nsx.lab.vcf",
        "useExistingDeployment": false,
        "nsxtAdminPassword": "xxx",
        "nsxtAuditPassword": "xxx",
        "rootNsxtManagerPassword": "xxx",
        "skipNsxOverlayOverManagementNetwork": true,
        "ipAddressPoolSpec": {
            "name": "host-overlay",
            "description": "",
            "subnets": [
                {
                    "cidr": "10.28.24.0/24",
                    "gateway": "10.28.24.1",
                    "ipAddressPoolRanges": [
                        {
                            "start": "10.28.24.21",
                            "end": "10.28.24.30"
                        }
                    ]
                }
            ]
        },
        "transportVlanId": "2024"
    },
    "vcfOperationsSpec": {
        "nodes": [
            {
                "hostname": "vcf09-ops.lab.vcf",
                "rootUserPassword": "xxx",
                "type": "master",
                "sslThumbprint": "XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX"
            }
        ],
        "adminUserPassword": "xxx",
        "applianceSize": "xsmall",
        "useExistingDeployment": true,
        "loadBalancerFqdn": null
    },
    "vcfOperationsFleetManagementSpec": {
        "hostname": "fleet.lab.vcf",
        "rootUserPassword": "xxx",
        "adminUserPassword": "xxx",
        "useExistingDeployment": true,
        "sslThumbprint": "XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX"
    },
    "vcfOperationsCollectorSpec": {
        "hostname": "vcf09-e03-ops.lab.vcf",
        "applianceSize": "small",
        "rootUserPassword": "xxx",
        "useExistingDeployment": false
    },
    "hostSpecs": [
        {
            "hostname": "vcf09-e03-esx01.lab.vcf",
            "credentials": {
                "username": "root",
                "password": "xxx"
            },
            "sslThumbprint": "XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX"
        }
    ],
    "networkSpecs": [
        {
            "networkType": "MANAGEMENT",
            "subnet": "10.28.22.0/24",
            "gateway": "10.28.22.1",
            "subnetMask": null,
            "includeIpAddress": null,
            "includeIpAddressRanges": null,
            "vlanId": "2022",
            "mtu": "1500",
            "teamingPolicy": "loadbalance_loadbased",
            "activeUplinks": [
                "uplink1",
                "uplink2"
            ],
            "standbyUplinks": null,
            "portGroupKey": "e03-cl01-vds01-pg-esx-mgmt"
        },
        {
            "networkType": "VM_MANAGEMENT",
            "subnet": "10.28.13.0/24",
            "gateway": "10.28.13.1",
            "subnetMask": null,
            "includeIpAddress": null,
            "includeIpAddressRanges": null,
            "vlanId": "2013",
            "mtu": "1500",
            "teamingPolicy": "loadbalance_loadbased",
            "activeUplinks": [
                "uplink1",
                "uplink2"
            ],
            "standbyUplinks": null,
            "portGroupKey": "e03-cl01-vds01-pg-vm-mgmt"
        },
        {
            "networkType": "VMOTION",
            "subnet": "10.28.23.0/24",
            "gateway": "10.28.23.1",
            "subnetMask": null,
            "includeIpAddress": null,
            "includeIpAddressRanges": [
                {
                    "startIpAddress": "10.28.23.20",
                    "endIpAddress": "10.28.23.29"
                }
            ],
            "vlanId": "2023",
            "mtu": "1500",
            "teamingPolicy": "loadbalance_loadbased",
            "activeUplinks": [
                "uplink1",
                "uplink2"
            ],
            "standbyUplinks": null,
            "portGroupKey": "e03-cl01-vds01-pg-vmotion"
        },
        {
            "networkType": "VSAN",
            "subnet": "10.28.16.0/24",
            "gateway": "10.28.16.1",
            "subnetMask": null,
            "includeIpAddress": null,
            "includeIpAddressRanges": [
                {
                    "startIpAddress": "10.28.16.40",
                    "endIpAddress": "10.28.16.50"
                }
            ],
            "vlanId": "2016",
            "mtu": "9000",
            "teamingPolicy": "loadbalance_loadbased",
            "activeUplinks": [
                "uplink1",
                "uplink2"
            ],
            "standbyUplinks": null,
            "portGroupKey": "e03-cl01-vds01-pg-vsan"
        }
    ],
    "dvsSpecs": [
        {
            "dvsName": "e03-cl02-vds01",
            "networks": [
                "MANAGEMENT",
                "VM_MANAGEMENT",
                "VMOTION",
                "VSAN"
            ],
            "mtu": 9000,
            "nsxtSwitchConfig": {
                "transportZones": [
                    {
                        "transportType": "OVERLAY",
                        "name": "VCF-Created-Overlay-Zone"
                    }
                ]
            },
            "vmnicsToUplinks": [
                {
                    "id": "vmnic0",
                    "uplink": "uplink1"
                },
                {
                    "id": "vmnic1",
                    "uplink": "uplink2"
                }
            ],
            "nsxTeamings": [
                {
                    "policy": "LOADBALANCE_SRCID",
                    "activeUplinks": [
                        "uplink1",
                        "uplink2"
                    ],
                    "standByUplinks": []
                }
            ],
            "lagSpecs": null
        }
    ],
    "sddcManagerSpec": {
        "hostname": "vcf09-e03-sddc.lab.vcf",
        "useExistingDeployment": false,
        "rootPassword": "xxx",
        "sshPassword": "xxx",
        "localUserPassword": "xxx"
    }
}

Once the typing is done, you can simply upload the JSON file via the installer’s GUI and start the validation. In the best case, it should now look like this.

You can ignore the DNS warning. I simply made a fat-finger mistake when deploying my vcf installer and mistyped one of the two DNS servers. The ESX Host vSAN warning is expected, as I am using 100% unsupported consumer hardware—server vendors hate this trick. The same applies to my vSAN Network Gateway. Since this network does not need to route, it does not have a gateway, so this message can also be ignored. So fill up your coffee cup and press deploy.

As I mentioned earlier, the installation will fail and we will have to perform a final workaround. The installer will abort at the “Configure the vSphere cluster” step because a Raid 1 vSAN policy is set by default, which we obviously cannot fulfill.

(Final) Workaround 5 (vSAN Policy):

The solution is super simple. Go to the Policies and Profiles settings via the burger menu in the vCenter and find the vSAN policy that the installer has set as default. This is usually a policy with the name of the VCF instance and Optimal Datasore Default Policy -Raid1. Lovely name. Edit this policy and set Failures to tolerate to “No data redundancy.” I don’t need to mention that this setting should never be used in production 99.9% of the time…

After that, you can simply restart the deployment in the VCF installer using retry. The deployment should now run without any problems. For me, the whole process took about 2 hours.

Nice, another successful deployment completed after a few workarounds, and this time really only on one host. I then deployed two more edge VMs and built my BGP peering. The Kubernetes cluster is still missing, but that’s a story for another blog. The finished VCF instance now looks like this.

Summary

Significantly fewer workarounds but also more RAM required than with NFS. That’s my feedback in a nutshell. Perhaps I should say a few words about performance, as I am using NVMes that are unfamiliar to me in this setup. At first glance, the Lexar EQ790s perform quite well. They have similar read values to a Samsung 990 Pro and slightly weaker write values than the aforementioned drive, but the bottom line is that I couldn’t get a Samsung 990 Pro at an affordable price.

I also did a quick test with a Windows Server VM and Crystal Disk Mark. As always, this isn’t a 100% accurate test, but it just to show that this setup is perfectly usable for a lab environment to run a few test VMs and Kubernetes workloads without me always having to turn on multiple servers. So I achieved my goal 100%. I hope I was able to give you some helpful ideas for your setups.

While preparing for an article that has not yet been published, I ran into an interesting VCF bug. For the article, I had to delete a VCF instance from my operations. For background information, I currently have three VCF instances onboarded in my fleet. Since I had to reinstall one of them, I removed it according to the official instructions, and that’s when the problems started.

Disclaimer

Error pattern and impact

At first glance, the VCF instance can be deleted normally and everything appears to be functioning correctly. The error only becomes apparent when the vCLM (aka Fleet Manager) is required and the old VCF instance has been deleted. Typical errors:

• Inventory Sync fails with the message that SDDC from instance X is not reachable
• Deployment of a new VCF instance to the existing instance of VCF Operations fails

The error shown in the screenshot occurs during the “Join the existing fleet management appliance” step in the VCF deployment. When I take a closer look at the error message, I notice that an attempt is being made to log in to an SDDC Manager:

"exceptionMessage":"Could not get APl token from SDDC Manager vcf09-e02-sddc.lab.vcf.

Coincidentally, the SDDC Manager belongs to the very VCF instance that was previously removed. A search in Operations revealed that there were still configuration remnants in Operations. Even after deleting these remnants, the error persisted—you don’t want to know how many times I deployed and deleted a VCF instance in the last two days.

Troubleshooting

After searching forever, I found an entry in the Fleet Manager database that explains the behavior, and that’s also the point where we end up in the danger zone. So buckle up, take a snapshot of the Fleet Manager, and jump into the SSH session.

I must apologize for the screenshots, but I’m glad I took them at all. What you can see in the screenshot is that the old SDDC Manager is listed in the vm_lcops_sddc_manager table. In total, there were four SDDC Managers listed here—including the certificate in the database. Together with the other issues I found in Operations, this was the reason why the VCF installer was blocked and my inventory sync wasn’t working. When deleting a VCF instance, the SDDC does not appear to be deleted from the fleet database. I had exactly the same problem when switching from the public beta 9 to the final version of VCF 9. However, I attributed this to the beta installation and therefore did not spend much time troubleshooting.

Problem solving

Since there is no official solution yet, here is my unofficial solution to the problem. I have tried this several times and was able to verify that it works. Would I do this in a production environment? Definitely not. You have been warned.

First, the integration must be deleted, just as described in the official guide.

• Administration-> Integrations - Delete Accounts

After that, the cloud proxy must be deleted, as this was overlooked.

• Administration->Cloud Proxies - Delete Cloud Proxy

The next step is to delete the deployment target.

• Fleet Management -> Lifecycle -> Settings -> Deployment Target - Delete vCenter
• If vCenter is not deletable trigger inventory sync (doublecheck if the Integration is still deletet).

At that point, I wiped my removed VCF instance.

To delete the SDDC entry in the Fleet Manager DB, you must connect to the appliance via SSH and the ROOT user.

Connect to the DB:

su - postgres
/opt/vmware/vpostgres/current/bin/psql -U postgres vrlcm

Find your SDDC Manager Entrys:

SELECT sddcmanagerhostname,vmid FROM vm_lcops_sddc_manager;

You will receive a list with the host names and the VMID.

vrlcm=# SELECT sddcmanagerhostname,vmid FROM vm_lcops_sddc_manager;

  sddcmanagerhostname   |                 vmid                 
------------------------+--------------------------------------
 vcf09-e03-sddc.lab.vcf | 0838e38b-ba5c-40e3-b50e-2c86f0b2cf3c
 vcf09-e01-sddc.lab.vcf | 06c98f38-d855-4b4a-b62c-f81fae9158e0
 vcf09-sddc.lab.vcf     | 52b7a8cc-7d41-4c30-ad5c-960eb6db515d
(3 rows)

Deleting the SDDC entry:

DELETE FROM vm_lcops_sddc_manager WHERE vmid = 'xxxx';

Reboot the fleet manager.

Summary

Congratulations, we have hopefully survived the open-heart surgery. After that, I was able to simply press resume in the VCF installer and the VCF Instanc installation ran smoothly.

Of course, I can’t say for sure whether all hidden entries from the old SDDC have been removed from the environment. The only thing to do here is wait until the bug is officially fixed. As I already mentioned, I was able to reproduce the behavior several times. Now that this problem has been solved, I can get back to working on the article I had originally planned.

I haven’t written anything in a while, and I need to change that here with another niche topic. I want to take a closer look at VCF9 Edge. This does not refer to the Edge cluster in NSX; no, VCF Edge is a special form of a VCF instance. VCF offers nine complete designs for this form, and I would like to take a closer look at one special version — the so-called Dak Site Edge.

Welcome to the dark si(d)te

The “Dark Site” has its own independent VCF instance, separate from the primary VCF Fleet. A dark site has limited or no external network connectivity. An independent VCF instance ensures full functionality and management within the isolated environment without relying on external services. Okay, cool, so what do I do with it now? Well, my plan is to deploy a VCF9 instance on a single NUC 14 with an Ultra 7 CPU. I want to run a VKS Supervisor and deploy VM workloads on the dark site using automation. The cool thing is that Dark Site Edge can also be used without the fleet. So I have a VCF instance with VKS, NSX, and vCenter in a case measuring approximately 0.7 liters, or for our friends who don’t use the metric system, with the volume of ¾ of a standard coffee cup. (The volume specifications come from an AI; unfortunately, I can’t work with the non-metric system.)

Prerequisites

In order to set everything up, I first need my NUC with memory tiering. This will initially be loaded with ESX 9.0.1. If you want to know which workarounds are needed for an Intel NUC, you can check here.

Furthermore, an existing fleet is required. Even if the VCF instance works without a permanent connection to the fleet, the fleet is still required for licensing, and therefore the instance must also be in contact at least every 180 days.

In addition, an offline repository is required, as the software must be obtained from somewhere. In our scenario, we assume that I do not have a permanent or stable connection to the fleet.

Design

In terms of design, the setup is intended to complement my VCF 9 setup as described here.

• Local Management Domain:

  VCF Instance 2 includes its own Management Domain components (VCF Operation Collector, SDDC Manager, vCenter and NSX). Essential for the local management of the compute, network, and storage resources within the dark site’s VCF instance. It provides the necessary control plane for the dark site’s infrastructure.

• Local Compute Cluster:

  The VCF instance is set up as a single node cluster and thus shares resources for management and workload.

• VCF Operations Collector:

  Data collected locally during offline mode and synchronized with VCF operations, when connectivity is established with central site. here might be periodically exported or analyzed locally.

Usecase

• Strict Isolation and Security Requirements
• Unreliable or Intermittent Network Connectivity
• Autonomous Operations Requirement - Situations where the edge location needs to function independently for extended periods without external dependencies, including management, monitoring, and software updates.

At least, those are Broadcom’s official best-fit scenarios. Personally, I have other use case. Since the dark site comes with its own NSX instance, I can also test things in NSX without turning on my MS-A2, saving around 150 watts.

Okayyyyyyyy, let’s go!

The whole thing can’t be that difficult, I thought, and naively set to work, and what can I say—I had no idea what wouldn’t work and where I would have to improvise.

William Lam has an article online where he writes about how to deploy VCF with a single host. I tried for several hours and failed repeatedly. Maybe I’m too stupid, maybe I overlooked something, but in the end it doesn’t matter, because we’re in unsupported territory here. Slight foreshadowing—this won’t be the last time a plan doesn’t work out. At this point, I’ll count the workarounds so I can get the setup to work.

Workaround 1 and 2:

Activating E/P cores and memory tiering. These workarounds are well described and I use them all the time. You can find a description of the workaround in the introduction to the article.

Workaround 3:

Since my NUC only has only one 2.5 Gb/s network connection and the VCF installer actually requires 2x10Gb/s networking, workaround 3 is used. The VCF Installer checks in the pre-checks whether suitable network adapters are configured; fortunately, this can be disabled.

To do this, log in to the VCF Installer via SSH, obtain root privileges with su, and then simply add the following to /etc/vmware/vcf/domainmanager/application.properties

echo "enable.speed.of.physical.nics.validation=false" >> /etc/vmware/vcf/domainmanager/application.properties

Next, restart the service again and that’s it.

echo 'y' | /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Workaround 3.5

This is not a software workaround but a hardware workaround/ Since I use NFS storage, the VCF installer attempts to mount the NFS share during the precheck. Even if you have configured the VDS in the JSON to use a single NIC, this precheck is always performed with the second NIC. If the server does not have a second NIC, this will always fail. I used an old 1 Gb/s USB network card for this. It has an Intel chip and was recognized without any problems. However, the adapter does not support jumbo frames, and as soon as more than one VLAN is configured, the adapter no longer works. But that’s enough for the precheck.

I just had to assign the USB adapter as an uplink to the temporarily created VDS on the ESX server at the right moment, and the NFS check was successful. The actual deployment ran via the onboard card.

I will realize later that all the trouble with the USB NIC was actually unnecessary. Like many other things, but I want to write down the entire journey here.

Workaround 4:

My first attempt was a greenfield deployment as a single host using the workaround described by William. The interactive GUI of the VCF installer does not allow greenfield deployment with a single host. This can be suppressed with two simple entries.

For VCF 9.0.1, the feature properties file must be modified on the VCF installer using the following command

echo "feature.vcf.vgl-29121.single.host.domain=true" >> /home/vcf/feature.properties

For VCF 9.0, the feature properties file must be modified on the VCF installer using the following command

echo "feature.vcf.internal.single.host.domain=true" >> /home/vcf/feature.properties

Finally, the services on the installer must be restarted.

echo 'y' | /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

The workaround does not allow the interactive installer to be used; it only allows a single cluster to be deployed via a JSON file. The deployment looked fine at first, but then stalled after deploying the SDDC Manager.

I found this lovely error message in the VCF Installer log.

"Failed to validate domain spec", "nestedErrors":
K"errorCode": "INVALID_NUMBER_OF_MINIMUM_HOSTS", "arguments":
['2", "vLCM"], message':"Minimum 2 hosts are required for vLCM cluster with external storage to be created."

Since it didn’t work after several attempts, I needed a new idea. What if I built a 2-node cluster? The problem was that I didn’t have a second NUC available, and everything was supposed to run on one NUC. Then I had the idea of building a nested host (on one of my MS-01s), deploying VCF, and then turning off the nested host. Nice idea, but it leads me to

Workaround 5:

Since VCF does not allow you to create a cluster when different manufacturers are involved (in my case, ASUTEK NUC and Minisforum MS-01), I had to find a way around this. Fortunately, it is possible to rewrite SMBIOS hardware strings. I found the information I needed here on William’s website. I must say that I have visited his site many times over the last few days.

First, you must set an ESX kernel setting so that the default hardware SMBIOS information can be overwritten. To do this, edit the file /bootbank/boot.cfg and add the following to the line with kernelopt:

kernelopt=autoPartition=FALSE ignoreHwSMBIOSInfo=TRUE

After that, the server must be rebooted.

Next, I use Williams Powershell Script to generate a vsish command that I can use to rewrite my ESXiSMBIOS.

Function Generate-CustomESXiSMBIOS {
    param(
        [Parameter(Mandatory=$true)][String]$Uuid,
        [Parameter(Mandatory=$true)][String]$Model,
        [Parameter(Mandatory=$true)][String]$Vendor,
        [Parameter(Mandatory=$true)][String]$Serial,
        [Parameter(Mandatory=$true)][String]$SKU,
        [Parameter(Mandatory=$true)][String]$Family
    )

    $guid = [Guid]$Uuid

    $guidBytes = $guid.ToByteArray()
    $decimalPairs = foreach ($byte in $guidBytes) {
        "{0:D2}" -f $byte
    }
    $uuidPairs = $decimalPairs -join ', '

    

    Write-Host -ForegroundColor Yellow "`nvsish -e set /hardware/bios/dmiInfo {\`"${Model}\`", \`"${Vendor}\`", \`"${Serial}\`", [${uuidPairs}], \`"1.0.0\`", 6, \`"SKU=${SKU}\`", \`"${Family}\`"}`n"
}

Generate-CustomESXiSMBIOS -Uuid "43f32ef6-a3a8-44cb-9137-31cb4c6c520b" -Model "SDN-Warrior 3000" -Vendor "SDN-Warrior" -Serial "0816" -SKU "NUC" -Family "sdn-warrior.org"

The UUID and serial number should be unique. As a result, I now receive a string that I can simply send to the ESX server via SSH

vsish -e set /hardware/bios/dmiInfo {\"SDN-Warrior 3000\", \"SDN-Warrior\", \"0815\", [246, 46, 243, 67, 168, 163, 203, 68, 145, 55, 49, 203, 76, 108, 82, 10], \"1.0.0\", 6, \"SKU=NUC\", \"sdn-warrior.org\"}

/etc/init.d/hostd restart

However, I have set the option to permanent, and for this to work, you simply need to insert the two lines into /etc/rc.local.d/local.sh. After that, ESXiSMBBios will be set after every reboot.

Congratulations, I just became my own hardware manufacturer. May I introduce the SDN Warrior 3000!

But now the deployment should work, right? - NOPE!

I hadn’t thought about the network. VCF actually requires two network adapters for good reason. The initial setup of ESX is with a standard switch and, after the SDDC has been deployed and the cluster has been formed, it is migrated to a VDS that is initially formed with the second network card. Of course, this is not possible without a second network card. That was also the moment when I decided against a greenfield deployment and switched to Converge.

Redesign and Converge

Actually, I’ve long since reached the point where the project should be considered a failure, but I really wanted a VCF cluster on a NUC. To make that work, however, I have to migrate to a VDS, otherwise Converge will fail. First, I manually deployed a vCenter and created my vSphere cluster with my two SDN Warrior 3000 servers (a NUC and a nested host—a dream team).

As far as migration to the VDS is concerned, it’s a chicken-and-egg problem. The VDS does offer the option of migrating all VMK adapters and the VM network at once, but as soon as I do that, there is a network interruption and shortly afterwards the vSphere protection mechanism kicks in and rolls back the configuration. Fortunately, you can disable this, which brings me to

Workaround 6

To disable this behavior, you can disable the function in vCenter under Advanced Settings. To do this, set config.vpxd.network.rollback to false.

Half of the work is done. At least I could now migrate the VMK to the VDS, just like the VMK for NFS. However, my vCenter is then suspended in a vacuum on the standard switch and can no longer be connected to another port group. To make this possible directly via the ESX server, you have to change the port group where the vCenter is to be migrated from static binding to ephemeral. This allows the vCenter to be migrated directly via the ESX to the port group on the DVS and is then accessible again. Wonderful, the migration to the DVS with an adapter was successful. Perhaps just a quick note: the distributed port group should be reconfigured before the vCenter becomes unavailable.

Now the Converge could begin. Since I now had a 2-node cluster with VDS, vMotion, and so on, the Converge could begin without any further hacks via the VCF installer, right? Well, not quite. There was still a small error, but it was easy to fix. The installer checks whether the VDS has 2 uplinks, but it does not check whether they actually have a link. So the VDS is quickly given an uplink and lo and behold, the validation runs successfully.

Converge

The actual Converge process is relatively unspectacular. Broadcom and VMware have done a lot to make it as easy as possible (unless you have an adventurous setup like mine). You no longer need Python scripts, as was the case with VCF 5.x for brownfield imports. The process is relatively simple: the VCF installer first deploys the SDDC, then converts the existing vCenter to the new VCF instance, rolls out NSX and a cloud proxy, and then joins the instance to Fleet and VCF Operations. It can be that simple.

Are you lying to us?

One might ask this question, because the plan was to build a single node cluster, and when it comes to that, I can only say mission successfully failed, but there is a but. The virtual ESX is only a temporary solution, and in future I will only activate it for the lifecycle, as the NUC’s performance is more than sufficient. I also simply removed the ESX server from the cluster and shut it down.

So, technically speaking, I cheated a little bit, but as you can see in the screenshot, I deployed a working VCF instance. What’s still missing are the NSX Edge and my supervisor cluster, but looking at the clock, that’s a topic for another blog article.

just one more thing

To ensure that I still have logs and metrics from my cluster in operation even when the cluster is disconnected or my operations are down, I have enabled data persistence on the cloud proxy. This means that logs are stored until the environment reaches my fleet again.

Final remarks and summary

It was an exciting and thrilling project that definitely cost me a lot of sleep, as I worked tirelessly for the last three days and definitely didn’t get enough sleep. However, I couldn’t accept that it wasn’t possible or that I couldn’t do it. It also gave me back the motivation I had been lacking to write something again. I must also say that I learned a lot. And as always, I have to express my gratitude to William, because at times I was on the verge of despair.

I hope you enjoyed the article and maybe it will help some of you, or you can laugh at me and my journey through the valley of tears.

In the past, I built all my labs in a nested structure, which worked well, but now the demands on me and my lab are increasing because I want more. I want to try more, even in areas I previously neglected. The idea of a redesign took shape in my head, and I knew there had to be more possibilities with my hardware. Above all, I wanted to work with VCF Automation, because it is essential for multi-tenancy environments and replaces vCloud Director. And let’s be honest, automation is a resource hog.

The idea was quite simple: I would take my two MS-A2 servers and install a VCF management domain on them. Both servers have 128 GB of RAM—that should be enough, right? Unfortunately not really, but there is still memory tiering. And that could be the end of the blog article—could be, if it weren’t for AMD Ryzen. Fortunately, there are workarounds, and thanks to William Lam, I found them.

But let’s take it slowly, one step at a time.

Hardware Setup

My plan is to build a two-node cluster without vSAN and with memory tiering. NFS will be provided via my good old Unraid storage system. I also started directly with VCF 9.0.1 as a greenfield deployment. I will continue to build my workload domains nested. This allows me to set up a complete VCF9 setup with three physical hosts, including VCF Operations for Network, Operations for Logs, and Automation.

First, I install a 1 Tb Samsung 990 Pro in the first slot of each MS-A2. The first slot in the MS-A2 is set to PCIe4x4 in the BIOS by default. The other slots are set to PCIe3x4 by default. You can change this in the BIOS, but you risk the NVMes overheating. The active cooler does not seem to be sufficient to cool all NVMes. Here is an excerpt from the official FAQ:

To prevent the SSD from overheating, the two SSDs on the right side of the back support a 
maximum speed of PCIE4.0x4, but are set to PCIE3.0x4 by default. You can adjust it to PCIe Gen4 
speed in OnBoard Setting in BIOS, but it may cause the SSD to overheat, which may result in blue 
screen, freeze, or data loss.

In the middle slot, I have a 2 TB NVMe for booting and as local storage. The third slot is free in case I ever want to use vSAN.

Memory Tiering and first steps after setup ESX

For VCF, SSH and NTP must be enabled after setting up the ESX server. It is also important that only the first 10Gb/s adapter is used. The second must be left unconfigured, as VCF deployment involves migrating from a standard switch to a distributed switch. Also, don’t forget to recreate the TLS certificate, otherwise the VCF installer will throw an error when adding the hosts.

• Cert regeneration

/sbin/generate-certificates

To enable memory tiering, the following must now be entered in the ESX CLI.

• This command turns on memory tiering

esxcli system settings kernel set -s MemoryTiering -v TRUE

• This command selects the NVMe

esxcli system tierdevice create -d /vmfs/devices/disks/<Your NVME>

• Enter the factor here (0-400%)

esxcli system settings advanced set -o /Mem/TierNvmePct -i 200

I use a factor of 200% because I need enough reserve to be able to move everything to one host for the entire lifecycle. But what’s the problem? The commands are well known, and I already described them in an article from 2024.

Ryzen Workaroud No1.

The problem lies with the AMD Ryzen processor and does not affect AMD EPYC or Intel Core, Ultra, or Xeon CPUs. To be fair, it must be said that Ryzen is not on the compatibility list and that VMware Homelabs has often insisted on NUC in the past, with many VMware employees themselves building labs with Intel NUCs. Whether this is the reason why Intel NUCs work so well is something I can only speculate about. But what exactly are the problems?

Well, the error was a bit strange. The VCF installer first installed vCenter and everything ran smoothly so far. The first problems arose with the SDDC Manager, which was deployed after vCenter. It simply refused to boot up. The VM was successfully turned on, and you could see the Photon OS splash screen, but after that, everything remained dark. My first suspicion was my NFS, as I had had a lot of problems with the reliability of my NFS shares after an Unraid update before my vacation. However, that turned out to be a dead end.

I then tried to install the VCF Installer (aka SDDC Manager) manually on local storage and got the same result. The same thing happened with NSX Manager. It boots a little further than SDDC, but also stops quite early on with no error message.

After a long search on Google, I found the solution at William’s blog.

For memory tiering to work properly with AMD Ryzen, a VM advanced setting must be configured. Of course, this is not scalable if it has to be set for each individual VM, and it also does not work with the automatic deployment process of VCF. Fortunately, it can also be set globally. To do this, you must log in to each ESX host via SSH and execute the following command:

echo 'monitor_control.disable_apichv ="TRUE"' >> /etc/vmware/config

The workaround takes effect immediately and the ESX server does not need to be rebooted. If VMs were running, they must first be shut down for the workaround to take effect for these VMs.

Ryzen Workaroud No2 (NSX Edge CPU Check).

The next workaround concerns the problem with the Data Plane Development Kit (DPDK) and the lack of support for Ryzen CPUs. I already described how to solve this problem manually in my MS-A2 test.

However, William has a better solution here too, and I am using his Powershell script, which he has kindly made publicly available. Nothing has changed in terms of the actual fix; you simply comment out the CPU check and the Edge VM will then work as it should. I also didn’t notice any performance issues. I get 10 Gb/s north-south over my edges without any problems.

Connect-VIServer -Server vc01.vcf.lab -User administrator@vsphere.local -Password VMware1!VMware1!

$edges = @("edge01a","edge01b")
$edgeUser = "root"
$edgePass = "VMware1!VMware1!"

### DO NOT EDIT BEYOND HEREx

$edgeScript = "sed -i `'/if `"AMD`" in vendor_info and `"AMD EPYC`" not in model_name:/s/^/        #/;/self.error_exit(`"Unsupported CPU: %s`" % model_name)/s/^/        #/`' /opt/vmware/nsx-edge/bin/config.py"

foreach ($edge in $edges) {
    Invoke-VMScript -VM (Get-VM $edge) -ScriptText $edgeScript  -GuestUser $edgeUser -GuestPassword $edgePass
}

Disconnect-VIServer * -Confirm:$false

The script is part of Williams Script Collection, which I highly recommend to everyone.

New Ryzen CPU NSX Workaround (NSX Edge CPU Check)

The improved workaround is both simple and sustainable. Instead of deactivating the CPU check in the Edge every time during deployment, you simply trick the system into thinking it has a suitable CPU. May I introduce the EPIC Ryzen processor. To do this, you must log in to each ESX server once as root and enter the following command.

echo 'cpuid.brandstring = "AMD EPYC Ryzen 9 9955HX"' >> /etc/vmware/config

A reboot of the ESX server is not necessary, but the Edge VM must undergo a full power cycle once. Edge VMs deployed in the future will then recognize the EPIC Ryzen CPU directly, thus bypassing the CPU check in the EdgeVM script. Simple and update-safe.

VCF Setup

Now we come to the exciting part. After all the workarounds have been implemented and I was able to successfully complete the deployment, the question remains as to what exactly my design looks like now. Because I think that’s what most readers are interested in.

I currently have three physical hosts in use for my full-blown VCF9 setup. Two MS-A2s form my management domain and one MS-01 is a standalone host (without memory tiering, but with P/E cores enabled – the workaround can be found here and has nothing directly to do with the MS-A2).

Why do I need the standalone host? Quite simply, I have two nested ESX servers running on it, which form my workload domain. Since the workload domain requires significantly fewer resources (in a lab), I can easily map it with an MS-01.

Since pictures say more than a thousand words, I have created a design drawing here. The whole thing is based on Broadcom’s official design blueprints.

And here are the deployed components in the Managment Domain and the required resources:

CPU resources

Wow, that’s a lot of appliances and a lot of power required. First, the good news: when I turn off the automation appliance, the entire setup runs smoothly and quickly on an MS-A2 without any problems.

With automation turned on, the vCenter times out and many components no longer work properly. However, without test VMs in my management domain, I have an overbooking factor of 2.4 to 1, and the actual bottleneck is neither RAM nor storage, but actually the CPU.

The recommendation is to overbook a management domain by a maximum of 2 to 1. If I move the management domain to a host, I have an overbooking factor of 4.75 to 1 with automation and 3.75 to 1 without automation. So it’s not surprising that a host only works properly without automation.

Storage resources

As usual, I use an NFS 3 share from my self-built Unraid NAS, which is connected with 2x10Gb/s, for storage. All VMs are thin deployed, and I have provided 2 shares with 4 TB NVMe storage each. My VMs from the management domain go into the first share, and my VMs from the workload domain go into the second share.

Since my Unraid also has a 2x10 Gb/s network and I have 2 VLANs for NFS, both the management domain and the workload domain can access the full 10 Gb/s network performance, as they each have a different physical path to the storage and therefore cannot slow each other down.

The actual storage usage of less than one terabyte is actually not that much. In addition, I would also have a third 4 TB share that I could connect if space becomes scarce in the future.

RAM resources

My MS-A2s are each equipped with a 128 GB DDR5 kit. I also selected a factor of 200% for memory tiering. This means that each ESX server has a theoretical 384 GB of RAM. However, I am somewhat surprised that no Tier 1 memory (memory tiering) is currently being used in my management domain.

Let’s take a closer look. If you look in vCenter, you can see that I currently have an approximate Tier 0 RAM allocation of 188 GB. A large part of this comes from automation. But even with automation, I still have physical Tier 0 RAM left over, which is why you can see in the screenshot that Tier 1 RAM is listed as -1 MB everywhere.

Without the operation appliance, I have a RAM utilization of 130 GB, so I don’t really need the factor of 200%. I chose it anyway because, among other things, SSP is to be deployed in the future and SSP requires 2x16 vCPUs and 64 GB RAM. So it’s another little resource hog and I have more leeway to run everything on one host. You also have to think about the future lifecycle. In addition, another nested workload domain is to be added, which will also bring with it another vCenter.

VCF 9.0.1 NSX Edge Workaround

My colleague Christian and I encountered a spontaneous error. If, like him or me, you don’t have the lab running 24/7, it can happen that the uplink port groups are deleted from the Edge VM. This should only happen if the environment is on and the port groups are 24 hours without a connected Edge VM.

What is the result? Well, a huge mountain of error messages. The Edge VMs no longer have an uplink or a TEP network, meaning that all north-south communication is dead.

But first, let’s take a step back. In VCF 9, a separate trunk port group is created for each Fastpath interface of the Edge VM. With two edges, each with two uplinks, you therefore have four port groups.

If you take a closer look at the port groups, you will see that they are completely normal trunk port groups. To fix this error, you simply need to create two new trunk port groups (note the teaming policy—trunk 1 uplink 1 active/uplink 2 standby and vice versa for trunk 2). These port groups must then be assigned as Adapter 1 and Adapter 2 in the vCenter of the Edge VM. Apparently, only Edge VMs are affected by the error if they were created via the vCenter.

Conclusion

With the new setup, I am prepared for all eventualities. I have enough space in my physical lab to create additional nested labs, and I have a complete solution lab where I can test all aspects of VCF 9. Best of all, I only need three physical hosts, which means I can get by with about 500 watts of power consumption. My always-on equipment requires about 200 watts, bringing the total power consumption of the VCF 9 setup to 300 watts. Not bad at all, to be honest. So, in total, I have 5 more MS-01s that I can use to test things out alongside VCF 9.

I have written a lot about the actual lab setup in the past, but it was always about the hardware and not how I manage my labs or, more recently, how I have built up my vcf fleet. I want to change that here now, as I have received an increasing number of requests for more information.

So let’s jump right in. First things first, let’s take a look at the physical lab infrastructure. That’s my basis for everything else.

Physical lab infrastructure

I currently own eight Minisforum servers and one NUC Ultra 7. The Minisforum servers are for workloads and are divided into two clusters, as some of the hardware has AMD CPUs and the other servers have Intel CPUs. My NUC is its own cluster and runs my always-on workload, as the Minisforum servers generate too much power and heat for 24/7 operation.

As you can see in the image, two Minisforum servers are currently not part of my cluster for licensing reasons. This will change in the coming weeks, but Proxmox is currently running on them for testing purposes.

Management NUC

This setup is the foundation for my entire lab. Most of my fleet runs on my small NUC, including a VKS Supervisor cluster as a single deployment. Important services provided by the NUC are:

• Active Directory (Microsoft)
• Veeam Backup Server
• VCF Operations
• VCF Operations for Logs
• VCF Cloud Proxy
• VCF Fleetmanager
• Fortigate Logserver (for my Fortigate 40F)
• VKS Supervisor with Foundation Loadbalancer
• VCF Offline Repository
• vCenter for the Lab
• Netbox
• mDNS Repeater (It’s not relevant for the lab, but it is for my number one service owner, because without it, you can’t turn on the TV with Siri and watch Netflix.)
• Windows 11 Client (There are things I can’t do with my Mac.)

Of course, the NUC is pretty busy, and I’m considering getting a second one, because the boxes don’t cost that much (used) and don’t take up much space. Bonus points for the fact that every NUC is certified for 24/7 operation.

Alternatively, I could use memory tiering, but that would require reinstalling the little guy, and somehow I’m too lazy to do that. My self-built storage (which I’ve described here before) runs another domain controller, my root CA and my Ansible host.
I obtain NTP globally from my ToR switch, as Mikrotik, the old Swiss Army knife, can do this wonderfully.

Storage

Let’s move on to the exciting topic of storage. Here, I have a combination of local and shared storage. Each server has a 2 TB local NVMe installed on which the hypervisor itself is located, and the rest of the NVMe is provided as VMFS storage. On my Unraid server, I provide 4 TB as iSCSI storage and 2x 4 TB as NFS storage. The storage on Unraid is implemented on NVMes, but without Raid. If a disk fails, I just lose a share.

Now you might ask yourself why I did it this way. Well, iSCSI has the advantage that I can implement a multi-path fabric and is basically my fastest storage, since I have 2x 10 Gb/s iSCSI and can utilize it to its full capacity. All my servers are equipped with 2x10 Gb/s networks (except for my small management NUC). My iSCSI storage runs VMs that require maximum storage performance. I use NFS for VCF nested labs where I don’t want to use vSAN, and most of the time I don’t want to use vSAN in my lab for resource reasons. Fortunately, this is now supported without any problems with VCF9. But why did I build two NFS shares and not just one large share?

Well, that’s because my storage only supports NFS3 (NFS 4 keeps causing problems with Unraid, and VCF 9 requires NFS3 anyway if you want to use it as primary storage), which means multipathing is not possible. For maximum performance, the first NFS share uses the first physical adapter of my storage system and the second NFS share uses the second physical adapter. This means that the two shares do not interfere with each other. In addition, the PCIe bus to which the NVMes are connected does not exceed 10 Gb/s anyway.

If I want to use VCF with vSAN, I use the local storage of my hypervisors, as I have had problems with vSAN and nested ESXi on my shared storage from time to time.

DRS and Network

Let’s move on to my DRS and network settings. This is relatively simple. My DRS is set to fully automated but in conservative mode, as I don’t want any “unnecessary” DRS actions from nested ESX servers. As a rule, I think about where to place my VMs beforehand and then they should stay there.

Since I don’t always test nested VCF labs in my lab, DRS is generally still enabled, as there are scenarios where it is checked whether DRS is enabled or not. For example, with VKS.

My network settings are currently still under construction. At the moment, I have a distributed switch for all three clusters, but last year showed that this is not the best setup when hosts are frequently down. My new plan is to have a separate vDS for each cluster. This is more flexible and I don’t constantly have the problem that my vDS is no longer in sync when I need a new network on my management NUC.

My port groups are usually active/active on both uplinks. There are three exceptions: my trunk port groups with MacLearning. Here, I have a trunk that only uses uplink 1 of the vDS and a trunk port group that uses uplink 2 of the vDS. This is important for NSX Labs if you want to test traffic steering of the NSX Teaming Policy. Another exception is my port group on which promiscuous mode is enabled. Here, I use active/standby, as otherwise I would have duplicated packets and performance losses.

If you want to know more about this topic, you can read my article on Mac Learning. Here, I address the problem and also show the differences in performance. For VCF 9 installation, promiscuous mode is required, otherwise the deployment will fail when the VCF installer attempts to migrate the ESX servers to the distributed switch.

VCF Fleet

My fleet setup is a bit of a hodgepodge. Since I need VCF Operations for licensing, this is already running on my management NUC, as mentioned, but my VCF Automation is on an MS-A2 as a VM and is only turned on when I really need it, as it is a real resource hog. In addition, a VKS cluster has recently been running for testing purposes. I want to use automation to automatically deploy VMs that are running in the vsphere Namespace - fancy new things.

I have several VCF instances onboarded in my central VCF Operations. Since I only have a certain number of licenses, I delete Labs in VCF Operations that are not currently needed. This is a bit annoying, but with the current licensing model, there is unfortunately no other way to do this.

VCF Operations for Network is currently no longer deployed because my vDS kept getting out of sync, which caused problems with IPFIX. I will address this issue when I rebuild the vDS. After that, I will try again to get netflow running on my Mikrotik so that I can get useful network insights. However, this is still a work in progress.

Final thoughts

My setup isn’t particularly complicated, and everything is designed so that I can save resources and minimize 24/7 use without sacrificing too much comfort—anyone who has ever looked at the procedure for shutting down a VKS-enabled cluster knows what I mean. But not everything is perfect, and there is certainly room for improvement. My Ansible playbooks need to be revised because they no longer run properly with the latest version. In the future, I would also like to deploy much more automatically, i.e., I would prefer to deploy my standard VCF deployments fully automatically.

My monitoring with uptime Kuma with Discord bot and notifications is a good start, but it was never really finished. I’m also not quite sure where I want to go with my VCF automation setup, but I think the next few weeks and months will show. If you have any suggestions, questions, or improvements, please feel free to contact me. Homelab thrives on exchange.

I have been asked several times on LinkedIn for an article on the topic of brownfield upgrades from VCF 5 to VCF 9. Well, here we are. I thought it would be relatively easy and that you could just click your way through, but I was wrong. But let’s start at the beginning.

As things stand at the moment, if you have NSX version newer then 4.2.1.3 in your VCF environment, here’s the bad news—it’s not possible as of today. NSX 4.2.1.4 and newer has a newer code base than NSX 9.0.0.0 and is therefore not compatible. The problem should be fixed with VCF 9.0.1.

It is best to always check the Product Interoperability Matrix for this.

Since my last available VCF 5 lab was unfortunately already patched to the latest NSX version, I had no choice but to deploy a fresh VCF 5.2.1, as the NSX version here is low enough. And oh my God, I really didn’t miss the Excel or Json deployment of VCF 5.X, but for my readers’ sake, I’ll struggle through it once again.

Management domain and basic setup

I have deployed a management domain with 4 ESXi servers and vSAN. I have set up the whole thing in a nested configuration. Since the SDDC Manager does not yet support download tokens, I used the VMwareDepotChange script. I have already covered this process in my blog.

After the initial deployment, I updated the SDDC to version 5.2.1.1, as it contained several bug fixes. Finally, I took a snapshot of the SDDC Manager, and we were ready to go.

Uprade SDDC

In my setup, the first step is to upgrade the SDDC Manager. As I mentioned above, after the upgrade, the setup will be added to my VCF Operations, which is already up to date.

The SDDC is booted once during the upgrade. I had a minor problem with the SDDC in that it no longer displayed the upgrade status after rebooting. Instead, I only saw update status 0. I checked the log file /var/log/vmware/capengine/cap-update/workflow.log via SSH on the SDDC manager and looked to see if the following was visible:

YYYY-MM-DDTHH:MM:SS.426528 workflow_manager.go:221: Task stage-cleanup completed
YYYY-MM-DDTHH:MM:SS.426646 workflow_manager.go:183: All tasks finished for workflow
YYYY-MM-DDTHH:MM:SS.426699 workflow_manager.go:354: Updating instance status to Completed

After that, I rebooted the SDDC again and the error was fixed. I had never had this problem before and couldn’t find any KB articles about it. Maybe my nested environment was too slow. So take this fix with a grain of salt. Finally, I configured my VCF9 offline repo in the SDDC as a repository, as I don’t want to download all the VCF9 binaries from the network.

Binary Management

SDDC version 9 supports the Broadcom download token, so there is no need to change anything with a script on the SDDC, but I prefer my central offline repo. I built my own repo with the help of Timo Sugliani’s instructions. It is the easiest way to get your own offline repo, even if you have no Linux knowledge. However, you still need a valid download token.

Next, I download the upgrade binaries for NSX, vCenter, and ESX to my SDDC manager. The download is faster than the validation. It took me about 30 minutes to get everything I needed.

Starting the upgrade

I start the upgrade with the SDDC for the management domain. To do this, go to Workload Domains —> Management Domain —> Updates in the SDDC manager and then click on the Plan Upgrades button. The process is actually exactly the same as for a normal VCF update.

I select VCF version 9.0.0.0 as the target version and confirm the version summary.

This completes the upgrade plan. It is important to note that the SDDC will only start the upgrade once we have clicked Configure Update. The order of the components is as follows:

• NSX
• vCenter
• ESX hosts
• vSAN

You can only configure the update for one component at a time. Pre-checks must be performed before each update. Here, too, the process is no different from a normal VCF update.

NSX

Since I don’t have any edges deployed, the NSX upgrade runs without any problems. If I had edge VMs in my environment, I would have had to intervene because of the Ryzen processors, otherwise the edge upgrade would have failed. I explained why this is the case in my article on MS-A2.

vCenter

Next up is the vCenter. Here, the vCenter Reduced Downtime Update is used. This means that a second vCenter appliance is deployed in parallel and assigned a temporary IP address. There are two options here: either I assign a static temporary IP address or I use Auto.

I usually use IP addresses defined by my customers, as firewalls are usually involved in whatever form. Since I am working here in my lab without a firewall within the lab, I can use auto. This gives the old and new vCenter a temporary APIPA IP to communicate with each other. For the upgrade scheduler, I use Immediate and Automatic for the switchover.

ESX - Houston, we have a problem!

The update ran successfully, but I have the problem that my vCenter simply does not want to display ESX9 images, which is essential because without ESX9 images, you cannot upgrade the ESX servers to 9. I thought I was being clever and forced the vCenter to search for new images. You can do this using the vCenter Lifecycle Manager and push Sync Updates.

Interestingly, the process is not running correctly. I hadn’t encountered this before, and after a quick check in the task log in vCenter, I saw the following error message, which wasn’t really helpful.

A general system error occurred: Down load patch definitions task failed while synci ng depots. 
Error: 'integrity.fault.Vclntegrityf ault: VMware Sphere Lifecycle Manager hi d an unknown error. 
Check the events and I og files for details.

Of course, there is nothing specific in the logs either. So Google research…Because the second most important skill in my IT career is knowing how to Google properly—maybe it’s even my most important skill.

Turns out it’s a known bug in vCenter 9 when performing a brownfield upgrade from 8 to 9. VMware has created a KB article on this topic. The solution is relatively simple: you need to delete metadata from the vCenter DB in the tables vci_updates and vci_update_packages.

vCenter workaround

• Lgin to vCenter as Root
• Stop updatemgr service

service-control --stop updatemgr

• Change to ‘updatemgr’ user.

su updatemgr -s /bin/bash

• Run psql command to patch the DB

psql -U vumuser -d VCDB -c "UPDATE vci_updates set deleted = 0, hidden = 0 where meta_uid like 'ESXi7%' or meta_uid like 'esxi7%' or meta_uid like 'ESXi_7%' or meta_uid like 'esxi_7%'; DELETE FROM vci_updates where id not in (select update_id from vci_update_packages);"

• Start updatemgr service

service-control --start updatemgr

After the changes and the restart of the service, vCenter automatically syncs the image data. Now a new image can be created.

ESX Image

For Image Mode, we still need to import an image into the SDDC; without this, the ESX servers cannot be upgraded to 9. After fixing the vCenter, I create a dummy cluster in the vCenter and select the correct image for VCF9.

The image is then imported into the SDDC and the dummy cluster can be deleted again in vCenter. The image is assigned to the management domain cluster in the “Configure Update” step, and after the pre-checks, the ESX servers can be updated. The process ran smoothly - easy, that’s what I expected from the start.

vSAN

The final step in the actual upgrade process is to update the vSAN storage. The process is straightforward, and all you need to do is press the upgrade button under cluster –> configure –> vSAN. Depending on the hardware and size of the vSAN storage, this process can take a relatively long time.

Now everything is ready for me to start onboarding in Operations.

Onboarding in VCF Operations

I have a central VCF Operations because I don’t want to deploy an entire fleet every time I need a new nested lab. I described how I built it in this article. When onboarding in Operations, there are really only two things to keep in mind. I have to configure the new VCF environment as a deployment target, and I have to enable activate management, otherwise I cannot distribute licenses for the environment with my VCF Operations instance.

To onboard, I need to add the VCF instance to my Ops via VCF Operations —> Administration —> Integrations —> Accounts —> Vmware Cloud Foundation. Once the environment appears under Integrations, you can select it and press the Activate Management button. This allows you to assign a license to the VCF instance.

Finally, the VCF instance is set up as an deployment target under —> Lifecycle —> VCF Management —> Settings —> Deployment Targets.

The upgrade is now complete and the VCF 9 instance can be used as normal. Optionally, you can install the ESX patch under Lifecycle. To do this, you must create a new image in vCenter and import it via Ops. The process is more or less identical to the previous update.

Final thoughts

The upgrade was more difficult than expected. The missing images in particular took me longer than I would like to admit. To be honest, I thought it would be a straightforward process. My VCF 5.2.1 environment was completely new, with no previous updates or other legacy issues.

I think VCF 9.0.1 will remedy this, because in theory the upgrade process is very simple and doesn’t really differ much from a normal patch update in VCF5.X.

What I still want to test is an import, i.e., converting a standalone NSX installation into a workload domain. Unfortunately, the same applies here as with my old NSX Labs: when updating my physical lab, I destroyed most of my old setups (because I’m sometimes just stupid). Only labs that are unusable in this case have survived.

Well, I’ll do another NSX 4.2.1 deployment at some point. But that will be another blog post. If you have any questions or comments, please feel free to send me a message on LinkedIn or to my blog email address.

At the beginning of the year, I had the pleasure of flying to the USA, and now I’m on my way to Las Vegas. After a slightly delayed departure, I arrived in Vegas after an 11.5-hour flight in a “comfortable” 40 degrees Celsius or 104 degrees Fahrenheit, for those who do not use the metric system. Perhaps one more small note. I won’t write much about the sessions; anyone who wants to can look them up in William Lam’s repository. I want to focus more on the vibe of the event here—that is, I’ll try to. I’m not really a travel blogger.

But before I talk or write about Explore itself, I’ll first write a little about Las Vegas. When you hear Las Vegas, you have a certain cliché in your head—at least I do. The city is loud, flashy, and never sleeps, and what can I say, it’s all true. The entertainment dial is turned up to 11—a sea of lights from flashing hotels, casinos, and some very obscure billboards, or even trucks that serve as rolling, illuminated billboards. As a German, this is a culture shock at first.

But if you prefer something a little quieter, I can give you two recommendations. These are certainly not insider tips, but then again, I’m not a travel blogger.

Hoover Dam and Mike O’Callaghan–Pat Tillman Memorial Bridge

Visiting Hoover Dam feels like standing in front of a monument that engineers built just to prove they could. In short, it’s a huge thing.

The concrete wall on the border between Nevada and Arizona was completed in 1936 and still supplies electricity to millions of households today. The Hoover Dam is so massive that it slows down the Earth’s rotation slightly when Lake Mead is full—only a tiny fraction of a second, but the idea alone makes me smile. When you’re at the dam, you can walk back and forth between two states and two time zones. That gives you an extra hour to drink beer—I mean coffee.

Right next to it is the Mike O’Callaghan–Pat Tillman Memorial Bridge, better known as the Hoover Dam Bypass Bridge. From here, you have a spectacular panoramic view of the dam—almost like something out of a postcard.

There is a museum at Hoover Dam, but I skipped it and therefore can’t really say anything about it. However, my colleagues who visited it thought it was good. I preferred to go directly to the Valley of Fire to have more time there.

Valley of Fire

The Valley of Fire is located about 80 km northeast of Las Vegas and can be reached in less than an hour by car. The park covers around 186 square kilometres, making it the oldest and largest state park in Nevada. It goes without saying, but it’s a desert and it’s hot, so be sure to take water with you. You can buy some at the visitor centre or fill up empty bottles for free. It was about 110 degrees Fahrenheit that day, which is around 43 degrees Celsius, which a taxi driver said was cool. To me, it felt like sticking my head in a hot air fryer. It may not be the best idea to drive to a desert in the summer, but the landscape makes up for it.

I don’t think I’ve ever sweated so much in my life, but it was a fantastic day. There are five signposted short walks, all between one and two miles long, which are manageable with sufficient sun cream and water. There is also a road through the park, allowing you to cruise through beautiful and striking scenery in a typical American car.

If that’s too low-tech for you, you can also visit the Sphere. It’s like an IMAX cinema on steroids, although that doesn’t really do it justice.

Sphere

Who doesn’t know it, the gigantic sphere that peeks shyly into one hotel room or another? The Sphere is approximately 112 metres high and 157 metres in diameter, a rather large sphere that uses LEDs on the outside to display dancing cats, smilies and other things 24/7. You can see it as you approach the airport.

We watched Postcard from Earth by Darren Aronofsky, and what can I say? Audiovisually, it was the most impressive experience I’ve ever had. It’s very difficult to describe, and even the YouTube videos you can find don’t do the experience justice. If you have the opportunity to visit the sphere, then do it. Highly recommended.

Explore 2025

But enough with the travel tips, let’s get started with the event. Since I’ve only been to European events so far, the first big difference to Barcelona is that Explore is located on several floors of a hotel. This makes the event a little more compact and not quite as sprawling as at the Fira Gran Via. Depending on which hotel you’re staying in, you might not even see daylight—if you want to avoid it.

In terms of organization, everything was once again tip-top. Easy check-in, easy to find your way around, friendly staff everywhere who helped you find your way. What immediately stands out is that it’s less busy than expected. I don’t have any figures, but the days of 10,000 visitors and more are long gone. However, with the absence of EUC, Carbon Black, and smaller partners, this was to be expected. From a visitor’s point of view, however, this is not necessarily a bad thing, as you could get into every session without any problems, even if you hadn’t registered in advance or the session was officially fully booked. There were no long queues anywhere, and there were always plenty of snacks, refreshments, and coffee available. I’ve had different experiences in Barcelona.

Day 1

Day 1 started for me with a session on edge computing. It was co-hosted by Audi. Edge computing is a really exciting topic, and my employer evoila will also be presenting it at our in-house exhibition. So it was fascinating to hear about the topic from a customer perspective and learn how Audi is using it. The possibility of building a compact single-host edge cluster with vSAN storage, NSX networking, and automation tools such as ArgoCD in a VCF context has a lot of potential.

After that, I attended a session with Tim Burkhard, who explained the network control plane. Since NSX is my area of expertise, there was nothing new for me here, but sessions with vGandalf are always entertaining and bring out the fanboy in me.

11/10 I would definitely attend again. Of course, a photo with him is a must.

I then went on to other sessions, including William Lam’s, but this time I didn’t manage to get a photo. Next time, then. In between, I attended the vExpert Community meeting, picked up my swag, and talked to lots of vExperts.

Community

And while we’re on the subject of community, I was overwhelmed by how many people stopped to say hello. I can hardly put it into words, but it was incredible. Thank you so much, it was amazing, and I am grateful to everyone who reads my LinkedIn posts or my blog. I’m still surprised when someone asks me if they can take a photo with me. Thank you for this incredible support.

Photo credits go to Sebastian Garcia. An absolutely friendly person. Thank you for the nice conversation.

Day one ended with the welcome reception, after which my colleagues and I hit the Vegas nightlife.

Day 2

The second day started with little sleep and slight jet lag at breakfast on the Explore. Although breakfast is a funny thing—I understand it to mean something else. This was more like lunch. After enough coffee, I headed to the general keynote, which had the somewhat unwieldy title Shaping the Future of Private Cloud and AI Innovation. What was special about the general keynote for me was that I was asked if I would like to contribute a video clip for Paul Turner’s part.

My contribution was shown alongside four other well-known vExperts, including William Lam, Melissa Palmer, Eric Sloof and Duncan Epping. Seeing my own face on the main stage in Las Vegas, alongside the other great vExperts, was a bit strange but also great at the same time.

I filmed my part from the audience and uploaded it to YouTube.

The complete keynote can currently only be viewed directly at VMware.

After this highlight, I withdrew to prepare myself mentally for the upcoming certification. With the full pass, you could complete various certifications on site, and I took the opportunity to take my VCP VCF9 Administrator exam, which I successfully completed. As a small bonus, everyone who successfully certified received a pair of sneakers.

In addition to the general keynote, the highlight for me was the session by Duncan Epping and Rakesh Radhakrishnan entitled Innovations Redefining Storage and Disaster Recovery for VMware Cloud Foundation 9. I found it particularly exciting that it will be possible to use vSAN over Fibre Channel in the future, which makes me, as an old storage guy, especially happy. It should be possible to connect a vSAN storage-only cluster to an existing FC fabric via FC, allowing vSAN and FC storage to be used simultaneously. This combines the best of both worlds and allows drop-in replacements to be implemented without making my FC fabric obsolete.

The day ended with the VMUG reception at Wakuda, where Hock Tan personally addressed the VMUG community. He promised VMUG support and emphasized how important VMUG is for VMware and Broadcom.

After that, I threw myself into the nightlife just like before. That’s a pattern and it will continue.

Day 3

This day started off rather leisurely for me. I didn’t have a session until 11, so I used the time to prepare this article and chat with a few vExperts. I also took a closer look at the expo and the community hub and had a few conversations there as well. I then attended the VMUG Leaders and VMware Champions Meet-up and tried in vain to take another exam. Unfortunately, there were no more slots available, and my place on the waiting list didn’t help either.

As part of VMware Champions, there will also be a session on VPCs with me. Exact details are not yet known.

Sadly, I missed the vExpert group photo while trying to get a spot for the exam. My colleague Steven Schramm and I then went to the Knight Welcome Reception.

Work hard, play hard. Afterwards, there was the Explore closing party on the pool deck of the Palazzo hotel with music and cool refreshments. Which was, of course, fantastic.

For most of the evoila team, this was also the last day of Explore. For Steven and me, things continued on Thursday with an exclusive full-day event for Broadcom Knights.

Knight Day

The day started at 8 a.m. with an update on the Knight program and what changes will be made. Spoiler alert: I have to complete a number of certifications. The rest of the day was spent looking at the roadmap and a condensed version of the last three days.

Last words

Since everything must come to an end, Friday was our departure day, and we spent our last dollars on food and boarded our flight back across the pond. What remains? A great event, wonderful conversations, new and old faces, and a lot of gratitude for everything. Thanks also to my employer, evoila, who sent 13 people to Las Vegas, making us the largest German group there. Oh, and we achieved Pinnacle status, which is also worth mentioning. I’m already looking forward to the upcoming Knight exclusive events and Explore on Tour in Germany.

In the last 25 years of working in IT, I have never met anyone who is happy when certificates expire and need to be replaced – quite the opposite, in fact. Certificates are like printers: everyone hates them, but we need them. Fortunately, VCF and VCF9 have greatly simplified the entire procedure. In this article, I would like to briefly explain how you can use VCF Operations and a Microsoft CA to exchange certificates easily and, in some cases, automatically – guaranteed pain-free. However, it is still not an enjoyable process.

Let’s get started

VCF (VCF9) currently supports two integrations of certificate authorities. It supports Microsoft CA or OpenSSL. Since I already have a Windows domain running, I simply use Microsoft CA for the sake of simplicity. Nevertheless, the completely manual method is also supported, which I will show at the end of this article.

I will use the Microsoft Certification Authority Web Enrollment Service. If you want to use fully automated certificate enrolment with a microsoft CA, there is no way around this. Please note that the web service requires basic authentication and generally poses a security risk if everything is not configured correctly.

Preparing the RootCA

The following server roles must be enabled on the CA:

• Certification Authority
• Certification Authority Web Enrollment

After installing the roles, Basic Authentication must be enabled in IIS.

Next, a dedicated certificate template must be created. This template defines the attributes used to sign the VCF component certificates. To do this, I start the Certificate Templates console by running certtmpl.msc. I duplicate the default template Web Server.

In the new template, adjust the compatibility settings so that Windows Server 2008 R2 is specified as the certification authority and Windows 7 / Server 2008 R2 as the certificate recipient. Give the template a unique name, e.g. VMware.

Remove ‘Server authentication’ from the application policies under Extensions, enable the ‘Basic Constraints‘ extension, and in the key usage settings, enable ‘Signature is proof of origin (non-repudiation)’ while keeping the default settings for the rest. On the Subject Name tab, make sure that ’Supply in the request’ is selected. Save the new template once all settings have been applied.

Finally, add the template to the Microsoft certification authority. To do this, open ‘certsrv.msc’, go to ‘Certificate Templates’ and select ‘New → Certificate Template to Issue’. Select the template you just created and activate it.

The final step is to assign the rights. The CA user requires the following rights:

• Issue and Manage Certificates
• Request Certificates

This completes the process on the Windows side. The CA can now be used either manually or automatically via VCF Operations.

Preparing VCF Operations

The cool thing is that you can now not only change VCF certificates, but also, if you use a central VCF operations like I do, you can also change these certificates.

This is also practical, as I can store different CAs per VCF instance in VCF Operations. Perhaps I will write a second article in which I do the whole thing via OpenSSL CA. To set the CA for a VCF, go to Fleet Management -> Certificates and then select the VCF instance or VCF Management, depending on whether it is to be configured for the management components or the VCF instance.

The settings are self-explanatory. It is important that the complete username with domain is used and that the template is written correctly (case sensitive). For the CA server URL, https is recommended, as basic authentication is used, which means that the user and password are transmitted in plain text. Therefore, as a minimum security measure, transport encryption with TLS should be used here.

Request certificate

In my case, I want to replace the vCenter certificate in my workload domain. To do this, I select my workload domain in VCF Operations, click on the component that is to receive a new certificate, and select Generate CSR.

The Generate CSR dialogue box appears and some information must be added. The Common Name, Host and Subject Alternative Name (SAN) are pre-filled and should not be changed. The following fields must be completed or modified:

• Organisation
• Organisation Unit
• Country
• State/Province
• Locality
• Key Size

Once saved, a new certificate request is created in the background. The private key remains on the server that created it. You can download the request if a manual signing request is used or required. This would be the case if you do not want to use the Microsoft Certification Authority Web Enrollment Service for security reasons or if you do not use either Microsoft or OpenSSL CA.

Once the certificate request has been successfully created, all you need to do is select Replace With Configured CA Certificate and the automatic enrolment process will start.

The SDDC now takes care of all further steps. A new certificate is automatically requested from the CA, signed and then exchanged on the relevant system. Depending on the system, the process takes a few minutes, so it’s best to go and make a cup of coffee while you wait, as you can’t do anything else at this point.

And (e)voilà, here is our beautiful new certificate (sorry for the bad pun about my employer evoila). The cool thing is that certificate renewal can now be done with a simple button press. No more filling out annoying requests or interacting with the CA. One button press, a cup of coffee, a little wait, and the job is done – that’s how I like it.

But Daniel, you wanted to show us how to do it manually!

Extra - the manual way

It all starts as before with a request. To ensure that the certificate can be easily exchanged with VCF Operations, the request must be executed via Operations and then simply downloaded via the GUI.

With the request, I now create a certificate at my CA and save it as Base 64 encoded. I now have to import the certificate, together with the the CA certificate. To do this, the certificate must be PEM encoded. Alternatively, the certificate can also be imported as a certificate chain encoded in Base64. VCF Operations supports both options.

Server Certificate

-----BEGIN CERTIFICATE-----
MIIDdzCCAl+gAwIBAgIEbmh8+TANBgkq...
...dein Server/Leaf-Zertifikat...
-----END CERTIFICATE-----

Root Certificate

-----BEGIN CERTIFICATE-----
MIIFQTCCAymgAwIBAgISA9pL1W...
...Intermediate CA Zertifikat...
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpF3...
...Root CA Zertifikat (optional)...
-----END CERTIFICATE-----

Once the certificate has been successfully imported, it can now be assigned. To do this, select the certificate in VCF Operations that is to be replaced and click on the button Replace with imported certificate.

VCF Operations suggests the certificate. If this does not work, you must select the correct certificate from the drop-down menu. In my case, it worked and the correct certificate was selected. Push replace and go to the coffee machine.

Conclusion

Certificates will still be a pain in the neck in 2025, but VCF Operations has made the process much, much easier. The ability to easily exchange certificates for both management components and the components of my VCF instances is brilliant. Unfortunately, there is still no nice built-in feature that allows me to do this on Microsoft without the Web Enrollment Service.

The fact that I can automatically renew my certificates with a single button is also something that makes the whole process much easier for me. Many of the things just shown were already possible in older VCF versions, but now that everything is centralised in VCF Operations and works for all my VCF instances, it’s just great. I will take a look at enrolment with an OpenSSL CA when I have time, but for now I am focusing on the Microsoft CA, as it is really widespread in practice and I am more familiar with the Windows CA than with the OpenSSL CA.

Maybe one more thing: It is also possible to exchange the certificates of the ESX servers. You just have to activate the Show ESX Hosts slider.

Now that you’ve read this far, how painful is certificate exchange for you, and how much coffee have you had? Feel free to let me know on LinkedIn.

In my two other VPC articles, I showed you the basics and the differences between centralized deployment and distributed deployment. Today, I would like to write something about security, as VPCs offer various possibilities in this area. VPCs provide a certain degree of isolation through their private networks, as these cannot be easily routed from outside the VPC. But what if we want to protect a VM from the public network or a private VM that has been assigned a public IP? This is where the distributed firewall and gateway firewall come into play. Let’s find out exactly how.

NSX Project

When it comes to isolation, we must also briefly address the topic of NSX Project. A Project is nothing more than a tenant in NSX. The name is debatable; personally, I don’t find it particularly apt.

There is always the default project, and most of my customers only use that and are happy with it. However, I don’t have any isolation (per default) through the distributed firewall in the default project. The default rule set initially allows all traffic, and in terms of VPCs, this means that everything that is routable can also be accessed.

But there is a solution. When creating a new project, you can have a default set of rules created that isolates the project from other tenants, including the default tenant. This can also be changed later. To do this, you must activate Activate Default Distributed Firewall Rules in the project settings.

The default Distributed Firewall rules allow communication between workloads within a Project, including communication with the DHCP server. All other communication is blocked.

The automatically generated rule set can be edited, expanded, and deactivated, but not deleted. A dynamic security group is automatically created, which includes all segments of the project. This security group creates an “allow any” rule within the project.

This completes the initial isolation between tenants, because even if the default tenant has an allow Any rule, the default drop rule of the project will reject the traffic if it comes from outside the tenant. However, this also means that my public VMs are no longer public, as they can only be accessed within the project. Internet access is also no longer possible, as all traffic to and from the project is blocked. I should change this.

Distributed Firewall

This is where the fun begins, and unfortunately we also have to take a look at the role model, because depending on which user role I am logged in with, I have different options for placing my distributed firewall rules. And the placement of the firewall rules is important for enforcing the rules or when they are enforced.

NSX VPC Role Model (Broadcom)

To better understand the table, it is important to know that rules can be created not only in the Security / Distributed Firewall tab, but also at the VPC level by going to the VPC settings and clicking E-W Firewall Rules for distributed firewall rules or N-S Firewall Rules for gateway firewall rules.

Rule order

It is important to note that if you create a firewall rule via the VPC and not via Security / Distributed Firewall, it will be created in the application category and will be applied according to the conventionally created Distributed Firewall rules, but will take precedence over the default Layer 3 policy. This means that the Project Admin or Enterprise Admin can override VPC rules at any time. By default, VPC rules are hidden in the Distributed Firewall. However, you can display the VPC objects in the DFW.

It is important to understand that the categories and apply to are still used to process the sequence. A deny all project rule in the Environment category takes precedence over a default project rule in the Application category with apply to dfw. However, the project rule is only valid within the project, even if apply to is set to dfw. A firewall rule from the default project with apply to set to dfw is enforced on all VMs. If rules from different sources are in the same firewall category, the default project rules are always processed first, then the project-specific rules, and only then the VPC rules – provided that the rules are relevant for the VM (apply to). If a default project rule is not enforced on the VPC VM via apply to, it is of course not processed. If no rule can be applied, our rule no. 2, also known as the default Layer 3 policy, comes into play. In production environments, this rule should always be a drop all and log rule.

To make this clearer, I have prepared an example. In my default project policy, I block http for all VMs (apply to dfw). In the project-specific policy, I block the https port for all project-specific VMs, and in the policy for VPC3, I allow icmp on all VPC3 VMs.

We can also view all of this on the CLI.

[root@vcf09-m01-esx01:~] vsipioctl getrules -f nic-1056617-eth0-vmware-sfw.2
ruleset mainrs {
  # generation number: 0
  # realization time : 2025-07-27T21:30:00
  # FILTER (APP Category) rules
  rule 12268 at 1 inout protocol tcp strict from any to any port 80 drop;
  rule 12264 at 2 inout protocol tcp strict from any to any port 443 drop;
  rule 11240 at 3 inout protocol icmp from any to any accept;
  rule 10216 at 4 inout protocol ipv6-icmp icmptype 136 from any to any accept;
  rule 10216 at 5 inout protocol ipv6-icmp icmptype 135 from any to any accept;
  rule 10217 at 6 inout protocol udp from any to any port {67, 68} accept;
  rule 10218 at 7 inout protocol udp from any to any port {546, 547} accept;
  rule 10219 at 8 inout protocol any from addrset 7a87cbca-ca80-4042-9507-558bb69b94c8 to addrset 7a87cbca-ca80-4042-9507-558bb69b94c8 accept;
  rule 10220 at 9 inout protocol any from any to any drop;
  rule 3 at 10 inout inet6 protocol ipv6-icmp icmptype 136 from any to any accept;
  rule 3 at 11 inout inet6 protocol ipv6-icmp icmptype 135 from any to any accept;
  rule 4 at 12 inout protocol udp from any to any port {67, 68} accept;
  rule 2 at 13 inout protocol any from any to any accept;
}

In the CLI output, you can clearly see the order in which the rules are processed. Here is the cross-check with a VM from VPC4. You can see that the default project rule and the project rule apply, but the VPC3-specific rule (ID 11240) does not exist.

[root@vcf09-m01-esx01:~] vsipioctl getrules -f nic-1056658-eth0-vmware-sfw.2
ruleset mainrs {
  # generation number: 0
  # realization time : 2025-07-27T21:36:42
  # FILTER (APP Category) rules
  rule 12268 at 1 inout protocol tcp strict from any to any port 80 drop;
  rule 12264 at 2 inout protocol tcp strict from any to any port 443 drop;
  rule 10216 at 3 inout protocol ipv6-icmp icmptype 136 from any to any accept;
  rule 10216 at 4 inout protocol ipv6-icmp icmptype 135 from any to any accept;
  rule 10217 at 5 inout protocol udp from any to any port {67, 68} accept;
  rule 10218 at 6 inout protocol udp from any to any port {546, 547} accept;
  rule 10219 at 7 inout protocol any from addrset 7a87cbca-ca80-4042-9507-558bb69b94c8 to addrset 7a87cbca-ca80-4042-9507-558bb69b94c8 accept;
  rule 10220 at 8 inout protocol any from any to any drop;
  rule 3 at 9 inout inet6 protocol ipv6-icmp icmptype 136 from any to any accept;
  rule 3 at 10 inout inet6 protocol ipv6-icmp icmptype 135 from any to any accept;
  rule 4 at 11 inout protocol udp from any to any port {67, 68} accept;
  rule 2 at 12 inout protocol any from any to any accept;
}

Gateway Firewall

To use the gateway firewall, it must first be activated for the project. The gateway firewall is implemented on the VPC gateway for VPCs and not on a T1 or T0 router. To activate the gateway firewall for a project, you must have at least project admin rights and can then activate it in the VPC Security Profile. It is then activated project-wide and can be configured by a VPC admin.

The rules are configured in the VPC setup under N-S Firewall Rules. They are not visible under Security / Gateway Firewall, regardless of the permissions my user has.

The configuration itself is very simple and straightforward. Of course, the gateway firewall does not work with the distributed transit gateway, as edge nodes are required for the gateway firewall. You also need to pay attention to the order in which they are processed. The gateway firewall only intervenes when traffic is sent via the VPC gateway to the transit gateway or vice versa. This means that internal VPC communication cannot be blocked.

For outgoing traffic, the distributed firewall is processed first, followed by the gateway firewall. For incoming traffic, the gateway firewall is processed first, followed by the distributed firewall. If the destination is located in another VPC or in another NSX project, the firewall is also processed at the destination as incoming traffic. If a gateway firewall is configured at T0, the order for outgoing traffic would be distributed firewall, gateway firewall on the VPC gateway, and then gateway firewall on T0, and vice versa for incoming traffic.

The gateway firewall does not offer us as many options as the distributed firewall, and it also requires edges. North-south traffic can also be effectively protected with the distributed firewall, although you may need to give a little more thought to the design of the rule set.

Bonus Round - TCP Strict

At the beginning of the article, I wrote that NSX has disabled TCP strict in the default policy. This is partly because TCP strict cannot work on an any/any rule; at least one TCP service must be specified. It is also irrelevant for a drop rule. But why is this the case? TCP Strict is a firewall mechanism that controls the 3-way handshake of TCP. In short, when the first packet is a SYN packet from the client, the firewall checks whether the subsequent packet is a SYN-ACK from the server. Only when the final ACK packet from the client arrives is the connection considered established and the firewall allows the user data to pass. Essentially, the firewall enforces clean TCP communication between the client and server and prevents half-open connections. Asymmetric routing is also blocked. This can cause problems with NSX in particular if you make the N/S peering of the edges to a firewall (this feature is called anti-spoofing in Checkpoint firewalls, for example).

If you now look at the default firewall rules of NSX for VPCs, you will quickly see that TCP Strict would not make sense in the policy.

This is because TCP Strict would not work with ICMP rules and the default DHCP rules (UDP) anyway, as there is no TCP traffic and TCP Strict does not apply with allow any/any or drop any/any.

No problem, right, Daniel? That’s exactly where my criticism comes in. Although you can’t delete the policy or enable TCP Strict, you can add as many firewall rules as you like – it’s so convenient and I don’t need to create a new policy. The only problem is that the new rules (part of the default policy) don’t have TCP Strict either, which weakens my firewall rules even though there’s no reason for it. The problem is that I can attack my server with modified TCP packets. This can happen intentionally or unintentionally (for example, due to a software bug). For example, I can easily bring down a test VM with TCP Ack packets using hping3. Normally, if TCP Strict is enabled, ACK packets without a preceding and matching SYN packet would simply be discarded by the firewall.

To perform a simple test, I added an Allow HTTPs rule to the default VPC firewall policy (this is the policy without TCP Strict) and bombarded my system with hping3 and TCP ACK packets.

hping3 -A --flood -p 443 -d 1200 10.28.11.35

Parameters:

• -A Sends TCP ACK packets (no connection establishment, simulates response traffic)
• –flood Sends packets as fast as possible without waiting for responses
• -p 443 Target port: 443 (HTTPS)
• -d 1200 Sets the payload size to 1200 bytes
• 10.28.11.35 Target IP address

The result is quite spectacular in BTOP. 100% CPU utilization of the VM and approx. 10 Gb/s network traffic, and everything passes through the firewall to the VM. To double-check, I create a new firewall policy. NSX always has TCP Strict enabled by default for new policies. The rule will be the same, allow HTTPs, but this time with TCP Strict enabled.

As you can see in the rule statistics, the new rules receive a large number of hits, but since there is no valid TCP handshake, the distributed firewall discards all packets and no packets arrive at the VM.

[root@vcf09-m01-esx03:~] vsipioctl getrules -f nic-1164294-eth0-vmware-sfw.2
ruleset mainrs {
  # generation number: 0
  # realization time : 2025-07-28T21:27:00
  # FILTER (APP Category) rules
  rule 13288 at 1 inout protocol tcp strict from any to any port 443 accept;
  rule 13289 at 2 inout protocol tcp from any to any port 443 accept;
  rule 5101 at 3 inout protocol ipv6-icmp icmptype 136 from any to any accept;
  rule 5101 at 4 inout protocol ipv6-icmp icmptype 135 from any to any accept;
  rule 5102 at 5 inout protocol udp from any to any port {67, 68} accept;
  rule 5103 at 6 inout protocol udp from any to any port {546, 547} accept;
  rule 5104 at 7 inout protocol any from addrset ba52d50f-55b3-453b-bff2-b2dd72beca3c to addrset ba52d50f-55b3-453b-bff2-b2dd72beca3c accept;
  rule 5105 at 8 inout protocol any from any to any accept;
  rule 3 at 9 inout inet6 protocol ipv6-icmp icmptype 136 from any to any accept;
  rule 3 at 10 inout inet6 protocol ipv6-icmp icmptype 135 from any to any accept;
  rule 4 at 11 inout protocol udp from any to any port {67, 68} accept;
  rule 2 at 12 inout protocol any from any to any accept;
}

In the CLI, you can also clearly see that rule 13288 uses TCP strict and rule 13289 does not.

What can we learn from this?

Firstly, you should think carefully about where and how you write your firewall rules, and secondly, you can use hping3 to disable your lab. :D

Furthermore, TCP Strict does not help with SYN flood attacks, as NSX creates a TCP session for each SYN packet and waits 120 seconds for the SYN-ACK. In the case of a hping3 attack, this would mean that the server would no longer be able to respond, especially since there is a parameter for random source and you end up simply occupying all of the server’s ports. Therefore, it may be useful to create a flood protection profile. These can be created separately for each project. However, you should be familiar with the traffic profile of the environment and, in general, caution is advised with such mechanisms, as overly strict settings can also block desired traffic.

Conclusion

But what can I do? My recommendation would always be to write your own policies for additional rules to the default VPC rule set. The VPC default rule set is very good for isolating the VPCs from each other, but that’s about it. Within the VPC, you should write your own policy, or at least one for traffic in and out of the VPC.

Use a perimeter firewall, as most attacks do not come from within, and learn how to write meaningful rules on the distributed firewall. Use Apply To, even on VPC rules, even if they are already restricted to the VPC. I have written about why Apply To is useful in another article. This also applies to the VPC topic.

Otherwise, all I can say is get yourself a KALI Linux and test your lab. I noticed the behavior with TCP Strict more by accident. It’s not a huge vulnerability, but you should be aware of what defaults and automatically generated rules or settings do. I don’t know yet if there will be another article on the topic of VPCs. My esteemed colleague Steven Schramm has taken a closer look at the topic of VPCs and VKS. It’s a very readable article.

So all that’s left for me to say is happy pen testing!

Where should I start? Perhaps I should describe my starting point. I deployed a nested lab with VCF9 for the VCF9 beta, so far so good. This lab came with its own VCF Operations. Now we are GA and I am lucky enough to have 256 core VCF9 licenses. I wanted to upgrade part of my hardware lab to ESX9 because I also want to test memory tiering in its final version, and that’s where my problems start. In VCF9, there are no more keys and you have to license your products via VCF Operations and vCenter. This also applies to VVF, or if you want to use ESX9 on your physical lab hardware.

Conclusion: I need a second VCF Operations and would have to split my licenses. However, this is not a good solution because my nested lab is not permanently on, which means that Lab VCF Operations is also not permanently on.

The solution: a central VCF Operations and Fleet Manager.

Let’s get started

First, I shut down my Nested VCF Operations, as well as the Cloud Proxy and Fleet Manager, because in the best case scenario, I won’t need them in my Nested Lab anymore. Since the licenses are activated online, I have until the end of this year before the next usage report is due. So, for now, nothing will be left inactive.

Then I deploy a new VCF Operations, a Fleet Manager, and a Cloud Proxy.

The actual deployment is done using standard OVAs that you download from the Broadcom portal beforehand. So far, so unspectacular. I won’t go into detail about the basic setup of the individual appliances, as this is fairly straightforward. I also won’t explain how to integrate a cloud proxy, as this is fairly self-explanatory and can be done via Operations under Administration -> Cloud Proxies.

Perhaps one more note: the fleet manager OVA is called VCF-OPS-Lifecycle-Manager-Appliance.

Now that we have deployed our VCF Operation and Fleet Manager, we need to bring the VCF Operation Cluster online and connect our Fleet Manager. To do this, open the Operations admin interface using the following URL: https://< VCFOPS FQDN >/admin

This is where the “cluster” is put into operation. The process is straightforward. The operating system is also updated if necessary, provided that an Internet connection is available.

Since we deployed VCF Operations manually, no Fleet Manager is connected yet, of course.

Here you need the fleet manager FQDN and the admin password. In addition, the admin password for VCF Operations is required; without it, the process cannot be completed.

The admin password must comply with the new complexity rules. The days of VMware1! are over. If a password that is too weak was set during deployment, onboarding will now fail. Okay, I could have written that earlier, but you are warned about this during OVA deployment, and I’m sure my readers would never just click “continue” without thinking – not that this would have happened to the author here.

Of course, I was able to connect to the Fleet Manager right away the first time—at least that’s what I’m going to say. The whole process takes a little time, so it’s best to have a cup of coffee ready.

When the Fleet Manager is connected, a VCF operation instance with Pending Import is now displayed in Operations under Fleet Management -> Lifecycle.

This should not cause us any concern and is completely normal. Before we can continue here, we need to onboard our future environments so that the Fleet Manager can do its job.

Onboarding of VCF and other ESX9 environments

At this point, it would also be possible to deploy a cloud proxy, but this is not necessary. To continue, my nested LAB and my ESX 9 servers must be integrated. This is done via Administration -> Integrations Here, you specify the accounts you want to onboard. For my ESX 9 servers, I select vCenter as the account type and specify the appropriate vCenter 9 and account details.

For my VCF configuration, I select vSphere Cloud Foundation for VCF. You need the login details for the SDDC Manager and not the vCenter. It is recommended to create separate credentials in Operations, even if you use the same password multiple times. I decided to enable “Domain Monitoring when creating” under Advanced Settings.

Under Collector / Group, you can select the collector you want to use for collecting data. The default collector group contains the VCF operations.

After successfully adding our accounts, data collection must also be activated.

Licensing

You must register the VCF Operations instance that you want to use for license management in the VCF Business Services console. There are two ways to do this: Connected and Disconnected.

• Connected Mode

If the environment is connected to the Internet, you can register your VCF Operations instance in connected mode. Connected mode provides a faster and simplified registration process and makes it easier to update your licenses. In addition, usage is automatically uploaded every night. The exact process for registering the VCF operation in connected mode can be found here. However, the entire process is guided very well by the GUI. The license must be updated every 6 months or when a subscription expires. In Connected mode, this can be done with a single click.

• Disconnected Mode

Disconnected mode is for all users who have a dark site and therefore no internet access for VCF operations. Onboarding is a little more complicated because license files and usage reports have to be uploaded and downloaded manually. Even in disconnected mode, you have to create and upload usage reports regularly. The license must be updated at least every 6 months or when a subscription expires. Instructions for disconnected mode can be found here. However, the process is really very simple and very well guided via the GUI.

Assigning licenses

After 15 minutes, the previously added vCenter will be displayed in VCF Operations. Under License Management – Licenses, i should now see my vCenter and VVF or VCF licenses that I previously assigned to my operations via the VCF Business Service Console. Because we have deployed a central VCF Operations, the licenses do not need to be split. Instead, I can assign the same license to all of my added vCenter9 instances, as long as there are enough cores available in my license. To do this, I select the vCenter instances and assign a primary license. This license is then imported into vCenter and used to license all other VCF products, such as NSX. vSAN must be assigned to the vCenters as a so-called add-on license.

Now all that’s missing is the VCF lifecycle. Remember, we still had one point left to cover.

VCF Lifecycle

We still had our fleet manager who hadn’t been fully onboarded yet. First, we need to create deployment targets. These can be found under Fleet Management -> Lifecycle -> VCF Management -> Deployment Targets.

The deployment target is important so that the Lifecycle Manager can deploy components. Here, too, the vCenter is selected and the correct account must be specified. If no suitable account exists, a new one can be created from the dialog box. The connection is then validated and the deployment target is configured.

Finally, we take care of our operation that is still displayed as Pending Import. To do this, we click on the Import button under Fleet Management -> Lifecycle -> VCF Management -> Overview for VCF Operations.

Most of it is already preselected; you just need to enter the root password for the vCenter instance where my components are located (VCF Operations and Cloud Proxies). After confirming with Next, you need to do the same for the Cloud Proxy.

Congratulations, our deployment should now look like this.

We are now able to deploy an operation for logs via the Fleet Manager. It is important that you have configured your Broadcom token under Depot Configuration or, if you have an offline repo, that this is configured.

But there are also a few limitations:

• VCF Management supports only one instance each of VCF Automation, VCF Operations, VCF Operations for logs, or VCF Operations for networks. All components integrate with VCF Operations.
• VCF Management supports multiple instances of VCF Identity Broker. For each VCF instance, VCF Management allows you to deploy one and only one VCF Identity Broker instance.
• You deploy VCF Identity Broker into either an embedded form factor such as vCenter or an external form factor such as a VCF appliance.
• The first VCF Automation or VCF Operations for networks instance that you import or deploy will be in integrated mode.

Resource requirements

I’m pretty sure that the required resources can still be adjusted in the lab. Especially when it comes to RAM, you should easily be able to get by with half, but I haven’t been able to test it properly yet.

Summary

Manual deployment is not as intuitive and simple as it is with the VCF installer. However, this setup allows me to easily control and manage my licenses via VCF Operations, which is particularly useful in an environment with nested Labs. For a production environment, I would prefer deployment in a VCF Management Domain. The components can still be shared by multiple instances if the security concept allows it. I think the blog has shown that import is also possible at a later date.

This is the second article in a series of articles (number of articles unknown) about VPCs in VCF 9. If you are not familiar with VPCs and have not yet come across this topic, I recommend reading Part 1 first. I will not go into all the basics again in this article, as these were already explained in the first article. In this article, I will look at the distributed deployment of VPCs and show how VPCs can also be used without an edge cluster.

Wait, did he really just write VPCs without Edge Clusters?! Sounds good, right?

Let’s go - NSX project

First things first. Since I am using the same VCF 9 installation as in my first article and we already have VPCs in centralized deployment here, I first create a new NSX project. Perhaps I should mention NSX Projects again here. VMware writes in the VCF 9 Guide: A project in NSX is analogous to a tenant. By creating projects, you can isolate security and networking objects across tenants in a single NSX deployment.

I think that description fits quite well. In addition, we can currently only have one transit gateway per project. Since our default tenant (aka default project) has a transit gateway in centralized deployment, I need a new tenant aka project.

To create a new tenant, click on the Project drop-down menu in the NSX GUI and then click on Manage.

The most important setting besides the name (yes, I recently watched Oppenheimer) is the External Connection. These settings determine whether the transit gateway is distributed or centralized. The External IP Block defines the public VPC network (I explained the different networks in VPCs in Part 1). We have the option of connecting the project to a T0 gateway independently of the transit gateway. However, this is only relevant for non-VPC networks and will not be considered here. The same applies to the option of assigning an edge cluster to the tenant. Since I really want to avoid edges entirely in this article, we will also ignore this option. We also have the option of setting distributed firewall rules directly, which only allows communication within the tenant, but since firewalling will be covered in another article, we will ignore this option and disable it.

External Connection - the distributed version

We should take another closer look at the topic of external connections.

Since we have a distributed connection here, we need a VLAN and a gateway in my physical network. In my lab, it is VLAN 1011 and my Mikrotik Core Router has 10.28.11.1/24. My external IP block also comes from the network area of VLAN 1011, as my physical router has to route to this network.

Let’s build a VPC, or two, or…

Readers of Part 1 should now be quite familiar with all of this.

First, we set up our private transit gateway IP block in the VPC Connectivity Profile or create one. Remember, this is an unrouted network, but it must not overlap with the other private VPC networks for all tenants. This network is used to connect VPCs to each other and is an NSX overlay network. Next, I configure the service profile, which is where I can configure a DHCP service for my VPC, and finally I create the private PVC IP CIDRs. As in my other article, this will again be 192.168.5.0/24.

That’s it, our VPC has been created and we can now create our first VPC subnets. I will create a public, a private, and a transit network, each with 16 IPs and DHCP enabled. In addition, I will create the VPC Core in the same way. I hope the Demon Core doesn’t get too close - nudge nudge wink wink.

The tenants’ VPC networks are displayed in vCenter, but they are managed by NSX. This means that they cannot be extended or created via vCenter as in our default project.

Test architecture

I created six test VMs and assigned one VM to each VPC network. For a quick overview, here is a table showing all VMs.

The final VPC structure of the Manhattan tenant looks like this:

We can also test the interconnectivity between the VPCs from Part 1. The overall network topology looks like this:

Here are the test VMs from the first article:

Test scenarios

I will perform certain tests for each VM from VPC-Demon. First, I will test Internet connectivity, then intra-VPC connectivity, i.e., communication with all VMs in VPC-Demon, and finally inter-VPC connectivity, i.e., communication with all VMs in VPC-Core. Finally, I will perform a connectivity test from outside the NSX environment and a connection test to the VMs from VPC-BLUE. I will do all of this with simple ICMP tests. To do this, all firewalls on NSX and the VMs have been disabled. That’s a lot of tests, so let’s get started.

alpine01-demon-private

I will describe the tests using examples. For the other VMSs, I will present the results in a table.

Internet connectivity

In my first test, I want to see where alpine01-demon-private can communicate. The VM was assigned the IP address 192.168.5.3 by DHCP. Since our transit gateway was created in distributed mode, we cannot use auto SNAT and generally have no way of using stateful services. This means that the VM cannot communicate with the internet because the VPC private network is not routed and no SNAT is performed in the public network.

Intra VPC connectivity

Next, I test the connectivity to the alpine02-demon-transit vm. This test is also successful, as the transit VM can also be accessed via the VPC gateway. I also reach alpine03-demon-public vm. This happens fully distributed. Cool.

Inter VPC connectivity

This is where it gets exciting. Basically, I can only access VMs from the same VPC here. If I had stateful SNAT, I could also access the public VMs of other VPCs belonging to other tenants or VMs of other VPCs in the transit network of my own tenant (provided the firewall allows this). Since we do not have NAT, a VM from a private VPC network cannot establish an intra-VPC connection.

External connectivity

The VM cannot be accessed externally. My core router also does not have a route for the 192.168.5.0/28 subnet. Even if my router has a route for this network, a VM in the private VPC network cannot communicate with the outside world. There is a special case, but I will deal with that later.

alpine02-demon-transit

alpine03-demon-public

That’s a lot of connections. Why is that? Technically speaking, the VM shoud be located in VLAN 1011. But is that really true? Let’s take a closer look.

The magic of the distributed transit gateway

When we look at a traceflow of the VM alpine03-demon-public, we see that the VM is routing even though I am addressing my physical router.

alpine03-demon-public:~# traceroute 192.168.11.1
traceroute to 192.168.11.1 (192.168.11.1), 30 hops max, 46 byte packets
 1  10.28.11.17 (10.28.11.17)  0.224 ms  0.168 ms  0.048 ms
 2  100.64.0.0 (100.64.0.0)  0.064 ms  0.079 ms  0.038 ms
 3  192.168.11.1 (192.168.11.1)  0.265 ms  0.168 ms  0.188 ms

In NSX, it looks like this:

Maybe we should take a look at the trace flow from my router.

[admin@ToR.lab.home] > tool/traceroute 10.28.11.19
ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS                                                       
10.28.11.19                        0%    7   0.6ms     0.5     0.3     0.6     0.1                                                              

It appears unusual, as if the VM were directly in the VLAN. As a cross-check from my Mac computer, we also do not see 100.64.0.0 or 10.28.11.17 in the traceroute.

 danielkrieger@MBP-DKrieger  ~  traceroute 10.28.11.19
traceroute to 10.28.11.19 (10.28.11.19), 64 hops max, 40 byte packets
 1  firewall.lab.home (192.168.10.1)  1.076 ms  0.331 ms  0.204 ms
 2  192.168.9.2 (192.168.9.2)  0.651 ms  0.357 ms  0.309 ms
 3  10.28.11.19 (10.28.11.19)  0.810 ms  1.100 ms  0.482 ms

Wait a minute. What is VMware doing?

From an external perspective, my Mikrotik router can address the VM directly, and I can also see the MAC address of the VM on the switch port of the ESX server in the router’s ARP table. From the outside, it actually looks as if the VM is in VLAN 1011 as normal. However, the VM would then be unable to communicate because if we take a closer look at the VM’s IP address, we can see that it is in a subnet of 10.28.11.16/28 and its default gateway is not .1 (Mikrotik router) but .17.

alpine03-demon-public:~# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:bf:5e:0c brd ff:ff:ff:ff:ff:ff
    inet 10.28.11.19/28 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:febf:5e0c/64 scope link 
       valid_lft forever preferred_lft forever

alpine03-demon-public:~# route -e
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         10.28.11.17     0.0.0.0         UG        0 0          0 eth0
10.28.11.16     *               255.255.255.240 U         0 0          0 eth0

Another indication that the VM cannot be in a VLAN is the fact that I can route to private VPC networks without without an T0. The VM is therefore in an overlay network.

As we can see in the Traceflow Tool in NSX, the traffic is sent to the Transit Gateway and then directly to the VLAN. Since our transit gateway is distributed, all of this happens on the ESX server on which the VM is currently running. This clarifies the outgoing traffic, but how does the response get back?

Well, I mentioned that my Mikrotik switch knows the MAC address of the VM on the switch port of the ESX server. In addition, my Mikrotik does not know the subnets that NSX creates for the VPCs but the router has a /24 route for the complete Public network.

To ensure that data traffic arrives at the correct ESX, the distributed Transit Gateway running on the ESX hosting the VM responds to ARP requests for the VM, allowing my Mikrotik router to send the data traffic to the correct ESX server. The distributed Transit Gateway will never respond to ARP requests for VMs hosted on a different ESX. The ESX server receives the traffic and sends it directly to the VM.

Insert the “Nice Meme” here.

That was quite a lot, but we’re still not quite done, because there’s also a NAT column in my test table.

External IPs aka reflexive NAT

Similar to centralized deployment, it is possible to expose a VM from a private VPC network. However, since we do not have an edge, this is only possible via reflexive (stateless) NAT. The procedure is the same: we go to the settings of the VPC demon under Additional Configurations and select External IPs. In the dialog, we can only select VMs that have not yet been assigned an external IP and that have a connection in either a transit or private VPC network.

As we can see in the screenshot, an IP address from the public VPC network range has been assigned. The VM alpine01-demon-private can be accessed directly via the IP 10.28.11.2 and can also communicate with the Internet via this address. The complete communication is as follows:

Unlike a VPC with Auto SNAT enabled, it is not possible to communicate from a private VPC network with static NAT to a transit network of another VPC.

With get transport-node external-ip, you can view the external IPs implemented on the ESX host in nsxcli.

vcf09-m01-esx01.lab.home> get transport-node external-ip 
Thu Jun 26 2025 UTC 17:08:36.740
                                                  External IP Mapping Table                                                  
-----------------------------------------------------------------------------------------------------------------------------
       External IP         Internal IP Discovered       Internal IP Seen            VNI          MAC Address       Port ID   
       10.28.11.2                192.168.5.3                 0.0.0.0               69634      00:50:56:bf:5d:b1    67108886

The NSX Reference Design Guide contains traffic flow diagrams for almost all of my tests, for anyone who would like to review everything in detail.

Conclusion

This article shows how VPCs can be operated in NSX 9 in a distributed architecture without any edge clusters. Of particular interest is the functionality of the Distributed Transit Gateway, which transparently mediates between overlay and physical VLAN networks. Despite the absence of edges, many core functions such as routing, public access, and external connections remain fully usable via Reflexive NAT and distributed routing.

The setup allows virtual network functions to be deployed independently of the NSX Edges in a tenant. These could, for example, be provided by a tenant customer themselves. VPCs have become significantly more powerful and exciting than they were in NSX 4.X. This increases flexibility, but also, to be fair, increases complexity. When designing, you have to think more about traffic flow and when traffic is distributed and when it is not. The transit gateway is an exciting new feature and is already familiar from the public cloud sector.

I think in the next VPC article I will deal with the topic of security.

Today, I would like to keep it brief and share a quick tip for all Identity (distributed) firewall administrators. I’m sure everyone has been in this situation: you’ve written a nice new Identity firewall, but for some reason it doesn’t work. If it were a normal firewall rule in NSX, I would first check the Traceflow Tool, but unfortunately that doesn’t work with the Identity firewall.

What is the Identity Firewall?

An identity firewall does not make firewall decisions based on IP addresses, but uses user and group information to apply security policies. It identifies users based on their login credentials and dynamically assigns them appropriate access rights and firewall rules. This enables more granular control and improved protection, as access permissions are directly linked to the actual identity of the user. If you want to know how I used this for one of my customers, you can read about it here.

How do I now fix my firewall rule?

There may be several reasons why my Identity Firewall is not working. Here is a list of some of them:

• Group memberships in AD have been changed or new users have been created.

  This is one of the simplest cases. The default Delta Sync with AD is 180 minutes, and in larger environments, this sync interval may be higher for performance reasons. There are two solutions here: either wait or force a Delta Sync under System - Identity Firewall AD.

• The Identity Firewall is turned off.

  This may sound a bit silly, but it has actually happened to me. You can create Identity Firewall Rules and groups, push the policy, and then find that the rules do not work. You don’t get an error message or notification. If this is the case, the Identity Firewall may not be allowed for the cluster on which the source VMs are located. To do this, go to Security - Distributed Firewall and under Settings, you can enable or disable the Identity Firewall for each cluster.

• Guest Introspection not installed

  If you want to use Guest Introspection, i.e. the version with VMware Tools, then these must also be installed on the source VM. The standard VMware Tools installation does not install these. Either install VMware Tools full or select Guest Introspection explicitly. The source of the error is of course eliminated if you do log scraping (which I personally do not prefer).

What should you do if you want to know whether rules have actually been implemented for a VM?

As mentioned at the beginning, unfortunately you cannot use the Traceflow tool to check a firewall rule. However, there are ways to see if there are Identity Firewall user sessions and which rules have been implemented.

The easiest way to check the sessions is in the GUI under Security - Security Overview and then scroll all the way down to Identity Firewall User Sessions. The VM is not displayed in plain text, but Universal Search in NSX resolves the VM.

The other option is to view it via the CLI.

CLI - Find the rules!

To find out which firewall rules are actually implemented on a VM, we first need to log in to the ESX server on which the VM is currently running. First, we use a simple CLI command to find the network card of our VM.

summarize-dvfilter | grep -A16 WinClientA0001

The result should look like this:

[root@esxnuc03:~] summarize-dvfilter | grep -A16 WinClientA0001
world 1085633 vmm0:WinClientA0001 vcUuid:'50 27 d1 31 1f 89 df b1-65 58 00 d0 ab 64 1f 91'
 port 67108888 WinClientA0001.eth0
  vNic slot 2
   name: nic-1085633-eth0-vmware-sfw.2
   agentName: vmware-sfw
   state: IOChain Attached
   vmState: Attached
   failurePolicy: failClosed
   serviceVMID: 1
   filter source: Dynamic Filter Creation
   moduleName: nsxt-vsip-24765085
[root@esxnuc03:~] 

The name of the NIC is important for our next step. In my example, it is nic-1085633-eth0-vmware-sfw.2.

Next, let’s display all realized rules for the VM:

vsipioctl getrules -f nic-1085633-eth0-vmware-sfw.2

[root@esxnuc03:~] vsipioctl getrules -f nic-1085633-eth0-vmware-sfw.2
ruleset mainrs {
  # generation number: 0
  # realization time : 2025-06-23T18:55:56
  # FILTER (APP Category) rules
  rule 16360 at 1 inout protocol any from any to any with extended src ba9e01bc-5779-4eb2-813e-4c5b8e3ff1bf drop;
  rule 14312 at 2 inout protocol icmp from any to addrset a34212cb-acb2-49b3-b74c-7683c0345a19 accept;
  rule 14312 at 3 inout protocol ipv6-icmp from any to addrset a34212cb-acb2-49b3-b74c-7683c0345a19 accept;
  rule 14316 at 4 inout protocol any from addrset a34212cb-acb2-49b3-b74c-7683c0345a19 to any drop with log tag 'alpine-drop';
  rule 2 at 5 inout protocol any from any to any accept with log tag 'debug';
}

ruleset mainrs_L2 {
  # generation number: 0
  # realization time : 2025-06-23T18:55:56
  # FILTER rules
  rule 1 at 1 inout ethertype any stateless from any to any accept;
}

We can see in the output that firewall rule 16360 has been implemented on my test VM and is therefore active. This firewall rule is my identity firewall rule, and a user who is enabled for this rule is logged in.

We can also see that the VM has implemented two additional rules, namely 14312 and 14316. Although these rules have nothing to do with the VM, the VM is still assigned these rules because the Apply to field is set to dfw, meaning that all VMs receive these rules. I have described the importance of the Apply to field here.

Theoretically, you can also see this in the GUI, but with a delay, and in my experience it was more unreliable than reliable. Nevertheless, I want to show it. If you look at the firewall rule and then go to the group and look at the effective members, a VM should be implemented. As I said, the GUI is very slow here, and if I have problems with the IDFW, I prefer to rely on the CLI.

This method allows you to quickly check whether an IDFW / DFW rule has been realized on the VM. If, as in the example above, the IDFW is disabled for the cluster, I can see the FW rule in the GUI, but it is not displayed in the CLI. The same would be the case if I had an error with the apply to field. The rule would not be realized on my VM and also not shown in the cli.

Summary

Troubleshooting the IDFW can sometimes be a little difficult, but it’s not impossible. I hope this quick practical tip helps you. It has often helped me when the GUI has tricked me. In combination with the check mechanisms in the GUI (even if these are sometimes slow) and the logging the IDFW / DFW is easy to troubleshoot.

I’ve had an exciting few weeks. I was part of the VCF9 beta and ran several VCF instances in parallel, which meant I kept running into performance bottlenecks. On top of that, the Minisforum A2 was finally available. But first things first.

2.5 Gb/s LAN - and what I have learned

Well, where should I start? I bought the Mikrotik CRS326-4C+20G+2Q for my NUCs, which offered me 20x 2.5 GB/s ports and wasn’t exactly cheap at around 800 euros. In hindsight, I can say that it wasn’t a good investment. Not that the switch is bad, it’s not, but as of today, I only use exactly 2 2.5G ports. For this purpose, I utilize all 10G and 40G ports (with brakeout) as 10G ports are now in short supply. Following the recent upgrade, I have exactly one 10G port remaining in the entire lab. That’s why I’m actually considering selling the Switch again. But hey, that’s part of homelabbing. Sometimes you just make bad investments.

Another thing that bothers me about 2.5G is that I constantly have problems with autospeed. Sooner or later, I lose a network port when the switch or server is set to autospeed. This isn’t just a problem with Mikrotik switches; I’ve had the same issue with other manufacturers as well. It also affects my TP-Link access points. As a workaround, I just set everything to fixed speed, then Wakeup on LAN doesn’t work because the nucs can’t handle 2.5 Gb in power save mode – there’s always something. That’s why I decided three weeks ago to sell my beloved nuc cluster and say goodbye to the 2.5 GB LAN experiment. And because I spent a little too much money at Minisforum. But more on that later.

Post NUC era

It’s not just Taylor Swift who has different eras, my home lab does too. Does that mean I’ve now said goodbye to NUCs altogether? A resounding yes and no. I won’t be building any more NUC-based workloads, simply because although the ratio between CPU performance and RAM is good again with the 14th generation, the network is just too slow, especially with VCF9, vSAN ESA and other things I want to test.

However, the NUC 14 with Ultra 7 155H CPU, 16 threads, and a maximum of 128 GB RAM is the perfect management NUC, and since I got it used for around €310, it will be my last remaining NUC with ESX. For management, the single 2.5 Gb/s adapter is sufficient, as it is not an H version (high) and therefore no second network card can be installed.

Yes, I know that the NUC is dusty, but that’s hard to avoid with an open rack.

What’s new?

Now for the obvious: I bought two MS-A2s from Minisforum. These form my new AMD cluster with 32 cores, 64 threads, and 2x 128 GB RAM. In addition, everything on the network side has now been converted to 2x 10 GB per ESXi server.

Since the housing dimensions are completely identical to the MS-01, I was able to use the thingsINrack custom rackmount solution again. Unfortunately, they have become significantly more expensive than last time. You now have to pay around 90 euros for the 3D-printed bracket. The front is molded plastic and therefore has a nice finish. Of all the solutions on the market, this is still the cheapest and most flexible solution. I have had my MS-01 in a rackmount like this from the beginning and have not had any changes or problems with the rackmount so far (I am not being paid for this, I am just very happy with it).

In addition, two more MS-01s have been added. I posted this on LinkedIn weeks ago, but never got around to documenting it here on the blog. The four existing MS-01s have been upgraded to 96 GB of RAM. Now you might ask why we didn’t go straight to 128 GB – well, what can I say? Stupid decisions or bad timing. Shortly after I received the 96 GB RAM for all my MS-01s, the 128 GB RAM was announced. But since everything was already installed, I didn’t want to send the RAM back.

That’s life as an early adopter sometimes.

The lab is finished now! … The lab is finished now?

Please insert the Star Wars Anakin and Padmé meme here. I would put it in the blog, but I’m afraid Disney would sue me :D At least one update is still pending. My management NUC urgently needs 128 GB of RAM, and I think it will get that next month. Otherwise, I’m still thinking about memory tiering on the AMD servers, as they have a lot of CPU power. However, I would have to buy more NVMes for this, as only 2 TB per MS-A2 is currently installed and I don’t have any fast NVMes left in stock.

Another consideration is to run MS-01 on Proxmox – has anyone here ever called Jehovah? Don’t panic, I’m staying loyal to VMware. The idea is to use Proxmox and deploy a thick nested ESX VM on it. Proxmox can do hyperthreading with E/P cores, which unfortunately neither ESXi8 nor ESXi9 can do at the moment. But of course, ESXi is designed to run on enterprise server hardware, and there are currently no asymmetric cores available. In any case, the MS-01 would then have 20 threads and I could deploy nested servers with 16 vCores. But this is currently nothing more than a thought experiment. I’ve done this technically before, but without measuring performance. I could well imagine that a nested ESXi would run better on a physical ESXi than on a Proxmox server. Maybe a future experiment.

Pictures or it didn’t happen

Here is my network diagram, as I have often been asked which tool I use to draw these beautiful diagrams – it is Excalidraw. The whole thing runs as a docker container on my Unraid server.

Here you can see the lab capacity, but you should treat the GHz capacity with caution, as it is not accurate due to the E/P cores. vSphere calculates the capacity based on the first core and the maximum non-boosted frequency of the core, and then multiplies this by the number of cores in the server. This is why the display is never accurate for consumer CPUs, at least for the newer Intel generation.

In total, my lab now has 132 physical cores, including E and P cores. However, this does not matter when creating a VM. If, as with VCF Automation, there is a minimum number of vCores (24 vCPUs), then at the end of the day, 24 vCPUs must be executable on an ESX server, regardless of whether they ultimately run on E/P cores. This also explains why I bought two AMD boxes. I want to test VCF9 with Automation. I currently have 896 GB of RAM. I could upgrade the MS-01 to 128 GB, but that would cost another €1,800, which is too expensive for me at the moment – cost-benefit analysis and all that.

Yes, there is another NUC extreme, and it will remain. However, that is my self-built NAS and does not run any VMware workloads. It is also connected with 2x10G. Thus, it has never really been considered part of the NUC era.

Where to start? The VPC feature has been available in NSX for a long time, but it has often been somewhat under the radar. In VCF 9, VPC is now more than present and small spoiler - I say rightly so!

But wait, from the top, what are VPCs anyway? VMware says NSX Virtual Private Clouds is an abstraction layer that enables the creation of standalone virtual private cloud networks within an NSX project in order to use network and security services in a self-service usage model.

While in NSX 4.X VPCs were only visible in the GUI when an NSX project was created, VMware has now changed this completely. As soon as you open the NSX GUI, the VPCs tab immediately catches your eye.

As you can see here in the screenshot, VPC is very present in NSX and VCF 9, but before we can talk about VPCs, we need to talk about a new gateway in NSX 9 - the Default Transit Gateway.

Default Transit Gateway

What the hell is a Default Transit Gateway? Yes, that’s what I asked myself when I first tried out the beta of VCF 9. Whereas in NSX 4.X a VPC was attached directly to a T0 or a VRF, in VCF9 a Transit Gateway is connected upstream, or rather the default Transit Gateway. At present, you can only have one Transit Gateway per project. What is new in VCF 9 or NSX 9 is that the default project can now also have one or more VPCs.

The default Transit Gateway is present from the start and cannot be deleted. The configuration options are also very limited. I can specify the name, the External Connection and the VPC Transit Subnet. This network comes from the 100.64.x.x range and is normally assigned by NSX itself and does not need to be customized. The HA mode is taken over by the T0 to which my Transit Gateway is connected or, more precisely, my VPC connection profile determines whether the HA mode is taken over by my T0 or not.

However, this is only half the truth, because there is also the so-called distributed mode. However, this assumes that I have created my External Connection as a Distributed Connection and not as a Centralized Connection (with Edge Transportnodes). But that’s another story and I will work through the Distributed Connection in a separate article.

Let’s get started - Network Connectivity

First we need to configure our network connectivity. There are two ways to do this. 1. in the vCenter or 2. in the NSX Manager under System -> Setup Network Connectivity. This point is new. In VCF9, edges are no longer deployed via the SDDC as in VCF 5.X, but via the network connectivity. Here I can also expand my Edge Cluster or deploy a new Edge Cluster. Fortunately, the deployment process has been optimized and is less error-prone and also has a few quality of life improvements.

I now have to configure my VPC External IP Blocks in the Network Configuration. These IP blocks are then used for all public networks and must generally be routed by my network. This also means that these networks must not overlap. The Private - Transit Gateway IP Blocks are new. These are used so that the VPCs of the same project can communicate with each other. Which networks are used in VPCs will be explained in the course of the article. The Private - Transit Gateway IP Blocks do not have to be routed in the physical network. They are used exclusively for intra-VPC communication.

A few things are now happening in the background. Firstly, my VPC connectivity profile is being configured by NSX.

In addition, an external connectivity profile is created under Networking -> External Connections. This determines which T0 router the Transit Gateway is connected to. Finally, the External Connectivity Profile is assigned to my Transit Gateway. Can also be found under Networking -> Transit Gateway. Now that we have taken care of the external connection, we can concentrate on creating the actual VPC.

Create a Virtual Private Cloud

We are now about to create our first VPC. In VCF 9, there are two options. One is via vCenter and the other is directly via NSX. Since we have not yet configured a VPC service profile, we will create our first VPC via NSX. To do this, we go to Add VPC in the VPC menu and are greeted by this attractive dialog box.

Here we assign a name to our VPC, select the previously created connectivity profile, and can select or create a service profile. In the VPC service profile, we can activate a DHCP server and assign profile-specific DNS servers. These must be accessible from the VPC. The same applies to the NTP server. We can also create subnet profiles. The standard profiles that every normal NSX segment has are stored here. Normally, we don’t need to worry about anything here. In my VPC setup, I have activated a DHCP server in the service profile for my default project and specified my LAB DNS and global NTP server.

Next, I define a private VPC IP CIDR (maximum of 5 per VPC). In my lab, this is 192.168.5.0/24. I will explain what these are and what they do in the next section. I also assign a short, descriptive log identifier and assign the VPC name.

After saving and creating the VPC, the additional configurations must be carried out. Here, we can authorize users/groups for the VPC and assign different roles. More importantly, however, we can create our actual subnets. I think now would be a good time to explain the different types of subnets.

VPC Subnets

In VCF 9, there are now three different types of VPC subnets. Actually, there are four, but the fourth is an AVI service subnet, which is actually only a automatic created private VPC subnet - so that doesn’t really count. This means that we actually have three subnet types.

• Private VPC

This subnet is the network that is only routed within the VPC. That is why it is private – makes sense so far. This subnet cannot be accessed outside the VPC and the network is assigned from the private VPC IP CIDRs.

• Private - Transit Gateway

This network is new in VCF9. Similar to the private network, the network is created from the defined private transit gateway IP blocks (VPC connectivity profile). However, it is routed via the default transit gateway, which allows you to access workloads in these networks from all VPCs connected to the same transit gateway. Since you can currently only have one transit gateway per NSX project, you should use these networks with caution. The useful thing is that even though the networks are routed via the transit gateway, they are only accessible to VPCs. VMs connected to a T1 or T0 router via segments cannot reach these networks because the T0 does not have routes for private networks.

• Public

The name says it all. These are subnets created from the previously defined external IP blocks. They must be routed, must not overlap, and are managed via the VPC Connectivity Profile, as with the Transit Gateway. Workloads in public networks are generally accessible from anywhere.

Additional Configurations

Now that we have clarified which subnets exist in VPC, we can move on to the final settings so that the VPC can be created. I create one for each subnet and let it be assigned automatically. Since Default Outbound NAT is specified in the VPC Connectivity Profile, NSX automatically creates an Outbound NAT Rule. In addition, a group containing all VPC subnets is automatically created. This can be used in the distributed firewall. We could also define additional firewall rules (requires a vDefend license – without a license, only stateless N/S rules can be defined). I will explain the topic of network services later in this article.

For my test, I create another VPC. My finished topology now looks like this. As always, I will use Alpine Linux VMs for testing.

I created a second VPC via vCenter. This option has been integrated into the network overview and is essentially the same as in NSX. This creation option can also be controlled via permissions or even completely disabled. There is only one difference compared to creation via NSX Manager. In vCenter, you can currently only manage VPCs from the default project. All VPCs from other projects are read-only. In addition, you cannot modify the profiles. This is reserved for the NSX administrator.

Initial tests with the VPC-Blue

For my test, I created six VMs (see table) and assigned them to the corresponding networks of the VPCs in vCenter. The table shows the gateways automatically created by NSX.

As you can see, the public and transit networks do not overlap and are created consecutively. Since I have enabled DHCP in every VPC subnet, the first and second IP addresses are occupied. The first is the gateway, the second is the DHCP server.

Test scenarios

I will perform certain tests for each VM from VPC-BLUE. First, I will test Internet connectivity, then intra-VPC connectivity, i.e., communication with all VMs in VPC-BLUE, and finally inter-VPC connectivity, i.e., communication with all VMs in VPC-RED. Finally, I will perform a connectivity test from outside the NSX environment. I will do all of this with simple ICMP tests. To do this, all firewalls on NSX and the VMs have been disabled.

alpine01-blue-private

Internet connectivity

In my first test, I would like to see where the alpine01-blue-private can communicate. The VM has been assigned the IP address 192.168.5.3 by the DHCP. Since N-S Services and Default Outbound NAT are enabled in the VPC Connectivity Profile, alpine01-blue-private reaches the Internet and is nat-ed to the first public IP address of the public network IP for VPC SNAT.

Intra VPC connectivity

Next, I test the connectivity to the alpine02-blue-transit vm. This test is also successful, as the transit VM can also be accessed via the VPC gateway. Even though you might think that the SNAT rule applies here, it does not, because NAT takes place on the transit gateway and not on the VPC gateway. The traffic is also distributed routed between the two subnets. This can also be seen very clearly in the NSX Traceflow. The same applies to the connectivity to the alpine03-vpc-blue-public VM.

Inter VPC connectivity

In this test, I first try to reach the VM alpine04-red-private. This is not possible because private networks are not routed via the Transit Gateway. In addition, the private networks overlap. Next, I perform a connection test to alpine05-red-transit. This is successful, but the traffic is nated via SNAT on the transit gateway. This also means that the traffic is not distributed routed and must run via the edge VM. The same applies to traffic to alpine06-red-public.

External connectivity

The VM cannot be accessed externally. My core router also does not have a route for the 192.168.5.0/28 subnet. The routes are not announced to the T0 and therefore cannot be accessed by my physical test client on which I am currently writing this blog. To avoid writing a lengthy explanation, I will present the results in table form.

alpine02-blue-transit

alpine03-blue-public

I think I was able to clearly explain how VPC and the various subnets work. It opens up exciting possibilities. However, I would like to discuss one more feature, namely the option of assigning an external IP address to a VM from a private VPC subnet. This can be done via NSX as well as via vCenter.

Assign External IP

With this feature, you can make a VM from a private VPC network available with just two clicks via vCenter. You can think of it a bit like Elastic IP.

All you have to do is select the VM’s network adapter and confirm. The magic happens in the background. But let’s take a closer look. An external IP was created in NSX under VPC / Network Services. If we take a closer look, we see that an IP address from the public subnet of VPC Blue was assigned.

The vm alpine01-blue-private is now accessible from anywhere via the IP address 192.168.72.2. Pretty cool. You can also cancel the assignment at any time, and the public IP will be returned to the IP address pool and can be used elsewhere.

This is implemented via a DNAT, which cannot be found in the NSX GUI. You can check this on the Active Edge by selecting the relevant interface with the command get firewall Interfaces and then displaying the NAT rules with get firewall < uuid > ruleset rules.

vcf09-edge01> get firewall 0bbcf4b5-1321-40b4-9325-f8023c06bdf2 ruleset rules 
Tue Jun 17 2025 UTC 16:10:06.440
DNAT rule count: 1
    Rule ID   : 536875009
    Rule      : in inet protocol any postnat from any to addrset {192.168.72.2} dnat EIP table id: b1274d77-e09e-4cf5-8377-4501f54e296e

SNAT rule count: 2
    Rule ID   : 536875008
    Rule      : out inet protocol any prenat from addrset {192.168.5.3} to any snat EIP table id: b1274d77-e09e-4cf5-8377-4501f54e296e

    Rule ID   : 536873996
    Rule      : out protocol any prenat from ip 192.168.5.0/24 to any snat ip 192.168.72.1 port 37001-65535

Firewall rule count: 0

Here we see that a DNAT is being performed, making our VM accessible from outside the network. And the best part is that it’s all completely simple and requires no network knowledge.

Conclusion

VPCs are more prevalent than ever, and VMware is pursuing a clear path toward multi-tenancy in VCF9 and NSX9. Many familiar mechanisms have now been combined in NSX 9, and exciting developments are still on the way. In this article, I have only discussed centralized deployment. Distributed deployment is perhaps even more exciting, but it would have made this article too long. I also haven’t talked about security or NSX projects, because both deployment options can be combined to bring significant added value to an on-premises cloud. I will therefore be publishing further articles on VCF 9, NSX 9, and VPCs in the near future. I look forward to many more exciting topics with VCF9. In my opinion, VCF 9 is a really big leap forward, and a lot has really happened in NSX 9 in particular. One thing is clear: the next few weeks and months will not be boring.

One more thing

Perhaps a little fun fact. Anyone who has read my article on MS-A2 will know that I wrote that I run a VCF installation on it – well, now I can reveal that my VCF 9 lab is currently running nested on my single Minisforum MS-A2 server.

After being impressed with the performance and form factor of the Minisforum MS-01, I had to get my hands on their latest beast: the MS-A2, powered by AMD’s latest Zen 5 architecture. It’s also exciting to test a gadget before William Lam. Thanks for letting me be the first (I’m just kidding, of course).

In this post, I’ll give you a quick rundown of both systems and how they compare—especially from a homelab perspective focused on virtualization, low noise, and energy-efficient high performance.

A quick disclaimer first: I am not a hardware tester and do not have any real measuring equipment. Everything I report here is based on my experience with both devices in my environment.

Use Case

Perhaps a quick word about what I do with the machines (for anyone who doesn’t read my blog regularly and stumbled across the article via Google). The minicomputers primarily provide compute and storage resources for my labs. My labs are 99% nested, which means that I install another hypervisor on top of the hypervisor to then perform my actual tests. Nesting offers several advantages: I don’t have to rebuild a test lab, and I can move a finished lab back and forth between hardware platforms, for example. The VCF setup I used to perform the more practical tests allows me to change the hardware base relatively quickly.

But let’s start with the less exciting stuff—tech specs.

Tech Specs Comparison

At first glance, the two devices have a lot in common. Fortunately, the form factor is the same, so I don’t have to buy a new rack mount. The exciting details are in the small print. The MS-01 only officially supports 64 GB RAM, while the A2 only supports up to 96 GB RAM. Nevertheless, I have installed 128 GB of RAM in both systems. I am using the Crucial DDR5 RAM 128GB Kit (2x64GB 5600MHz SODIMM).

The second difference is the CPU architecture. While Intel relies on its Big/Little architecture, which gives us performance and efficiency cores, AMD Zen 5 has symmetrical cores and uses SMT, which is equivalent to Intel’s Hyperthreading. I have described how to make P and E cores usable with ESXi here.

In my opinion, this is already a plus point for the MS-A2, as I can install ESXi just like that. What I find a little strange is the network card selection. While the MS-01 has two 2.5G cards from Intel, the MS-A2 comes with a strange configuration of one 2.5G Intel card and one 2.5G Realtek card. Unfortunately, there is no Realtek driver for built-in cards in ESXi. Fun fact: Realtek USB network cards do work, though. Overall, this isn’t important to me, as I use the X710 Intel cards throughout, since my network is completely 10G.

In summary, the following key data can be noted:

• MS-01 14 Cores / 128 GB RAM / 2x10G / VMware ESXi, 8.0.3, 24674464
• MS-A2 32 Cores / 128 GB RAM / 2x10G / VMware ESXi, 8.0.3, 24674464

Cinebench Multicore Test – MS-01 vs MS-A2

As a first pragmatic test, I created a Windows 11 VM on both systems. Each VM was assigned the maximum number of vCPUs available on the host—meaning:

• MS-01: 14 vCPUs (max useeable cores on i9 13900H)
• MS-A2: 32 vCPUs (max useeable cores on Ryzen 9 9955HX)

Both ESXi server had the power policy set to High Performance, and Cinebench 2024.01 was executed in a continuous loop for 30 minutes to simulate thermal and sustained performance under load.

This may not be a perfect test, but since the MS-01 is already in lab operation, I didn’t want to install Windows 11 natively. Of course, the Windows 11 test VM ran exclusively on the two hosts so as not to distort the results.

The difference is substantial. Even though the MS-A2 pulls more power under load, it doubles the Cinebench score compared to the MS-01. Idle consumption is slightly better on the MS-A2, likely due to the newer and more efficient AMD Zen 5 architecture. Performance-per-watt in multicore scenarios clearly favors the MS-A2.

Minisforum mentions more efficient cooling on its website, which may well be true, but the MS-A2 is somewhat louder and, above all, significantly warmer than the MS-01, which seems logical considering the power consumption. However, both are still within acceptable limits. I wouldn’t want to have the MS-A2 running 24/7 at full load next to my pillow, but anyone who knows my blog knows that I shut down devices I don’t need anyway. So my colleagues on the teams call haven’t complained about a tornado raging next to me.

All in all, I would say that the noise is justified for the performance. When idle, however, the MS-A2 is inaudible. My Mikrotik switch is louder, as is my Dyson fan, which is currently running thanks to the 28 degrees Celsius temperature.

Internal Network Performance – VM to VM on the Same Host

The external network performance was identical on both servers, which is not surprising since both have the same network cards and both CPUs have enough power to run a 10 Gb/s network. That’s why I’m not going to bother with detailed tests at this point.

What interests me much more is how the performance compares between two VMs, as I do a lot of nested labs and also vSAN nested, which is often a decisive factor.

Test Setup

• 2x Alpine Linux VMs (2 vCPU, 2 GB RAM each)
• alpine-a: runs iperf3 -s
• alpine-b: runs iperf3 -c 192.168.16.11
• No optimizations applied (default config, no tuning)
• Only these two VMs were running on the host

MS-01

alpine-b:~# iperf3 -c 192.168.16.11
Connecting to host 192.168.16.11, port 5201
[  5] local 192.168.16.12 port 55930 connected to 192.168.16.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.03 GBytes  43.2 Gbits/sec    0   2.80 MBytes
[  5]   1.00-2.00   sec  4.33 GBytes  37.2 Gbits/sec   60   2.75 MBytes
[  5]   2.00-3.00   sec  4.96 GBytes  42.6 Gbits/sec    0   2.75 MBytes
[  5]   3.00-4.00   sec  5.88 GBytes  50.5 Gbits/sec    0   3.01 MBytes
[  5]   4.00-5.00   sec  6.93 GBytes  59.6 Gbits/sec    0   3.01 MBytes
[  5]   5.00-6.00   sec  6.99 GBytes  60.1 Gbits/sec    0   3.24 MBytes
[  5]   6.00-7.00   sec  6.54 GBytes  56.2 Gbits/sec    0   3.24 MBytes
[  5]   7.00-8.00   sec  6.32 GBytes  54.3 Gbits/sec    0   3.24 MBytes
[  5]   8.00-9.00   sec  6.62 GBytes  56.9 Gbits/sec  139   2.59 MBytes
[  5]   9.00-10.00  sec  6.37 GBytes  54.8 Gbits/sec    0   2.78 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  60.0 GBytes  51.5 Gbits/sec  199
[  5]   0.00-10.01  sec  60.0 GBytes  51.5 Gbits/sec           receiver

MS-A2

alpine-b:~# iperf3 -c 192.168.16.11
Connecting to host 192.168.16.11, port 5201
[  5] local 192.168.16.12 port 35168 connected to 192.168.16.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  9.47 GBytes  81.3 Gbits/sec    0   2.06 MBytes
[  5]   1.00-2.00   sec  10.0 GBytes  85.8 Gbits/sec    0   3.58 MBytes
[  5]   2.00-3.00   sec  10.2 GBytes  87.4 Gbits/sec    0   3.78 MBytes
[  5]   3.00-4.00   sec  10.1 GBytes  86.7 Gbits/sec    0   3.78 MBytes
[  5]   4.00-5.00   sec  10.1 GBytes  87.1 Gbits/sec    0   3.78 MBytes
[  5]   5.00-6.00   sec  10.1 GBytes  87.0 Gbits/sec    0   3.78 MBytes
[  5]   6.00-7.00   sec  10.3 GBytes  88.6 Gbits/sec    0   3.97 MBytes
[  5]   7.00-8.00   sec  10.3 GBytes  88.7 Gbits/sec    0   3.97 MBytes
[  5]   8.00-9.00   sec  10.1 GBytes  87.2 Gbits/sec    0   3.97 MBytes
[  5]   9.00-10.00  sec  10.1 GBytes  86.9 Gbits/sec    0   3.97 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   101 GBytes  86.7 Gbits/sec    0
[  5]   0.00-10.03  sec   101 GBytes  86.4 Gbits/sec           receiver

An attempt to interpret the results

The difference is likely due to the internal memory bandwidth, PCIe performance, and possibly better NUMA handling on the MS-A2’s Zen 5 platform. Since this is purely intra-host, the NICs themselves aren’t part of the data path, which makes this a good synthetic benchmark for VM-to-VM memory and CPU path efficiency. The MS-A2 has a clear lead here—offering nearly 70% higher throughput compared to the MS-01. In addition, the results on the MS-01 fluctuate, which in my opinion can happen when a vCPU is moved to an E Core. Since AMD does not have E Cores, the result is significantly more stable and subject to less fluctuation.

Noise - Methodologically incorrect investigation

I measured the noise development of the MS-A2 “professionally” with my iPhone. There are 100 and 1 apps for this, but I chose the one with the most reviews (Dezibel X – no advertising for the app) and I can’t calibrate it. If anyone wants to give me a dB meter, I’ll do more professional tests in the future. :D

However, I don’t have the time to compare it with the MS-01 - I tested this subsequently at the request of a LinkedIn user. I measured from a distance of 2 meters, as I normally sit between 2 and 3 meters away from my lab. I get a base load of 32-34 dB. However, it should be noted that it is quite warm in my office today and my Mikrotik switch is generally a little louder than usual, but the point is to measure the relative additional load. The background noise in my homeoffice ranges from quiet whispering to the gentle rustling of leaves in the wind. I live in a relatively rural area, so the loudest sounds are my neighbors or the owl outside my window.

In idle mode, the MS-A2 is completely inaudible in my environment. I haven’t noticed any real change in the dB meter. The fan is drowned out by the rustling of the Mikrotik router – I really should replace the fans in that box. I started my VCF Lab on the MS-A2 and waited about 10 minutes. The noise level rises to 35-36 dB but regularly reaches up to 42,3-43 dB. I have the fan curve set to performance, which means that the MS-A2 spins up quickly and then calms down again quickly, which results in a wave-like background noise with the noise constantly rising and falling. I need to see if I can set the fan curve and response behavior in the same way as on an ASUS NUC. The NUC has an incredible number of options for fan management.

The blue curve is the average, and you can see a clear increase in the high frequencies, which are the frequencies that most people find disturbing. To put this into context, 42 dB is still considered relatively quiet and may not even be noticeable in a city apartment. However, a difference of up to 10 dB is also significant. Humans perceive this as approximately double the noise level.

But can it run doom VCF?

First of all, I did not install VCF on the MS-A2. Why? Time! But I did migrate an existing nested lab to the MS-A2. Remember? Introduction and all that? That’s exactly why I build my labs nested. Here’s a brief overview of how my VCF lab is set up.

It’s a consolidated design and doesn’t use vSAN. It’s also built on three nested ESXi servers. This isn’t officially supported with VCF 5.X, but to be honest, I’m constantly doing things that aren’t supported, and that’s not going to bother us here. I deployed the original lab on 2 MS-01 and have been using it for quite some time. The 3 nested ESXi hosts each have 8 vCPUs, 45 GB RAM, and a small boot disk. The principal storage is NFS. If you’re wondering how that works, I wrote a blog post about it here.

If you’re good at math, you’ll have noticed that 3x 45 GB RAM is more than 128 GB RAM, but a little oversubscription never hurt anyone (except when it comes to licenses).

After everything had been migrated offline, since Intel and AMD are not compatible at all and therefore no live migration between the two systems is possible, I started VCF and the poor MS-A2 immediately went into full load.

Poor sweet summer child - the MS-A2 becomes slightly louder and the temperatures rise.

• I received the first sign of life after approx. 3:30 minutes. My vCenter responded to the first ping. Of course, the ping times were far from ideal.
• After 6:50 minutes i can logon to the vCenter. Only NSX Manager is taking its time. But we are already familiar with that.
• After a good 8 minutes, NSX Manager is also available, but still a little sluggish.
• After 10 minutes, everything has settled down enough that you can use the lab really well.

I don’t notice any difference in responsiveness compared to when it’s running on 2 MS-01.

Where’s the poop, Robin?

I’m not sure which of my readers are familiar with the series, but never mind. AMD and NSX is one of those things. Officially, NSX does not work on Ryzen or other AMD consumer CPUs.

At first, I was a little surprised, because I knew that NSX Edges cause problems during installation or upgrade on AMD consumer CPUs, but I deployed my Edges on Intel. A quick look at the console of one of my Edge VMs revealed the following.

So, at first, I was pretty clueless. Luckily, there’s the awesome vExpert community, because a) I’m getting older and forgetting more than I want to admit, and b) I don’t always know everything. My thanks go to Abbed Sedkaoui for his quick help. He put me on the right track. By the way, he has a great blog.

The problem is that when the Edge VM is started, a Python script is executed that checks whether an AMD CPU is installed and, if it is not an AMD EPYC, simply throws an error and prevents DPDK from starting. This leaves Edge offline.

The solution is simple: log in to EdgeVM with root and edit the following script /opt/vmware/nsx-edge/bin/config.py.

# AMD only claims DPDK support of the following processors:
# (1) AMD EPYC 7XX1 series and newer.
# (2) AMD EPYC 3000 Embedded Family and newer.
vendor_info = "'
model_name = "'
for line in cpuinfo.splitlines():
    if line startswith( 'vendor_id'):
        vendor_info = line.split(':'1) [1].strip()
    if line startswith 'model name'):
        model_name = line-split(':', 1) [1] .strip()
    if vendor-info and model_name:
        break
#  if "AMD" in vendor_info and "AMD EPYC" not in model_name: <---here
#      self.error_exit("Unsupported CPU: %s" % model_name)   <---here

You just need to comment out the CPU check. After that, the Edge must be rebooted and our NSX will now work.

The problem is upgrading or redeploying the Edge VMs. You only have about a minute after the Edge has been deployed to adjust the script. That’s why I’ll be deploying my Edges via OVA in the future and using them on an Intel host. That’s the easiest way for me.

Conclusion - Is the MS-A2 the perfect home server?

I would give a clear yes and no. It depends greatly on what you intend to do with it. If you want to deploy VCF without workarounds and technical challenges, then I would definitely say go for the MS-01 and buy two of them. If you are using Proxmox or another hypervisor, then the MS-A2 is a no-brainer – buy it and be happy. Both the MS-01 and MS-A2 are excellent compact systems for homelab use, but the MS-A2 clearly sets a new bar. With twice the Cinebench score and significantly higher throughput on the internal network, this box is the clear winner for me. However, I don’t mind DIY solutions and unsupported solutions. I can completely understand if someone prefers to buy a home server on which they can run vcf without any tinkering—but that’s just not my thing.

Maybe one more thing—how much does it cost?

Since I haven’t had time to update my Lab BOM yet, I’ll briefly mention the costs here. I ordered the MS-A2 from Amazon because it was available immediately and I couldn’t wait for the one I actually ordered from Minisforum to arrive. I paid €1,395.88 for the MS-A2 with 2 TB NVMe and 128 GB RAM. On top of that, there’s a custom rack mount for €92, but I’m not counting that in the total cost. At €312, the RAM isn’t exactly cheap and is still hard to come by in some places. But I think that’s the price you pay for being an early adopter.

I have already written about my experiences with HCX in the past and as brilliant as the tool is for the migration of different workloads, the rework with the distributed firewall can be painful. Because what HCX can’t do is move firewall rules or security objects. Fortunately, there is a solution for this: ReSTNSX.

But what exactly is ReSTNSX? ReSTNSX describes itself as follows: The ReSTNSX platform simplifies consumption of VMware’s NSX API to increase visibility and reduce the risk of errors. Our mission is to enable administrators to easily configure and operate NSX with a straightforward and intuitive user experience. We will see in this post whether it is really that simple.

A quick note about transparency: I was provided with a free demo license, but I am not receiving any money and this does not influence my opinion. In this post, I’ll walk you through how RESTNSX complements HCX, especially when migrating applications with strict security policies or complex network requirements.

Why Combine HCX with ReSTNSX?

While HCX handles the heavy lifting of vMotion, replication, and network extension, it doesn’t touch NSX security rules or segment tagging. That’s where RESTNSX comes in. ReSTNSX can access various systems via API, including Checkpoint and Palo Alto firewalls (which are not the focus of this article), as well as external resources such as Active Directory and SMTP servers. The architecture is relatively simple: there is a Cloud Control OVA, and that’s it. It really is that simple.

Some key benefits:

• Automatically replicate security policies to the destination environment
• Tag migrated VMs for policy-based firewall rules
• Clean up or reassign segments and groups after cutover
• Automate post-migration tasks (e.g., updating DFW rules, creating new groups)

Getting started

To get started, you first need to deploy the OVA and register the product. The process is pretty straightforward, so I won’t go into detail here. Once the CloudControl OVA has been successfully deployed and registered, you need to register the data sources. To do this, go to Administration - Data Sources.

As you can see here, I have set up two NSX environments as data sources. NSXB is a pure NSX lab, and VCF02 is my VCF stretched cluster, which I have already discussed in my blog.

Then I got this test setup ready:

RESTNSX & HCX Test Scenario

The migration scenario involves moving workloads from a source environment named NSXB to a destination environment VCF02. The VCF02 NSX environment initially contains no custom firewall rules, tags, or groups—only the default settings provided by NSX.

Initial Environment (NSXB)

In the source NSXB environment, we have the following setup:

• Tag: dfg_alpine
• Security Group: dfg_all_Alpine
• Tagged VMs: Alpine1, Alpine2, Alpine3, and Alpine4

A custom firewall policy named ReSTNSX-Policy is configured as follows:

1. Allow ICMP to Alpine VMs
   Allow all ICMP traffic from any source to all VMs in the group dfg_all_Alpine.
   Applied to: All distributed firewall (DFW) VMs

2. Allow HTTP and HTTPS to WAN
   Allow HTTP and HTTPS traffic originating from VMs in the dfg_all_Alpine group to any external (non-RFC1918) destination (Internet).
   Applied to: dfg_all_Alpine

3. Allow SSH to Alpine VMs
   Allow SSH traffic (TCP port 22) from any source to all VMs in the dfg_all_Alpine group.
   Applied to: dfg_all_Alpine

4. Allow iPerf3 Between Alpine VMs
   Allow iPerf3 traffic (TCP port 5201) between all VMs within the dfg_all_Alpine group.
   Applied to: dfg_all_Alpine

5. Allow DNS from Alpine VMs
   Allow DNS traffic (TCP and UDP port 53) from all VMs in the dfg_all_Alpine group to any destination.
   Applied to: dfg_all_Alpine

6. Drop All from Alpine VMs
   Drop all other traffic originating from the VMs within the dfg_all_Alpine group to any destination.
   Applied to: All distributed firewall (DFW) VMs

Migration of the firewall rules and VMs

I am migrating the workloads (Alpine3-4) seamlessly to the VCF02 environment using VMware HCX. Of course, the VMs stop communicating after the migration because there are no firewall rules in the target environment yet, except for the default Layer 3, which is set to drop all on my system.

There are several ways to migrate my firewall ruleset to the target environment. I could import a ruleset using bulk provisioning and a CSV file, but in my opinion, the charm of this solution lies in the Policy Engine. Using the Policy Engine, which can be found under Tools in the menu, you can easily create a sync task that can synchronize a wide variety of tasks on a cyclical basis.

The screenshot shows my finished policy. It is relatively simple. I have selected to create a policy of type dFW Sections. By default, synchronization runs every 30 minutes when the policy is enabled. Under NSX Manager, I specify my source and target NSX Manager, and in the dFW Section field, I set the source policy. In my example, it is the previously created ReSTNSX policy from my NSXB environment. My destination is the VCF02 environment. Under Synchronization Method, I can specify various options for what exactly should be synchronized. Direction specifies the source and destination systems; a bi-directional sync is also possible. Further up in the screenshot, you can see the result of the preview sync. As you can see, sections, rules, security groups, static members, services, and context profiles are synchronized—that’s quite a lot. But first, a quick word about the synchronization method, as this has an impact on the actual implementation.

There are 4 synchronization options: Effective Members, Membership Criteria, Membership Criteria (MAT Mode), and Effective Members + Membership Criteria.

• Effective Members: This mode ensures that the realized members from the source group are written to the new group, which means that the new group receives a static IP assignment for its members. In this case, the membership criterion is an IP, even if it was previously a dynamic group. If the source VM does not have a realized IP, it is not part of the new group.

• Membership Criteria: In this mode, only the criteria for how the groups receive their members are synchronized. In the case of an IP-only group, the defined IPs are simply stored as criteria. This mode allows us to obtain our dynamic groups, but if TAGs are used as criteria, they must be created in the target environment (or synchronized with HCX), otherwise the groups will remain empty.

• Membership Criteria (MAT Mode): Only for V to T migration. Since I don’t have NSX-V in use, I couldn’t test anything here.

• Effective Members + Membership Criteria: This is a combination of both methods. I have my dynamic TAG definition and my effective members. This can be used if you cannot synchronize TAGs and want to make sure that communication works instantly in the new environment. However, this requires post-processing steps and is not necessary in my setup.

By default, ReSTNSX creates firewall sections, rules, and object names in the target environment that are identical to those in the source environment. This behavior can be changed when creating the policy.

A policy does not have to be activated; you can also start it manually with a forced sync and use this policy as a one-time synchronization. For long migration scenarios, performing the synchronization cyclically is invaluable, especially if there are rule adjustments during the migration phase. After successful synchronization, the complete policy is available 1:1 in my VCF02 and the migrated VMs alpine 3 and 4 are automatically in the dFG_all_Alpine group, as they retained their NSX security tags during the HCX migration and the groups created by ReSTNSX use the same definitions as in the source environment.

Additional features of ReSTNSX

ReSTNSX is not just a simple migration tool, although that was certainly the first use that came to mind. It can do much more than that. ReSTNSX has a very good, customizable dashboard that provides me with long-term information about my NSX environment.

Of course, the dashboard in my lab doesn’t show much, because I don’t have Aria Operations for Networks running, and I also shut down my labs after successful testing so as not to drive up my electricity bill even more. ReSTNSX also features a comprehensive reporting system and enables NSX documentation and auditing. Bulk Provisioning is a powerful tool for manipulating, creating, and detaching security tags, groups, firewall rules, and other items. You can even clone an entire NSX environment.

Conclusion

ReSTNSX is not just a simple migration tool, it also accelerates Day 2 operations in NSX and VCF. Combined with HCX, it has never been easier to migrate existing microsegmented workloads from one workload domain to a new workload domain or VCF instance. ReSTNSX uses APIs from vCenter and NSX and offers a centralized CLI. It also offers true multi-cloud management (VMware Cloud on AWS (VMCoAWS), Google (GCVE), Azure, Oracle, and IBM). I’m sure I’ve only scratched the surface, as some things are not so easy to test, especially when it comes to multi-cloud, but I hope I’ve been able to provide some insight into a cool tool that can make everyday life and migration tasks easier.

In NSX versions 4.1.x and 4.2.0, edge transport nodes and host transport nodes are instantiated using certificates with a validity period of only 825 days. This is obviously not desirable behavior and has been fixed in newer versions of NSX. Interestingly, I haven’t seen anything about this in the changelog. In NSX versions 3.X and 4.2.1 and higher, these certificates are valid for 10 years. But what exactly does that mean?

The Problem

First of all, don’t panic. The affected certificates are not visible in the GUI, but NSX Manager will issue a warning and display an error 30 days in advance. If you don’t see these messages in your NSX, you have more than 30 days to respond. Furthermore, it only affects transport nodes deployed with NSX 4.1.x - 4.2. If you have upgraded from NSX 3.X to one of the affected versions or have just deployed VCF version 5.2.x, you are safe.

Why It Matters

As the 825-day period approaches its end, you may encounter certificate expiration issues, potentially affecting NSX component communication and overall platform stability.

If you are affected, there are two scenarios.

• Scenario A) The certificates will expire sometime in the near future, you may have received a warning in NSX Manager and therefore found my blog, or you are not yet aware of the problem because the certificates will expire sometime between 31 and 825 days from now.
• Scenario B) The certificates have already expired and transport nodes are disconnected from NSX.

There is a solution for both scenarios. But let’s start with the simple scenario A first.

Find the problem

If you don’t know which NSX version you installed and want to check whether you are affected by this issue, Broadcom has a relatively simple solution. There is a script called CARR (Certificate Analyzer, Results and Recovery) that can be easily run on NSX Manager (NSX 4.1.x to 4.2) or on an external client (other NSX versions). Download Certificate Analyzer, Results and Recovery (CARR)from the official KB article.

In all cases, the script requires the following ports to be open between the client machine and the 3 NSX Managers

• ssh port 22
• https port 443
• corfu port 9000

If running on the NSX Manager directly, ports 443 and 9000 will already be open between the 3 Managers.

If you want to run the script on the NSX manager, then the script must be executed directly from the /root directory. To do this, copy the script to your NSX Manager or Global NSX Manager using sftp and unzip it with tar and run the script.

tar -xvf carr-1.15.tar.gz
cd carr-1.15
./start.sh -d

Script options:

• -o = this flag is used to force online mode

• -t = specify lead time for expiring certificates, between 31 and 825 days.

• -d = dryrun

I ran the script with the default settings (lead time for expiring 825 days) and the output was as follows:

CARR Script Validation Report

All validations done.

As you can see from the output for HOST and EDGE, my installation is not affected. However, it shows that my API and VIP certificate will expire in 680 days. CARR is also smart enough to recognize that it is a certificate signed by a CA and not a certificate issued by NSX itself.

To trigger TN cert replacement, environment details must be populated in a pre-existing file validation_config.yaml. This yaml file is located in the same folder as start.sh. My example yaml. I have disabled validation of all certificates except HOST and EDGE.

# user interface to provide the validation config
# user can specify if any certificate validation needs to be skipped.
# by default all certificate types will be validated.
# For Hosts , the vCenter cluster names for host must be specified. Script will validate hosts in those clusters only.
# For Edge node, Edge cluster name must be specified. Script will validate edge nodes in those clusters only.



HOST:
  validate: True
  clusters:
    - vcenter_name: vcf02-vcsa.lab.home
      vcenter_cluster_name: sfo-m01-cluster-001
EDGE:
  validate: True
  clusters: 
    - name: cl01
API:
  validate: False
VIP:
  validate: False
CBM-FILE-PERMISSIONS:
  validate: False
CBM_CORFU:
  validate: False
CBM_CLUSTER_MANAGER:
  validate: False
CORFU_SERVER:
  validate: False
CORFU_CLIENTS:
  validate: False
LOCAL-MANAGER-PI:
  validate: False
GLOBAL-MANAGER-PI:
  validate: False
STALE-CERTIFICATES:
  validate: False
APH_TN:
  validate: False
APH_AR:
  validate: False
CCP:
  validate: False
SITE-TO-SITE:
  validate: False
COMPUTE_MANAGER:
  validate: False