Daniel Micanek virtual Blog – Like normal Dan, but virtual.
VMware Tanzu Community Edition now includes advanced software supply chain tooling that helps application teams deliver software more rapidly, securely, and efficiently at scale. The 0.11 release of Tanzu Community Edition, available today, introduces new supply chain choreography capabilities […]
Cisco bug CSCvz43359: with GENEVE offload on VIC 14xx, overlay traffic can leave the wrong vNIC. Symptoms and the fix in UCS Manager 4.2(1l).
According Release Notes for Cisco UCS Manager, Release 4.2(1l) We have a fix for CSCvz43359 Traffic using GENEVE overlay sometimes leaves wrong VNIC when GENEVE Offload is enabled on VIC14xx:
| Defect ID | Symptom | First Bundle Affected | Resolved in Release |
|---|
| CSCvz43359 | On a Cisco UCS server using an NSX-T topology, data traffic using a GENEVE overlay sometimes left the wrong vNIC when GENEVE Offload was enabled on a VIC 1400 series Fabric Interconnect. This issue is resolved. | 4.2(1d)C | 4.2(1l)C |
Symptom: Rapid mac moves observed on Fabric Interconnect and northbound switches where mac address belongs to device using GENEVE overlay. pkcatp-uw in ESXi kernel was not able to observe this phenomenon. This was only observable via tcpdump on the physical VIC adapter in the debug shell.
Conditions: This was specifically seen in an NSX-T topology though more general use of GENEVE offloading in the hardware would likely show same behavior. The NSX-T TEP mac addresses should be ‘bound’ to a physical interface unless there is a topology change. In this circumstance, we observed the TEP macs rapidly moving from Fabric A to Fabric B and vice versa while the teaming/load balancing policy was set to Active/Active in ESXi and NSX. NSX-T uses BFD Control frames between hosts and BFD leverages GENEVE. When GENEVE Offloading is enabled in the VIC adapter policy, this causes some small number of these BFD frames to egress the wrong physical link which causes the unexpected mac move behavior on northbound devices.
Links:
ESXi PSOD “#PF Exception 14 in world xxxx:nsx-cfgagent” in NSX-T 3.1.3.x: how to identify it, cause, workaround and the fix in NSX-T 3.1.3.7 / VCF 4.4.1.
Short answer: “#PF Exception 14 in world xxxx:nsx-cfgagent” is a purple screen (PSOD) on ESXi hosts running NSX-T Data Center 3.1.3 or 3.1.3.x. In the message, xxxx stands for the world ID. It is triggered by bulk vMotions, which corrupt an internal data structure in the NSX distributed firewall. The fix is NSX-T 3.1.3.7 or later, or VCF 4.4.1 (NSX-T 3.1.3.7.4). The workaround is to set DRS to manual and avoid bulk vMotions. The original VMware KB 87352 is now Broadcom KB 318942.
The KB title writes xxxx because the number changes on every crash. It is the ID of the ESXi world (thread) that was running when the host panicked. nsx-cfgagent is the NSX configuration agent on the host, and the frames in the backtrace belong to the NSX distributed firewall module (nsxt-vsip-<build>). A real panic message from the KB looks like this:
Panic Message: @BlueScreen: #PF Exception 14 in world 58524293:nsx-cfgagent IP 0x4180320430b3 addr 0x10
Back trace:
0x451ac5c1b158:[0x4180320430b3]rn_walktree@(nsxt-vsip-19068435)#<None>+0x5b stack: 0x43261e082c18, ...The KB also lists a second form of the same bug. There the world is a NetWorld-VM- thread, and the backtrace goes through the packet filter functions:
Panic Message: @BlueScreen: #PF Exception 14 in world 99566160:NetWorld-VM- IP 0x41801bf58a33 addr 0x0
... pfp_policy_lookup@(nsxt-vsip-18504670) ...
... pf_test_tcp@(nsxt-vsip-18504670) ...
... PFFilterPacket@(nsxt-vsip-18504670) ...
... VSIPDVFProcessPacketsInt@(nsxt-vsip-18504670) ...#PF Exception 14 is a CPU page fault. It says nothing about the cause by itself; many different bugs produce one. What identifies this specific issue is the combination of three things:
nsx-cfgagent (or NetWorld-VM-)nsxt-vsip frames such as rn_walktree or pfp_policy_lookup in the backtrace#PF Exception 14 in world <number>:nsx-cfgagent.nsxt-vsip-<build>, for example rn_walktree, pfp_policy_lookup or PFFilterPacket./var/run/log/vmkernel.log around the vMotion import. The KB shows firewall state import messages like the ones below.ImportStateTLV entry type 12, len 52, cnt 1
Importing from source version RELEASEbuild-19068435
pfr_unroute_kentry: delete failed.
pfp_add_table_one_addr: failed to add ke
rn_addmask: mask impossibly already in tree
pfp_add_addr_with_rule: failed
VSIPConversionCreateRuleSet: Cannot insert #1060 rule 26238: 22
pf_rollback_rules: rs_num: 1, anchor: mainrs| Item | Per Broadcom KB 318942 |
|---|---|
| NSX version | NSX-T Data Center 3.1.3 and 3.1.3.x |
| ESXi | 6.7, 7.x |
| VCF | 4.x |
| Cause | Corruption in an internal data structure of the firewall code |
| Fixed in | NSX-T Data Center 3.1.3.7 and later; VCF 4.4.1 (NSX-T 3.1.3.7.4) |
| Workaround | Set DRS to manual on the cluster and avoid bulk vMotions |
The KB lists these probable triggers during bulk vMotions:
When I first wrote this post in 2022, the guidance was to go to 3.1.3.6.1. The current Broadcom KB now says the issue is resolved in NSX-T Data Center 3.1.3.7 and later. It also still points to a separate problem in 3.1.3.6: KB 318290 (old 87627), where an Edge with an L4 load balancer stops passing traffic because of a datapath memory leak. That issue is also resolved in 3.1.3.7. In practice:
Keep the support status in mind as well. Broadcom KB 421705 lists NSX 3.1.x End of Service as 7 April 2025. Any environment that still hits this PSOD is running an out-of-support NSX release, and the real fix is an upgrade to a supported NSX version.
vExpert 2022 Awards Announcement
Tweet Thank you to everyone who applied for vExpert and to the vExpert PROs for managing the voting process, it’s a lot of work! We are pleased to announce the list of 2022 vExperts. You can visit the vExpert Directory to see the list and profiles of each vExpert. All of the new and returning … Continued The post vExpert 2022 Awards Announcement appeared first on VMware vExpert Blog.
Potentially interesting VMware Homelab Kits for 2022
I am always on the lookout for new and interesting hardware platforms that can benefit the VMware Homelab Community. In the month of January, there were a number of new announcements that I had been following including news from the Computer Electronic Show (CES) but also a couple of product releases from vendors that I […]
Boot ESXi from a RAID 1 M.2 volume on Cisco UCS-M2-HWRAID: controller limits, UCS Manager, Intersight and CIMC steps, and ESXi 8/9 boot device rules.
Short answer: UCS-M2-HWRAID is Cisco’s boot-optimized M.2 RAID controller for UCS M5 and M6 servers. It holds two identical M.2 SATA SSDs and supports only RAID 1 or JBOD (default JBOD). It boots only in UEFI mode, and one virtual drive always uses the full capacity. To boot ESXi, create a RAID 1 LUN in UCS Manager, Intersight or CIMC, set a UEFI boot policy to local disk (MSTOR-RAID), and install ESXi to that volume.
The Cisco Boot-Optimized M.2 RAID Controller (PID UCS-M2-HWRAID) is a module that plugs into the mini-storage connector on the motherboard. It has two slots for SATA M.2 SSDs and an integrated 6 Gbps SATA RAID controller. In UCS Manager, Intersight and CIMC the controller shows as MSTOR-RAID. Its job is to give ESXi a mirrored local boot device without using the front drive bays or an SD card.
| Property | Value (Cisco documentation) |
|---|---|
| Servers | UCS C220 M5, C240 M5, C480 M5, B200 M5, B480 M5 (UCS Manager 4.1 guide); also required for M.2 SATA on C220 M6 (spec sheet) |
| Minimum firmware | UCS Manager 4.0(4a) for the RAID procedure (tech note); Cisco IMC / UCS Manager 4.2(1) on M6 |
| M.2 drives | 240 GB and 960 GB SATA M.2 (UCSM 4.1 guide); C220 M6 spec sheet lists UCS-M2-240G, UCS-M2-480G, UCS-M2-960G |
| RAID levels | RAID 1 or JBOD only (default JBOD) |
| Boot mode | UEFI only; legacy boot not supported |
| Virtual drives | One, using the entire disk capacity |
| Mixing drives | Not supported (use two identical SSDs) |
| Hot-plug | Not supported; power the server off to replace a drive (M6 spec sheet) |
Other UCS Manager limitations: the Local Disk Configuration policy is not supported, the LUN must be created in the Local LUNs tab of a storage profile (not with controller definitions), and existing LUN migration is not supported.
These screenshots are from my UCS Manager setup.
Create a policy (here M.2-RAID1) with RAID Level RAID 1 Mirrored, 2 drives, drive type SSD.
If the service profile association fails with Server does not fulfill Local Disk config requirements, Cisco’s tech note says to use Disk Group Configuration (Manual) with the two M.2 slots and to disable pSATA. With manual configuration, leave the span ID unspecified and set the disk role to normal.
Add a local LUN, select the disk group policy and enable Expand To Available. The controller supports only one LUN over the whole mirror anyway.
In the service profile, open Storage > Storage Profiles > Modify Storage Profile and select the profile. The LUN shows Not Applied until the profile is associated.
Use a boot policy with boot mode UEFI (the controller does not boot in legacy mode) and put the local disk first.
MStorBootVd or set your own. Select the M.2 controller slot: MSTOR-RAID-1 for C-Series and X-Series. Only the B200 M6 can have a second controller (MSTOR-RAID-2).Without CIMC access, the same controller menu is in BIOS setup: press F2 during POST, then go to Advanced > Cisco Boot optimized M.2 RAID Controller.
Cisco does not ship a separate ESXi driver download for this controller in the documents above. Whether a given ESXi release is supported on your server with this controller is defined by the Cisco UCS Hardware and Software Compatibility tool (ucshcltool.cloudapps.cisco.com) and the Broadcom Compatibility Guide. Check both for your exact server, firmware bundle and ESXi build before you upgrade. Install from the Cisco custom ESXi ISO so that the Cisco drivers match the firmware.
After installation, confirm which driver ESXi bound to the controller, and that the boot device is the RAID volume:
esxcli storage core adapter list
esxcli storage core device listBroadcom KB 317631 (formerly 85685, “SD card/USB boot device revised guidance”) is the reference for boot media. A RAID 1 M.2 SATA SSD volume is exactly the kind of local persistent boot device it recommends:
| Requirement | ESXi 8.x | ESX 9.x |
|---|---|---|
| Boot device size | 32 GB minimum, 128 GB recommended | 128 GB minimum |
| Endurance | 128 TBW over 5 years | |
| IO performance | At least 100 MB/s | |
| Media | PCIe NVMe, SAS, SATA/SATADOM SSD; industrial-grade flash (SLC or pSLC) | |
| SD card / USB | Still supported through VCF 9.0 on previously certified servers, but not recommended; new platforms are not certified with SD/USB boot | |
Every M.2 size listed above (240, 480 or 960 GB) meets the ESX 9 minimum of 128 GB. Check the endurance rating of your specific SSD model against the 128 TBW requirement. On ESXi 7.x, SD cards can still hold all partitions, but Broadcom strongly recommends moving to a persistent device. If you are replacing SD cards on M5/M6 servers, UCS-M2-HWRAID is the natural choice.
Add Cisco UCS fault F1705 (Rank VLS) as a custom fault in UCS Manager Plugin 3.0(6) for the vSphere HTML client, for Proactive HA.
New Cisco UCS firmware brings possibility to have notification about F1705 Alerts – Rank VLS.
In latest version of Cisco UCS Manager Plugin for VMware vSphere HTML Client (Version 3.0(6)) we could add Custom fault addition for proactive HA monitoring. How to do it?
Adding Custom Alert is only possible with unregistered Cisco UCS Provider, it is better to do it immediatly after Cisco UCS Manager Plugin instalation.
Now I can deceided If I will block F1705 or NOT. I personaly preffer to have F1705 Alert under Proactive HA. Then I only restart Blades with F1705. During reboot Hard-PPR permanently remaps accesses from a designated faulty row to a designated spare row.
Links:
The fastest workaround for Log4j CVE-2021-44228 (VMSA-2021-0028) on vCenter Server 7.0, 6.7 and 6.5, and why patching is the real fix.
Apache Log4j open source component has security bug (CVE-2021-44228 – VMSA-2021-0028). It is neccesary to fix vCenter Server 7.0.x, vCenter 6.7.x & vCenter 6.5.x.
Fastest and recommended is workaround with KB 87081 script (vc_log4j_mitigator.py).
Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"
Command> shell
Shell access is granted to root
root@localhost [ ~ ]# cd /tmp
root@localhost [ /tmp ]# vim vc_log4j_mitigator.py
root@localhost [ /tmp ]# python vc_log4j_mitigator.py
2021-12-21T10:38:20 INFO main: Script version: 1.6.0
2021-12-21T10:38:20 INFO main: vCenter type: Version: 7.0.2.00500; Build: 18455184; Deployment type: embedded; Gateway: False; VCHA: False; Windows: False;
A service stop and start is required to complete this operation. Continue?[y]y
2021-12-21T10:38:23 INFO stop: stopping services
2021-12-21T10:38:46 INFO process_jar: Found a VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar
2021-12-21T10:38:46 INFO backup_file: VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar backed up to /tmp/tmpxi89fco8/opt/vmware/lib64/log4j-core-2.13.0.jar.bak
2021-12-21T10:38:47 INFO process_jar: VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar backed up to /tmp/tmpxi89fco8/opt/vmware/lib64/log4j-core-2.13.0.jar.bak
2021-12-21T10:39:03 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar
2021-12-21T10:39:03 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:04 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:04 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar
2021-12-21T10:39:04 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:04 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:06 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar
2021-12-21T10:39:06 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar.bak
2021-12-21T10:39:06 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar.bak
2021-12-21T10:39:07 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar
2021-12-21T10:39:07 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak
2021-12-21T10:39:07 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak
2021-12-21T10:39:08 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:08 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:08 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:14 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:14 INFO backup_file: VULNERABLE FILE: /tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:15 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war
2021-12-21T10:39:15 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war.bak
2021-12-21T10:39:15 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war.bak
2021-12-21T10:39:15 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:15 INFO backup_file: VULNERABLE FILE: /tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:16 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war
2021-12-21T10:39:16 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war.bak
2021-12-21T10:39:16 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war.bak
2021-12-21T10:39:16 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:16 INFO backup_file: VULNERABLE FILE: /tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:17 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war
2021-12-21T10:39:17 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war.bak
2021-12-21T10:39:18 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war.bak
2021-12-21T10:39:21 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:21 INFO backup_file: VULNERABLE FILE: /tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:22 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war
2021-12-21T10:39:22 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war.bak
2021-12-21T10:39:24 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war.bak
2021-12-21T10:39:25 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:25 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:26 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:28 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar
2021-12-21T10:39:28 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:29 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:32 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:32 INFO backup_file: VULNERABLE FILE: /tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:33 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-lookupsvc/webapps/ROOT.war
2021-12-21T10:39:33 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT.war.bak
2021-12-21T10:39:34 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT.war.bak
2021-12-21T10:39:34 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:35 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:35 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:37 INFO _patch_file: Found VULNERABLE FILE: /usr/lib/vmware-vmon/java-wrapper-vmon
2021-12-21T10:39:37 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-vmon/java-wrapper-vmon backed up to /tmp/tmpxi89fco8/usr/lib/vmware-vmon/java-wrapper-vmon.bak
2021-12-21T10:39:37 INFO patch_vum: Found a VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/start.ini
2021-12-21T10:39:37 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/start.ini backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak
2021-12-21T10:39:37 INFO print_summary:
===== Summary =====
Backup Directory: /tmp/tmpxi89fco8
List of processed java archive files:
/opt/vmware/lib64/log4j-core-2.13.0.jar
/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar
/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar
/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar
/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar
/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar
/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war
/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war
/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war
/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war
/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar
/usr/lib/vmware-lookupsvc/webapps/ROOT.war
/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
List of processed configuration files:
/usr/lib/vmware-vmon/java-wrapper-vmon
/usr/lib/vmware-updatemgr/bin/jetty/start.ini
Total fixed: 16
NOTE: Running this script again with the --dryrun
flag should now yield 0 vulnerable files.
Log file: /var/log/vmsa-2021-0028_2021_12_21_10_38_20.log
===========================
2021-12-21T10:39:37 INFO start: starting services
2021-12-21T10:52:47 INFO main: Done.
root@localhost [ /tmp ]# python vc_log4j_mitigator.py -r 2021-12-21T11:10:01 INFO main: Script version: 1.6.0 2021-12-21T11:10:01 INFO main: vCenter type: Version: 7.0.2.00500; Build: 18455184; Deployment type: embedded; Gateway: False; VCHA: False; Windows: False; 2021-12-21T11:10:01 INFO main: Running in dryrun mode. 2021-12-21T11:11:01 INFO print_summary: ===== Summary ===== No vulnerable files found! Total found: 0 Log file: /var/log/vmsa-2021-0028_2021_12_21_11_10_01.log =========================== 2021-12-21T11:11:01 INFO main: Done.
root@localhost [ /tmp ]# python vc_log4j_mitigator.py -h
usage: vc_log4j_mitigator.py [-h] [-d dirnames [dirnames ...]] [-a] [-r] [-b BACKUP_DIR] [-l LOG_DIR]
VMSA-2021-0028 vCenter tool; Version: 1.6.0 This tool deletes the JndiLookup.class file from *.jar and *.war files. On Windows systems the tool will by default traverse the folders identified by the VMWARE_CIS_HOME, VMWARE_CFG_DIR, VMWARE_DATA_DIR and VMWARE_RUNTIME_DATA_DIR
variables. On vCenter Appliances the tool will search by default from the root of the filesystem. All modified files are backed up if the process needs to be reversed due to an error.
optional arguments:
-h, --help show this help message and exit
-d dirnames [dirnames ...], --directories dirnames [dirnames ...]
space separated list of directories to check recursively for CVE-2021-44228 vulnerable java archive files.
-a, --accept-services-restart
accept the restart of the services without having manual prompt confirmation for the same
-r, --dryrun Run the script and log vulnerable files without mitigating them. The vCenter services are not restarted with this option.
-b BACKUP_DIR, --backup-dir BACKUP_DIR
Specify a backup directory to store original files.
-l LOG_DIR, --log-dir LOG_DIR
Specify a directory to store log files.
Links:
On the 1st day of #12DaysOfK8s, let’s get started with the basics: What is Kubernetes? ☸️ Find out in 5 minutes or less by watching this lightboard video with @vstegeler. More to come tomorrow!
NSX Edge design on Cisco UCS for NSX 4.x and VCF 9: Edge VM vs bare metal, vNIC pairs without fabric failover, TEP MTU and VIC Geneve offload.
Short answer: On Cisco UCS, NSX Edge VMs usually go on a small, dedicated ESX edge cluster. Give each Edge a pair of vNICs, one pinned to Fabric A and one to Fabric B, with UCS fabric failover disabled. Redundancy then comes from the Edge’s fp-eth active/standby mapping, not from the Fabric Interconnect. Keep TEP MTU consistent: 1700 or more, ideally 9000 end to end. Enable Geneve offload in the VIC adapter policy for the host transport nodes. For VCF 9, follow the VCF Edge Cluster design library.
I wrote this in 2021 for NSX-T 3.x, because Cisco’s design guides did not cover Edge placement. This September 2026 refresh covers NSX 4.2 and VCF 9.0/9.1, checked against Broadcom TechDocs and Cisco documentation. Parts that apply only to the 2021 design are labelled.
| Version | Design reference |
|---|---|
| NSX-T 3.x (2021, this post’s original basis) | VMware NSX-T Reference Design Guide. The old communities.vmware.com links no longer work. |
| NSX 4.2 standalone / VCF 5.2 | NSX Reference Design Guide 4.2 (linked from Broadcom KB 412786) |
| VCF 9.x | VCF design library: NSX Edge Cluster Detailed Design, plus the VCF Virtual Networking Technical Reference, which replaces the NSX Design Guide from VCF 9.0 |
| Edge VM | Bare-metal Edge | |
|---|---|---|
| Deployment in VCF 9 | From vCenter / NSX. Passwords managed by VCF Operations. | Installed from ISO or PXE, then registered manually with NSX Manager |
| Data path interfaces | 4 datapath vNICs | Up to 16 physical interfaces |
| Strengths | Easy to scale up, vSphere HA recovery, enough performance for most cases | Better small-packet performance, can use several 100G links |
| UCS specifics | Runs on any UCS ESX host. Uses VIC vNICs through the VDS. | The VCF 9.0 NIC list includes Cisco VIC 1300 and 1400 series (VIC 15000 is not listed). You must configure enough RX/TX queues in UCS Manager for one queue per datapath core. The CPU needs AES-NI and 1 GB huge pages. |
Edge VM sizes (VCF 9.0 docs): Small 2 vCPU / 4 GB (proof of concept only), Medium 4 vCPU / 8 GB (L2 to L4 under 2 Gbps), Large 8 vCPU / 32 GB (2 to 10 Gbps, L7 load balancing, TLS inspection), Extra Large 16 vCPU / 64 GB (VPN, IDS/IPS, URL filtering, malware prevention). All sizes use a 200 GB disk.
The VCF 9.1 edge cluster design recommends a dedicated vSphere cluster for Edges. It limits Edges per host to the number of available physical NICs and gives each Edge its own pair of pNICs. fp-eth0 maps to pNIC 1 active / pNIC 2 standby, and fp-eth1 maps to pNIC 2 active / pNIC 1 standby. Each Edge peers BGP only with the ToRs in its own rack, with BFD. On an FI-attached UCS server, “pNIC 1 / pNIC 2” means a Fabric A vNIC and a Fabric B vNIC:
edge1-A on Fabric A and edge1-B on Fabric B), plus separate vNICs for host management, vMotion and storage.
The 2021 named teaming example still applies: two policies, PIN-TO-TOR-LEFT (“uplink-1” active) and PIN-TO-TOR-RIGHT (“uplink-2” active), one per external VLAN segment. On UCS, left and right become Fabric A and Fabric B.
NSX requires an MTU of at least 1600 bytes and recommends 1700 to allow for Geneve header growth. For better throughput, set the global TEP MTU, uplink profile MTU and VDS MTU to 9000 when the underlay supports jumbo frames. VCF also requires the Edge TEP MTU to match the host TEP MTU in the same overlay transport zone. On UCS, set the vNIC MTU to 9000, as Cisco’s VIC best practices recommend. The QoS system class and the upstream switches must also carry jumbo frames. Guest VMs can use up to 8800 on a 9000 underlay.
nenic-ens driver). Supported on VIC 15000 from 4.2(3). Not supported on VIC 1200/1300. Geneve offload is disabled by default.nenic driver). Check it on the host per Broadcom KB 397644:vsish -e get /net/pNics/vmnic0/properties | grep "Driver\|Device Hard\|Device Soft"
# look for VMNET_CAP_GENEVE_OFFLOAD in "Device Hardware Cap Activated"
# vmkernel.log: "nenic: ... Geneve: Overlay offload enabled, udp_port = 6081"Use the nenic driver and VIC firmware pair listed in the Broadcom Compatibility Guide for your ESX release. The history (nenic 1.0.35.0 on ESXi 7.0) is covered in my nenic post linked below.
The 2021 Dell EMC VxBlock NSX-T design put the edge on a C220 M5 host. vmnic0/1 were a Cisco VIC 1457 to the FIs for management, vmnic2/3 an Intel XXV710 for TEP/overlay, and vmnic4/5 a second XXV710 for BGP peering, cabled directly to the Nexus 9000 pair. The pattern is still valid when Edge traffic should bypass the FI domain, and it fits VCF’s dedicated-host, dedicated-pNIC recommendation.
For a PoC or homelab, configure FI uplink Ethernet ports for the edge VLANs and bind a vNIC template to them. For production, use the vNIC pairs described above.
