Cisco bug CSCvz43359: with GENEVE offload on VIC 14xx, overlay traffic can leave the wrong vNIC. Symptoms and the fix in UCS Manager 4.2(1l).
According Release Notes for Cisco UCS Manager, Release 4.2(1l) We have a fix for CSCvz43359 Traffic using GENEVE overlay sometimes leaves wrong VNIC when GENEVE Offload is enabled on VIC14xx:
Defect ID
Symptom
First Bundle Affected
Resolved in Release
The following caveats related to NSX-T are resolved in Release 4.2(1l)
CSCvz43359
On a Cisco UCS server using an NSX-T topology, data traffic using a GENEVE overlay sometimes left the wrong vNIC when GENEVE Offload was enabled on a VIC 1400 series Fabric Interconnect. This issue is resolved.
4.2(1d)C
4.2(1l)C
Traffic using GENEVE overlay sometimes leaves wrong VNIC when GENEVE Offload is enabled on VIC14xx
Symptom: Rapid mac moves observed on Fabric Interconnect and northbound switches where mac address belongs to device using GENEVE overlay. pkcatp-uw in ESXi kernel was not able to observe this phenomenon. This was only observable via tcpdump on the physical VIC adapter in the debug shell.
Conditions: This was specifically seen in an NSX-T topology though more general use of GENEVE offloading in the hardware would likely show same behavior. The NSX-T TEP mac addresses should be ‘bound’ to a physical interface unless there is a topology change. In this circumstance, we observed the TEP macs rapidly moving from Fabric A to Fabric B and vice versa while the teaming/load balancing policy was set to Active/Active in ESXi and NSX. NSX-T uses BFD Control frames between hosts and BFD leverages GENEVE. When GENEVE Offloading is enabled in the VIC adapter policy, this causes some small number of these BFD frames to egress the wrong physical link which causes the unexpected mac move behavior on northbound devices.
ESXi PSOD “#PF Exception 14 in world xxxx:nsx-cfgagent” in NSX-T 3.1.3.x: how to identify it, cause, workaround and the fix in NSX-T 3.1.3.7 / VCF 4.4.1.
Short answer: “#PF Exception 14 in world xxxx:nsx-cfgagent” is a purple screen (PSOD) on ESXi hosts running NSX-T Data Center 3.1.3 or 3.1.3.x. In the message, xxxx stands for the world ID. It is triggered by bulk vMotions, which corrupt an internal data structure in the NSX distributed firewall. The fix is NSX-T 3.1.3.7 or later, or VCF 4.4.1 (NSX-T 3.1.3.7.4). The workaround is to set DRS to manual and avoid bulk vMotions. The original VMware KB 87352 is now Broadcom KB 318942.
What “xxxx:nsx-cfgagent” in the PSOD means
The KB title writes xxxx because the number changes on every crash. It is the ID of the ESXi world (thread) that was running when the host panicked. nsx-cfgagent is the NSX configuration agent on the host, and the frames in the backtrace belong to the NSX distributed firewall module (nsxt-vsip-<build>). A real panic message from the KB looks like this:
Panic Message: @BlueScreen: #PF Exception 14 in world 58524293:nsx-cfgagent IP 0x4180320430b3 addr 0x10
Back trace:
0x451ac5c1b158:[0x4180320430b3]rn_walktree@(nsxt-vsip-19068435)#<None>+0x5b stack: 0x43261e082c18, ...
The KB also lists a second form of the same bug. There the world is a NetWorld-VM- thread, and the backtrace goes through the packet filter functions:
Panic Message: @BlueScreen: #PF Exception 14 in world 99566160:NetWorld-VM- IP 0x41801bf58a33 addr 0x0
... pfp_policy_lookup@(nsxt-vsip-18504670) ...
... pf_test_tcp@(nsxt-vsip-18504670) ...
... PFFilterPacket@(nsxt-vsip-18504670) ...
... VSIPDVFProcessPacketsInt@(nsxt-vsip-18504670) ...
#PF Exception 14 is a CPU page fault. It says nothing about the cause by itself; many different bugs produce one. What identifies this specific issue is the combination of three things:
the world name nsx-cfgagent (or NetWorld-VM-)
nsxt-vsip frames such as rn_walktree or pfp_policy_lookup in the backtrace
an NSX-T 3.1.3.x version on the host
How to confirm it in a PSOD screenshot or vmkernel.log
Read the first panic line. Take it from the purple screen (a screenshot from iLO, iDRAC or CIMC is enough). Look for #PF Exception 14 in world <number>:nsx-cfgagent.
Check the backtrace module. The frames should reference nsxt-vsip-<build>, for example rn_walktree, pfp_policy_lookup or PFFilterPacket.
Check vmkernel.log. After the host is back, look in /var/run/log/vmkernel.log around the vMotion import. The KB shows firewall state import messages like the ones below.
Check what was happening. Correlate with vCenter tasks. The crash follows a burst of vMotions, for example a DRS rebalance, maintenance mode or a host evacuation.
ImportStateTLV entry type 12, len 52, cnt 1
Importing from source version RELEASEbuild-19068435
pfr_unroute_kentry: delete failed.
pfp_add_table_one_addr: failed to add ke
rn_addmask: mask impossibly already in tree
pfp_add_addr_with_rule: failed
VSIPConversionCreateRuleSet: Cannot insert #1060 rule 26238: 22
pf_rollback_rules: rs_num: 1, anchor: mainrs
Affected versions and trigger scenarios
Item
Per Broadcom KB 318942
NSX version
NSX-T Data Center 3.1.3 and 3.1.3.x
ESXi
6.7, 7.x
VCF
4.x
Cause
Corruption in an internal data structure of the firewall code
Fixed in
NSX-T Data Center 3.1.3.7 and later; VCF 4.4.1 (NSX-T 3.1.3.7.4)
Workaround
Set DRS to manual on the cluster and avoid bulk vMotions
The KB lists these probable triggers during bulk vMotions:
Migrating many VMs that each have multiple vNICs.
Multiple IP sets in CIDR form per firewall rule.
Multiple rules that contain the same IP sets.
VMs moving from a host that has not been upgraded to a host that has been upgraded. This makes the upgrade window, when many hosts are evacuated, the riskiest time.
Fix: which NSX-T build to go to
When I first wrote this post in 2022, the guidance was to go to 3.1.3.6.1. The current Broadcom KB now says the issue is resolved in NSX-T Data Center 3.1.3.7 and later. It also still points to a separate problem in 3.1.3.6: KB 318290 (old 87627), where an Edge with an L4 load balancer stops passing traffic because of a datapath memory leak. That issue is also resolved in 3.1.3.7. In practice:
Standalone NSX-T 3.1.3.x: upgrade to 3.1.3.7 or later. Don’t stop at 3.1.3.6.
VCF 4.x: upgrade to VCF 4.4.1 (NSX-T 3.1.3.7.4). If you can’t, the KB tells you to open a Support Request before you patch NSX-T manually outside SDDC Manager.
Until you upgrade: set DRS to manual and move VMs in small batches.
Keep the support status in mind as well. Broadcom KB 421705 lists NSX 3.1.x End of Service as 7 April 2025. Any environment that still hits this PSOD is running an out-of-support NSX release, and the real fix is an upgrade to a supported NSX version.
Tweet Thank you to everyone who applied for vExpert and to the vExpert PROs for managing the voting process, it’s a lot of work! We are pleased to announce the list of 2022 vExperts. You can visit the vExpert Directory to see the list and profiles of each vExpert. All of the new and returning … Continued The post vExpert 2022 Awards Announcement appeared first on VMware vExpert Blog.
