xxxx:nsx-cfgagent PSOD: “#PF Exception 14 in world xxxx:nsx-cfgagent” During Bulk vMotions in NSX-T 3.1.3.x (KB 87352 / 318942)

ESXi PSOD “#PF Exception 14 in world xxxx:nsx-cfgagent” in NSX-T 3.1.3.x: how to identify it, cause, workaround and the fix in NSX-T 3.1.3.7 / VCF 4.4.1.

Short answer: “#PF Exception 14 in world xxxx:nsx-cfgagent” is a purple screen (PSOD) on ESXi hosts running NSX-T Data Center 3.1.3 or 3.1.3.x. In the message, xxxx stands for the world ID. It is triggered by bulk vMotions, which corrupt an internal data structure in the NSX distributed firewall. The fix is NSX-T 3.1.3.7 or later, or VCF 4.4.1 (NSX-T 3.1.3.7.4). The workaround is to set DRS to manual and avoid bulk vMotions. The original VMware KB 87352 is now Broadcom KB 318942.

What “xxxx:nsx-cfgagent” in the PSOD means

The KB title writes xxxx because the number changes on every crash. It is the ID of the ESXi world (thread) that was running when the host panicked. nsx-cfgagent is the NSX configuration agent on the host, and the frames in the backtrace belong to the NSX distributed firewall module (nsxt-vsip-<build>). A real panic message from the KB looks like this:

Panic Message: @BlueScreen: #PF Exception 14 in world 58524293:nsx-cfgagent IP 0x4180320430b3 addr 0x10
Back trace:
  0x451ac5c1b158:[0x4180320430b3]rn_walktree@(nsxt-vsip-19068435)#<None>+0x5b stack: 0x43261e082c18, ...

The KB also lists a second form of the same bug. There the world is a NetWorld-VM- thread, and the backtrace goes through the packet filter functions:

Panic Message: @BlueScreen: #PF Exception 14 in world 99566160:NetWorld-VM- IP 0x41801bf58a33 addr 0x0
... pfp_policy_lookup@(nsxt-vsip-18504670) ...
... pf_test_tcp@(nsxt-vsip-18504670) ...
... PFFilterPacket@(nsxt-vsip-18504670) ...
... VSIPDVFProcessPacketsInt@(nsxt-vsip-18504670) ...

#PF Exception 14 is a CPU page fault. It says nothing about the cause by itself; many different bugs produce one. What identifies this specific issue is the combination of three things:

  • the world name nsx-cfgagent (or NetWorld-VM-)
  • nsxt-vsip frames such as rn_walktree or pfp_policy_lookup in the backtrace
  • an NSX-T 3.1.3.x version on the host

How to confirm it in a PSOD screenshot or vmkernel.log

  1. Read the first panic line. Take it from the purple screen (a screenshot from iLO, iDRAC or CIMC is enough). Look for #PF Exception 14 in world <number>:nsx-cfgagent.
  2. Check the backtrace module. The frames should reference nsxt-vsip-<build>, for example rn_walktree, pfp_policy_lookup or PFFilterPacket.
  3. Check vmkernel.log. After the host is back, look in /var/run/log/vmkernel.log around the vMotion import. The KB shows firewall state import messages like the ones below.
  4. Check what was happening. Correlate with vCenter tasks. The crash follows a burst of vMotions, for example a DRS rebalance, maintenance mode or a host evacuation.
ImportStateTLV entry type 12, len 52, cnt 1
Importing from source version RELEASEbuild-19068435
pfr_unroute_kentry: delete failed.
pfp_add_table_one_addr: failed to add ke
rn_addmask: mask impossibly already in tree
pfp_add_addr_with_rule: failed
VSIPConversionCreateRuleSet: Cannot insert #1060 rule 26238: 22
pf_rollback_rules: rs_num: 1, anchor: mainrs

Affected versions and trigger scenarios

ItemPer Broadcom KB 318942
NSX versionNSX-T Data Center 3.1.3 and 3.1.3.x
ESXi6.7, 7.x
VCF4.x
CauseCorruption in an internal data structure of the firewall code
Fixed inNSX-T Data Center 3.1.3.7 and later; VCF 4.4.1 (NSX-T 3.1.3.7.4)
WorkaroundSet DRS to manual on the cluster and avoid bulk vMotions

The KB lists these probable triggers during bulk vMotions:

  • Migrating many VMs that each have multiple vNICs.
  • Multiple IP sets in CIDR form per firewall rule.
  • Multiple rules that contain the same IP sets.
  • VMs moving from a host that has not been upgraded to a host that has been upgraded. This makes the upgrade window, when many hosts are evacuated, the riskiest time.

Fix: which NSX-T build to go to

When I first wrote this post in 2022, the guidance was to go to 3.1.3.6.1. The current Broadcom KB now says the issue is resolved in NSX-T Data Center 3.1.3.7 and later. It also still points to a separate problem in 3.1.3.6: KB 318290 (old 87627), where an Edge with an L4 load balancer stops passing traffic because of a datapath memory leak. That issue is also resolved in 3.1.3.7. In practice:

  • Standalone NSX-T 3.1.3.x: upgrade to 3.1.3.7 or later. Don’t stop at 3.1.3.6.
  • VCF 4.x: upgrade to VCF 4.4.1 (NSX-T 3.1.3.7.4). If you can’t, the KB tells you to open a Support Request before you patch NSX-T manually outside SDDC Manager.
  • Until you upgrade: set DRS to manual and move VMs in small batches.

Keep the support status in mind as well. Broadcom KB 421705 lists NSX 3.1.x End of Service as 7 April 2025. Any environment that still hits this PSOD is running an out-of-support NSX release, and the real fix is an upgrade to a supported NSX version.

Sources

Related on vdan.cz

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere.