Short answer: When VMs on the same NSX overlay segment can talk while they sit on one ESX host, but not when they run on different hosts, the Geneve tunnel between the two hosts’ TEPs is not working. Either the TEPs cannot reach each other, or the path between them does not carry the larger encapsulated packets because of an MTU mismatch. Test it with vmkping ++netstack=vxlan -d -s 1572 <remote-TEP>, then read the tunnel state in /var/log/vmkernel.log and the uplink events in /var/log/vobd.log.
Why the TEP path matters
Two VMs on the same host never leave the host, so their traffic does not touch the physical network. As soon as they run on different hosts, every frame is wrapped in Geneve by the TEP (a vmkernel interface on the overlay transport zone) and sent to the TEP of the other host over the TEP VLAN. Geneve adds at least 50 bytes of headers, plus options, so a full 1500-byte VM frame needs a larger MTU on every hop. NSX needs at least 1600 and recommends 1700; VCF designs typically use 9000.
This gives a quick first test: migrate both VMs to the same host. If they can talk there but not across hosts, the problem is the TEP path, not the VMs, the segment or the firewall.
Step 1: find the TEPs
On each ESX host, list the TEP interfaces of the overlay TCP/IP stack (named vxlan for historical reasons):
esxcli network ip interface ipv4 get -N vxlan
esxcli network ip interface list -N vxlan # vmk10, vmk11 ... with their MTU
esxcli network ip route ipv4 list -N vxlan # TEP gateway if TEPs are in different subnetsOr from NSX Manager: get transport-node <uuid> vtep. In the UI, System > Fabric > Hosts shows the tunnels of each host with their status.
Step 2: test TEP-to-TEP reachability and MTU
Ping the remote TEP from the overlay stack, first with a small packet, then with the largest packet your design allows and fragmentation disabled (-d). The size is the MTU minus 28 bytes of IP and ICMP headers:
vmkping ++netstack=vxlan <remote-TEP-ip> # basic reachability
vmkping ++netstack=vxlan -d -s 1572 <remote-TEP-ip> # MTU 1600
vmkping ++netstack=vxlan -d -s 1672 <remote-TEP-ip> # MTU 1700
vmkping ++netstack=vxlan -d -s 8972 <remote-TEP-ip> # MTU 9000
# with several TEPs, test each source interface
vmkping ++netstack=vxlan -I vmk11 -d -s 1572 <remote-TEP-ip>| Result | Meaning | Check |
|---|---|---|
| Small ping fails | No IP path between the TEPs | TEP VLAN on the physical switch ports, TEP IP pool and gateway, routing between TEP subnets |
| Small ping works, large ping fails | MTU too small somewhere on the path | VDS MTU, NSX uplink profile or global MTU, physical switch ports, and the router or SVI between TEP subnets |
| Both work | The TEP path is fine | Segment configuration, distributed firewall rules, the VM itself |
Check the MTU on every layer: the vSphere Distributed Switch, the MTU in the NSX uplink profile (or the global tunnel MTU), the physical switch ports toward all hosts, and, if TEP subnets are routed, the router interfaces between them. One switch port or one SVI with 1500 is enough to break east-west traffic while small packets, such as ping and ARP, still work.
Step 3: read the host logs
/var/log/vmkernel.log: tunnel BFD state
NSX runs BFD over each Geneve tunnel to know whether the remote TEP is alive. Every state change is logged:
grep BFD_HandleStatusChange /var/log/vmkernel.log | tail -20Each line names the local and remote TEP, the old and new state, a diagnostic code and the tunnel type (overlay). The diagnostic Control Detection Time Expired means BFD packets from the remote TEP stopped arriving within the detection time, which points to loss on the TEP path. Lines that flap between down and up for the same remote TEP usually mean an unstable uplink or a link with drops.
/var/log/vobd.log: uplinks and redundancy
vobd records physical uplinks going down or up and the loss or return of uplink redundancy on distributed switch ports:
grep -E "uplink.transition|connectivity|redundancy" /var/log/vobd.log | tail -30
esxcli network nic listLook for vob.net.dvport.uplink.transition.down or lost redundancy around the time the tunnels went down. Match the timestamps with the BFD lines in vmkernel.log: if an uplink went down first, start with the cable, the NIC, the switch port or the teaming policy that maps TEPs to uplinks.
If the TEP path is fine
- Check that both VMs are really attached to the same segment and that both hosts are in the overlay transport zone of that segment.
- Check the distributed firewall rules applied to both VMs; see the NSX distributed firewall CLI.
- Check that the host agents are running, so that the host received the segment configuration; see troubleshooting NSX host agents.
Further reading
Designing the TEP and uplink layout from scratch on Cisco UCS? See NSX Edge design on Cisco UCS for NSX 4.x and VCF 9.