Tag: NSX

NSX Load Balancer 502 Bad Gateway: Header Size Limits, Logs and Edge CLI Checks

NSX native load balancer returns 502 Bad Gateway? Check the header size limits in the HTTP application profile, find the 502 in the Edge LB error and access logs, and verify pool and server-side SSL.

Short answer: A 502 Bad Gateway from the NSX native load balancer means the virtual server accepted the client request but did not get a usable response from the pool member. A frequent cause is a header that is larger than the limit set in the HTTP application profile, typically large response headers or cookies from the application. Raise the request and response header size in the profile (Networking > Load Balancing > Profiles > Application), and confirm the cause in the load balancer error log and access log on the Edge.

Which 5xx error points where

ErrorUsual meaning on the NSX LBCheck first
502 Bad GatewayThe member answered with something the LB could not use, or closed the connection.Header size limits, server-side SSL, the application on the member
503 Service UnavailableNo pool member is available.Health monitor and pool status
504 Gateway TimeoutThe member did not answer in time.Response timeout in the HTTP profile, application performance
400 Bad RequestThe client request itself was rejected, for example a request header over the limit.Request header size, client cookies

Cause: header size over the limit

The HTTP application profile has two limits: Request Header Size and Response Header Size. When the application returns headers larger than the response limit, which happens with large cookies, long redirect URLs, many Set-Cookie headers or big authentication tokens, the load balancer cannot pass the response on and the client gets a 502. Requests with oversized headers, for example a browser that sends a pile of cookies, fail in a similar way on the request side.

Typical signs: only some users or only some URLs fail (often after login, when the session cookie or token is set), the backend log shows the request as successful, and the LB log shows the 502.

Fix: in the NSX Manager UI, open Networking > Load Balancing > Profiles > Application, edit the HTTP profile used by the virtual server and raise Request Header Size and Response Header Size (values in bytes). Raise them to what the application really needs rather than to the maximum, and use a dedicated profile for that application so other virtual servers keep the default.

Other common causes of 502

  • Server-side SSL mismatch: the pool members listen on HTTPS but the virtual server has no server SSL profile, or the other way round. The LB then speaks the wrong protocol to the member.
  • Member resets the connection: the application crashes or a firewall on the member drops the connection from the SNAT address of the load balancer.
  • Health monitor too weak: a TCP monitor marks the member up while the application behind the port is broken. Use an HTTP or HTTPS monitor that checks a real URL and response code.

Edge CLI checks

On the Edge node that runs the load balancer (the active Edge of the Tier-1 gateway it is attached to):

get load-balancers
get load-balancer <lb-uuid> status
get load-balancer <lb-uuid> virtual-servers status
get load-balancer <lb-uuid> pools status
get load-balancer <lb-uuid> health-check-table

# logs straight from the CLI
get load-balancer <lb-uuid> error-log reg-filter 502
get load-balancer <lb-uuid> virtual-server <vs-uuid> access-log follow

# built-in diagnosis
get load-balancer <lb-uuid> diagnosis summary

Logs on the NSX Edge

Load balancer error log

From the root shell of the Edge, each load balancer has its own log folder under /var/log/lb/ with an error.log. Search it for the 502 and for the reason around it:

grep -r "502" /var/log/lb/ | tail -20
grep -ri "too big header\|upstream" /var/log/lb/ | tail -20

A line with HTTP/1.1 502 Bad Gateway confirms that the load balancer generated the error itself. Error log entries that mention a header that is too big while reading the response from the upstream (the pool member) point straight to the response header size limit.

Access log in syslog

When access logging is enabled on the virtual server, every request is written to /var/log/syslog on the Edge with subcomp="lb" and s2comp="access":

grep 's2comp="access"' /var/log/syslog | grep "HttpResponse.Status: '502'" | tail -20

Each entry is a full record of the request: virtual server name, IP and port, pool and the pool member that handled it, client IP, SNAT IP and port, HTTP method, URI and host, and the response status, size and server time. That tells you which member and which URL return the 502. A response size of 0 together with a long server time means the member never sent a usable response.

Further reading

NSX East-West Connectivity: VMs on the Same Segment Cannot Talk Across ESX Hosts

VMs on the same NSX segment talk on one host but not across hosts? Check TEP-to-TEP connectivity and MTU with vmkping, read tunnel BFD state in vmkernel.log and uplink events in vobd.log.

Short answer: When VMs on the same NSX overlay segment can talk while they sit on one ESX host, but not when they run on different hosts, the Geneve tunnel between the two hosts’ TEPs is not working. Either the TEPs cannot reach each other, or the path between them does not carry the larger encapsulated packets because of an MTU mismatch. Test it with vmkping ++netstack=vxlan -d -s 1572 <remote-TEP>, then read the tunnel state in /var/log/vmkernel.log and the uplink events in /var/log/vobd.log.

Why the TEP path matters

Two VMs on the same host never leave the host, so their traffic does not touch the physical network. As soon as they run on different hosts, every frame is wrapped in Geneve by the TEP (a vmkernel interface on the overlay transport zone) and sent to the TEP of the other host over the TEP VLAN. Geneve adds at least 50 bytes of headers, plus options, so a full 1500-byte VM frame needs a larger MTU on every hop. NSX needs at least 1600 and recommends 1700; VCF designs typically use 9000.

This gives a quick first test: migrate both VMs to the same host. If they can talk there but not across hosts, the problem is the TEP path, not the VMs, the segment or the firewall.

Step 1: find the TEPs

On each ESX host, list the TEP interfaces of the overlay TCP/IP stack (named vxlan for historical reasons):

esxcli network ip interface ipv4 get -N vxlan
esxcli network ip interface list -N vxlan        # vmk10, vmk11 ... with their MTU
esxcli network ip route ipv4 list -N vxlan       # TEP gateway if TEPs are in different subnets

Or from NSX Manager: get transport-node <uuid> vtep. In the UI, System > Fabric > Hosts shows the tunnels of each host with their status.

Step 2: test TEP-to-TEP reachability and MTU

Ping the remote TEP from the overlay stack, first with a small packet, then with the largest packet your design allows and fragmentation disabled (-d). The size is the MTU minus 28 bytes of IP and ICMP headers:

vmkping ++netstack=vxlan <remote-TEP-ip>                  # basic reachability
vmkping ++netstack=vxlan -d -s 1572 <remote-TEP-ip>       # MTU 1600
vmkping ++netstack=vxlan -d -s 1672 <remote-TEP-ip>       # MTU 1700
vmkping ++netstack=vxlan -d -s 8972 <remote-TEP-ip>       # MTU 9000

# with several TEPs, test each source interface
vmkping ++netstack=vxlan -I vmk11 -d -s 1572 <remote-TEP-ip>
ResultMeaningCheck
Small ping failsNo IP path between the TEPsTEP VLAN on the physical switch ports, TEP IP pool and gateway, routing between TEP subnets
Small ping works, large ping failsMTU too small somewhere on the pathVDS MTU, NSX uplink profile or global MTU, physical switch ports, and the router or SVI between TEP subnets
Both workThe TEP path is fineSegment configuration, distributed firewall rules, the VM itself

Check the MTU on every layer: the vSphere Distributed Switch, the MTU in the NSX uplink profile (or the global tunnel MTU), the physical switch ports toward all hosts, and, if TEP subnets are routed, the router interfaces between them. One switch port or one SVI with 1500 is enough to break east-west traffic while small packets, such as ping and ARP, still work.

Step 3: read the host logs

/var/log/vmkernel.log: tunnel BFD state

NSX runs BFD over each Geneve tunnel to know whether the remote TEP is alive. Every state change is logged:

grep BFD_HandleStatusChange /var/log/vmkernel.log | tail -20

Each line names the local and remote TEP, the old and new state, a diagnostic code and the tunnel type (overlay). The diagnostic Control Detection Time Expired means BFD packets from the remote TEP stopped arriving within the detection time, which points to loss on the TEP path. Lines that flap between down and up for the same remote TEP usually mean an unstable uplink or a link with drops.

/var/log/vobd.log: uplinks and redundancy

vobd records physical uplinks going down or up and the loss or return of uplink redundancy on distributed switch ports:

grep -E "uplink.transition|connectivity|redundancy" /var/log/vobd.log | tail -30
esxcli network nic list

Look for vob.net.dvport.uplink.transition.down or lost redundancy around the time the tunnels went down. Match the timestamps with the BFD lines in vmkernel.log: if an uplink went down first, start with the cable, the NIC, the switch port or the teaming policy that maps TEPs to uplinks.

If the TEP path is fine

  • Check that both VMs are really attached to the same segment and that both hosts are in the overlay transport zone of that segment.
  • Check the distributed firewall rules applied to both VMs; see the NSX distributed firewall CLI.
  • Check that the host agents are running, so that the host received the segment configuration; see troubleshooting NSX host agents.

Further reading

Designing the TEP and uplink layout from scratch on Cisco UCS? See NSX Edge design on Cisco UCS for NSX 4.x and VCF 9.

NSX BGP Neighbor Not Established: Troubleshooting Tier-0 BGP on the NSX Edge

Why a Tier-0 BGP session to the ToR does not come up (VLAN ID or BGP configuration mismatch), how to check it from the NSX Edge CLI, and how to read frr.log and syslog.

Short answer: When the BGP session between a Tier-0 gateway uplink and the top-of-rack switch does not come up, the cause is almost always one of two mismatches: the VLAN ID of the Edge uplink does not match the VLAN on the ToR interface, so there is no layer 2 path, or the BGP settings (neighbor IP, AS numbers, password, timers) do not match on both sides. Check the session state with get bgp neighbor summary in the Tier-0 SR context on the Edge, then read /var/log/frr/frr.log for the exact reason.

Start with the BGP state

Log in to the Edge node that runs the Tier-0 SR, find the SR and enter its VRF context:

get logical-routers
vrf <vrf-id-of-the-TIER0-SR>
get bgp neighbor summary
get bgp neighbor <peer-ip>
get interfaces

The state in the summary tells you where to look:

StateWhat it meansLook at
Idle, Connect, ActiveThe TCP session on port 179 never comes up.Layer 2 and IP reachability: uplink VLAN, IP and subnet, firewall or ACL on the ToR.
OpenSent, OpenConfirmTCP works, but the peers reject each other’s OPEN message.BGP parameters: AS numbers, neighbor IP, password, capabilities.
Established, then dropsThe session flaps.Hold timer, BFD, MTU, physical link or Edge resources.

Two more commands do much of the analysis for you: get bgp neighbor <peer-ip> session-diagnostics and get bgp-bfd diagnostics report.

Cause 1: VLAN ID mismatch on the uplink

The Tier-0 uplink interface sits on a VLAN segment, and that VLAN has to arrive tagged at the ToR interface where the BGP peer IP lives. If the VLAN ID differs anywhere on the path, the two sides never see each other and the session stays in Idle or Active. Check every hop:

  • The VLAN ID of the NSX uplink segment used by the Tier-0 interface.
  • The trunk port group or VLAN range on the vSphere Distributed Switch that carries the Edge uplinks (for an Edge VM), or the Edge uplink profile and teaming for a bare-metal Edge.
  • The physical switch port toward the ESX host or Edge, which must allow the VLAN, and the SVI or routed sub-interface on the ToR that holds the peer IP.

Then test reachability from the Tier-0 SR context, sourced from the uplink IP:

ping <peer-ip> source <tier0-uplink-ip> repeat 5

No reply means the problem is below BGP; fix the VLAN, IP or subnet first. Add size 1472 dfbit enable to the ping to check that a full 1500-byte packet passes; an MTU mismatch typically lets the session come up and then drop when larger UPDATE messages are sent.

Cause 2: BGP configuration mismatch

If ping works but the session does not establish, compare the Tier-0 BGP configuration with the ToR line by line:

  • Local AS on the Tier-0 and remote AS on the ToR neighbor, and the other way round.
  • Neighbor IP addresses: the ToR must peer with the Tier-0 uplink IP, and the Tier-0 with the ToR interface IP (or loopback, with multihop set on both sides).
  • MD5 password: set on both sides or on neither.
  • Keepalive and hold timers, and BFD: enabled on both sides with compatible intervals.
  • Address families (IPv4, IPv6, EVPN) enabled on both sides.

Logs on the NSX Edge

The Edge keeps two logs that explain BGP state changes. Read them from the root shell of the Edge node.

/var/log/frr/frr.log

The routing daemon log. It records every FSM transition of a neighbor and the reason for it:

grep -E "bgp_fsm_change_status|ADJCHANGE" /var/log/frr/frr.log | tail -30
grep -i "notification" /var/log/frr/frr.log | tail -30

A good session ends with the peer going from OpenConfirm to Established and an %ADJCHANGE: neighbor <peer-ip> in vrf default Up line. When it goes down, look at the reason next to it:

Message in the logMeaning
Network or config errorGeneral failure; check connectivity, the VLAN path, firewall rules and the configuration.
NOTIFICATION code 4 (Hold Timer Expired)No keepalive or update within the hold time: packet loss, congestion, MTU, or a peer that stopped answering.
NOTIFICATION code 2 (OPEN Message Error)The OPEN was rejected, for example a wrong peer AS (subcode 2).
Administratively down (NSXA_OP_STATE_DOWN)The neighbor is disabled in the NSX configuration.
BFD downBFD detected the path failure first and took BGP down with it; check get bfd-sessions.

/var/log/syslog

The NSX routing service raises an alarm for each BGP state change of a Tier-0 SR. Filter it to get a clean timeline per peer:

grep "state=BGP" /var/log/syslog

Each line names the peer IP, the SR and the new state: BGP_UP, BGP_DOWN or BGP_DELETE. Match the timestamps with frr.log and with the ToR log to see which side dropped first.

Further reading

For the Edge uplink, named teaming and ToR peering design on Cisco UCS, see NSX Edge design on Cisco UCS for NSX 4.x and VCF 9.

NSX Host Agents on ESX: Troubleshooting nsx-proxy, nsx-opsagent, nsx-cfgagent and nsx-nestdb

Host Down or Host Disconnected in NSX? Which NSX host agent is behind it (nsx-proxy, nsx-opsagent, nsx-cfgagent, nsx-nestdb), how to check and restart it on ESX, and which logs to read.

Short answer: When a transport node shows Down in the NSX UI, one of the local agents that apply and report configuration (nsx-cfgagent, nsx-opsagent or nsx-nestdb) has usually stopped or crashed. When it shows Host Disconnected, look first at nsx-proxy, the agent that holds the connection to NSX Manager. Check the agent with /etc/init.d/<agent> status, start or restart it, then confirm the manager and controller connections and read /var/run/log/nsx-syslog.log.

The NSX agents on an ESX host

When you prepare a host for NSX, several user-space agents are installed next to the NSX kernel modules. Four of them matter most when a host goes red in the UI:

AgentWhat it doesIf it stops
nsx-proxyHolds the TLS connections from the host to NSX Manager (TCP 1234) and to the central control plane (TCP 1235) and passes messages to the other agents.The host loses its connection to the managers: Host Disconnected.
nsx-cfgagentReceives the desired configuration (segments, routing, firewall) and programs it into the kernel datapath.New or changed configuration is not applied; host status goes Down.
nsx-opsagentReports operational state and status of the transport node back to NSX Manager and handles VM port attachments.The manager gets no status from the host; host status goes Down.
nsx-nestdbLocal database on the host where the agents keep the configuration they received.The other agents cannot read or store configuration; host status goes Down.

The agents are management and control plane. Traffic that is already programmed into the kernel keeps flowing while an agent is down, but nothing new reaches the host: no new segments, no firewall rule changes, no vMotion of VMs onto NSX ports that the host does not know yet.

Symptom 1: host is Down in the NSX UI

In System > Fabric > Hosts the transport node shows Down, while the host itself is up in vCenter. The usual cause is that nsx-cfgagent, nsx-opsagent or nsx-nestdb stopped or crashed. Check all agents in one go from the ESX shell:

for a in nsx-proxy nsx-cfgagent nsx-opsagent nsx-nestdb; do
  echo "== $a"; /etc/init.d/$a status
done

Start the agent that is not running, or restart it if it is running but the node stays Down:

/etc/init.d/nsx-opsagent start
/etc/init.d/nsx-cfgagent restart
/etc/init.d/nsx-nestdb status

Restart one agent at a time and wait a minute before you look at the UI again; the node status is refreshed by the next report from the host. On ESX the NSX CLI has no restart service command (that is for Manager and Edge), so /etc/init.d is the way.

Symptom 2: Host Disconnected

The NSX configuration of the host shows Host Disconnected. Here the host cannot talk to NSX Manager at all, which points to nsx-proxy or to the network path between the host and the managers.

/etc/init.d/nsx-proxy status
/etc/init.d/nsx-proxy restart

# Which managers and controllers does the host use, and is it connected?
nsxcli -c get managers
nsxcli -c get controllers

# TCP sessions to NSX Manager (1234) and the central control plane (1235)
esxcli network ip connection list | grep -E ':1234|:1235'

If nsx-proxy runs but there is no ESTABLISHED session on 1234 and 1235, the problem is not the agent: check routing and firewalls between the ESX management network and the NSX Manager nodes (or the cluster VIP), DNS, and the manager certificates. For the full list of host-side NSX CLI commands see the NSX CLI interactive mindmap with the ESXi node filter.

Logs: nsx-syslog.log

All four agents write to /var/run/log/nsx-syslog.log on the host (/var/log is a link to /var/run/log). Filter by agent name to see why it stopped:

grep -i "nestdb"    /var/run/log/nsx-syslog.log | tail -50
grep -i "cfgagent"  /var/run/log/nsx-syslog.log | tail -50
grep -i "opsagent"  /var/run/log/nsx-syslog.log | tail -50

# nsx-proxy: the connection to the managers, filtered by a manager IP
grep -i "<nsx-manager-ip>" /var/run/log/nsx-syslog.log | tail -20

A healthy nsx-proxy logs a heartbeat about once a minute, with subcomp="nsx-proxy" and a line like MessagingClientService: Heartbeat message received … from endpoint: ssl://<manager-ip>:1234. If the heartbeats stop at a certain time, that is when the host lost the manager; look at the lines just before it for TLS, certificate or connection errors.

For more detail while you reproduce the problem, raise the nsx-proxy log level and set it back afterwards:

nsxcli -c get service nsx-proxy logging-level
nsxcli -c set service nsx-proxy logging-level debug
# ... reproduce ...
nsxcli -c set service nsx-proxy logging-level info

If the agent keeps stopping

  • Check for core dumps in /var/core: a crashing agent leaves one there each time.
  • Check that the host is not short of memory or that its scratch/OSDATA partition is not full; agents fail to start when they cannot write their state.
  • Compare the NSX VIB version on the host (esxcli software vib list | grep -i nsx) with the NSX Manager version; a half-finished upgrade often shows up as agents that crash after start.
  • Collect a support bundle for the host from NSX Manager before you reboot it, so the logs and cores are kept for Broadcom support.

Further reading