Troubleshooting Archives - Page 2 of 2

Short answer: When the BGP session between a Tier-0 gateway uplink and the top-of-rack switch does not come up, the cause is almost always one of two mismatches: the VLAN ID of the Edge uplink does not match the VLAN on the ToR interface, so there is no layer 2 path, or the BGP settings (neighbor IP, AS numbers, password, timers) do not match on both sides. Check the session state with get bgp neighbor summary in the Tier-0 SR context on the Edge, then read /var/log/frr/frr.log for the exact reason.

Start with the BGP state

get logical-routers
vrf <vrf-id-of-the-TIER0-SR>
get bgp neighbor summary
get bgp neighbor <peer-ip>
get interfaces

The state in the summary tells you where to look:

State	What it means	Look at
Idle, Connect, Active	The TCP session on port 179 never comes up.	Layer 2 and IP reachability: uplink VLAN, IP and subnet, firewall or ACL on the ToR.
OpenSent, OpenConfirm	TCP works, but the peers reject each other’s OPEN message.	BGP parameters: AS numbers, neighbor IP, password, capabilities.
Established, then drops	The session flaps.	Hold timer, BFD, MTU, physical link or Edge resources.

Two more commands do much of the analysis for you: get bgp neighbor <peer-ip> session-diagnostics and get bgp-bfd diagnostics report.

Cause 1: VLAN ID mismatch on the uplink

The Tier-0 uplink interface sits on a VLAN segment, and that VLAN has to arrive tagged at the ToR interface where the BGP peer IP lives. If the VLAN ID differs anywhere on the path, the two sides never see each other and the session stays in Idle or Active. Check every hop:

The VLAN ID of the NSX uplink segment used by the Tier-0 interface.
The trunk port group or VLAN range on the vSphere Distributed Switch that carries the Edge uplinks (for an Edge VM), or the Edge uplink profile and teaming for a bare-metal Edge.
The physical switch port toward the ESX host or Edge, which must allow the VLAN, and the SVI or routed sub-interface on the ToR that holds the peer IP.

Then test reachability from the Tier-0 SR context, sourced from the uplink IP:

ping <peer-ip> source <tier0-uplink-ip> repeat 5

No reply means the problem is below BGP; fix the VLAN, IP or subnet first. Add size 1472 dfbit enable to the ping to check that a full 1500-byte packet passes; an MTU mismatch typically lets the session come up and then drop when larger UPDATE messages are sent.

Cause 2: BGP configuration mismatch

If ping works but the session does not establish, compare the Tier-0 BGP configuration with the ToR line by line:

Local AS on the Tier-0 and remote AS on the ToR neighbor, and the other way round.
Neighbor IP addresses: the ToR must peer with the Tier-0 uplink IP, and the Tier-0 with the ToR interface IP (or loopback, with multihop set on both sides).
MD5 password: set on both sides or on neither.
Keepalive and hold timers, and BFD: enabled on both sides with compatible intervals.
Address families (IPv4, IPv6, EVPN) enabled on both sides.

Logs on the NSX Edge

The Edge keeps two logs that explain BGP state changes. Read them from the root shell of the Edge node.

/var/log/frr/frr.log

The routing daemon log. It records every FSM transition of a neighbor and the reason for it:

grep -E "bgp_fsm_change_status|ADJCHANGE" /var/log/frr/frr.log | tail -30
grep -i "notification" /var/log/frr/frr.log | tail -30

A good session ends with the peer going from OpenConfirm to Established and an %ADJCHANGE: neighbor <peer-ip> in vrf default Up line. When it goes down, look at the reason next to it:

Message in the log	Meaning
Network or config error	General failure; check connectivity, the VLAN path, firewall rules and the configuration.
NOTIFICATION code 4 (Hold Timer Expired)	No keepalive or update within the hold time: packet loss, congestion, MTU, or a peer that stopped answering.
NOTIFICATION code 2 (OPEN Message Error)	The OPEN was rejected, for example a wrong peer AS (subcode 2).
Administratively down (NSXA_OP_STATE_DOWN)	The neighbor is disabled in the NSX configuration.
BFD down	BFD detected the path failure first and took BGP down with it; check `get bfd-sessions`.

/var/log/syslog

The NSX routing service raises an alarm for each BGP state change of a Tier-0 SR. Filter it to get a clean timeline per peer:

grep "state=BGP" /var/log/syslog

Each line names the peer IP, the SR and the new state: BGP_UP, BGP_DOWN or BGP_DELETE. Match the timestamps with frr.log and with the ToR log to see which side dropped first.

The NSX agents on an ESX host

When you prepare a host for NSX, several user-space agents are installed next to the NSX kernel modules. Four of them matter most when a host goes red in the UI:

Agent	What it does	If it stops
`nsx-proxy`	Holds the TLS connections from the host to NSX Manager (TCP 1234) and to the central control plane (TCP 1235) and passes messages to the other agents.	The host loses its connection to the managers: Host Disconnected.
`nsx-cfgagent`	Receives the desired configuration (segments, routing, firewall) and programs it into the kernel datapath.	New or changed configuration is not applied; host status goes Down.
`nsx-opsagent`	Reports operational state and status of the transport node back to NSX Manager and handles VM port attachments.	The manager gets no status from the host; host status goes Down.
`nsx-nestdb`	Local database on the host where the agents keep the configuration they received.	The other agents cannot read or store configuration; host status goes Down.

The agents are management and control plane. Traffic that is already programmed into the kernel keeps flowing while an agent is down, but nothing new reaches the host: no new segments, no firewall rule changes, no vMotion of VMs onto NSX ports that the host does not know yet.

Symptom 1: host is Down in the NSX UI

In System > Fabric > Hosts the transport node shows Down, while the host itself is up in vCenter. The usual cause is that nsx-cfgagent, nsx-opsagent or nsx-nestdb stopped or crashed. Check all agents in one go from the ESX shell:

for a in nsx-proxy nsx-cfgagent nsx-opsagent nsx-nestdb; do
  echo "== $a"; /etc/init.d/$a status
done

Start the agent that is not running, or restart it if it is running but the node stays Down:

/etc/init.d/nsx-opsagent start
/etc/init.d/nsx-cfgagent restart
/etc/init.d/nsx-nestdb status

Restart one agent at a time and wait a minute before you look at the UI again; the node status is refreshed by the next report from the host. On ESX the NSX CLI has no restart service command (that is for Manager and Edge), so /etc/init.d is the way.

Symptom 2: Host Disconnected

The NSX configuration of the host shows Host Disconnected. Here the host cannot talk to NSX Manager at all, which points to nsx-proxy or to the network path between the host and the managers.

/etc/init.d/nsx-proxy status
/etc/init.d/nsx-proxy restart

# Which managers and controllers does the host use, and is it connected?
nsxcli -c get managers
nsxcli -c get controllers

# TCP sessions to NSX Manager (1234) and the central control plane (1235)
esxcli network ip connection list | grep -E ':1234|:1235'

If nsx-proxy runs but there is no ESTABLISHED session on 1234 and 1235, the problem is not the agent: check routing and firewalls between the ESX management network and the NSX Manager nodes (or the cluster VIP), DNS, and the manager certificates. For the full list of host-side NSX CLI commands see the NSX CLI interactive mindmap with the ESXi node filter.

Logs: nsx-syslog.log

All four agents write to /var/run/log/nsx-syslog.log on the host (/var/log is a link to /var/run/log). Filter by agent name to see why it stopped:

grep -i "nestdb"    /var/run/log/nsx-syslog.log | tail -50
grep -i "cfgagent"  /var/run/log/nsx-syslog.log | tail -50
grep -i "opsagent"  /var/run/log/nsx-syslog.log | tail -50

# nsx-proxy: the connection to the managers, filtered by a manager IP
grep -i "<nsx-manager-ip>" /var/run/log/nsx-syslog.log | tail -20

A healthy nsx-proxy logs a heartbeat about once a minute, with subcomp="nsx-proxy" and a line like MessagingClientService: Heartbeat message received … from endpoint: ssl://<manager-ip>:1234. If the heartbeats stop at a certain time, that is when the host lost the manager; look at the lines just before it for TLS, certificate or connection errors.

For more detail while you reproduce the problem, raise the nsx-proxy log level and set it back afterwards:

nsxcli -c get service nsx-proxy logging-level
nsxcli -c set service nsx-proxy logging-level debug
# ... reproduce ...
nsxcli -c set service nsx-proxy logging-level info

If the agent keeps stopping

Check for core dumps in /var/core: a crashing agent leaves one there each time.
Check that the host is not short of memory or that its scratch/OSDATA partition is not full; agents fail to start when they cannot write their state.
Compare the NSX VIB version on the host (esxcli software vib list | grep -i nsx) with the NSX Manager version; a half-finished upgrade often shows up as agents that crash after start.
Collect a support bundle for the host from NSX Manager before you reboot it, so the logs and cores are kept for Broadcom support.

Tag: Troubleshooting

NSX BGP Neighbor Not Established: Troubleshooting Tier-0 BGP on the NSX Edge