NSX Host Agents on ESX: Troubleshooting nsx-proxy, nsx-opsagent, nsx-cfgagent and nsx-nestdb

Short answer: When a transport node shows Down in the NSX UI, one of the local agents that apply and report configuration (nsx-cfgagent, nsx-opsagent or nsx-nestdb) has usually stopped or crashed. When it shows Host Disconnected, look first at nsx-proxy, the agent that holds the connection to NSX Manager. Check the agent with /etc/init.d/<agent> status, start or restart it, then confirm the manager and controller connections and read /var/run/log/nsx-syslog.log.

The NSX agents on an ESX host

When you prepare a host for NSX, several user-space agents are installed next to the NSX kernel modules. Four of them matter most when a host goes red in the UI:

Agent	What it does	If it stops
nsx-proxy	Holds the TLS connections from the host to NSX Manager (TCP 1234) and to the central control plane (TCP 1235) and passes messages to the other agents.	The host loses its connection to the managers: Host Disconnected.
nsx-cfgagent	Receives the desired configuration (segments, routing, firewall) and programs it into the kernel datapath.	New or changed configuration is not applied; host status goes Down.
nsx-opsagent	Reports operational state and status of the transport node back to NSX Manager and handles VM port attachments.	The manager gets no status from the host; host status goes Down.
nsx-nestdb	Local database on the host where the agents keep the configuration they received.	The other agents cannot read or store configuration; host status goes Down.

The agents are management and control plane. Traffic that is already programmed into the kernel keeps flowing while an agent is down, but nothing new reaches the host: no new segments, no firewall rule changes, no vMotion of VMs onto NSX ports that the host does not know yet.

Symptom 1: host is Down in the NSX UI

In System > Fabric > Hosts the transport node shows Down, while the host itself is up in vCenter. The usual cause is that nsx-cfgagent, nsx-opsagent or nsx-nestdb stopped or crashed. Check all agents in one go from the ESX shell:

for a in nsx-proxy nsx-cfgagent nsx-opsagent nsx-nestdb; do
  echo "== $a"; /etc/init.d/$a status
done

Start the agent that is not running, or restart it if it is running but the node stays Down:

/etc/init.d/nsx-opsagent start
/etc/init.d/nsx-cfgagent restart
/etc/init.d/nsx-nestdb status

Restart one agent at a time and wait a minute before you look at the UI again; the node status is refreshed by the next report from the host. On ESX the NSX CLI has no restart service command (that is for Manager and Edge), so /etc/init.d is the way.

Symptom 2: Host Disconnected

The NSX configuration of the host shows Host Disconnected. Here the host cannot talk to NSX Manager at all, which points to nsx-proxy or to the network path between the host and the managers.

/etc/init.d/nsx-proxy status
/etc/init.d/nsx-proxy restart

# Which managers and controllers does the host use, and is it connected?
nsxcli -c get managers
nsxcli -c get controllers

# TCP sessions to NSX Manager (1234) and the central control plane (1235)
esxcli network ip connection list | grep -E ':1234|:1235'

If nsx-proxy runs but there is no ESTABLISHED session on 1234 and 1235, the problem is not the agent: check routing and firewalls between the ESX management network and the NSX Manager nodes (or the cluster VIP), DNS, and the manager certificates. For the full list of host-side NSX CLI commands see the NSX CLI interactive mindmap with the ESXi node filter.

Logs: nsx-syslog.log

All four agents write to /var/run/log/nsx-syslog.log on the host (/var/log is a link to /var/run/log). Filter by agent name to see why it stopped:

grep -i "nestdb"    /var/run/log/nsx-syslog.log | tail -50
grep -i "cfgagent"  /var/run/log/nsx-syslog.log | tail -50
grep -i "opsagent"  /var/run/log/nsx-syslog.log | tail -50

# nsx-proxy: the connection to the managers, filtered by a manager IP
grep -i "<nsx-manager-ip>" /var/run/log/nsx-syslog.log | tail -20

A healthy nsx-proxy logs a heartbeat about once a minute, with subcomp="nsx-proxy" and a line like MessagingClientService: Heartbeat message received … from endpoint: ssl://<manager-ip>:1234. If the heartbeats stop at a certain time, that is when the host lost the manager; look at the lines just before it for TLS, certificate or connection errors.

For more detail while you reproduce the problem, raise the nsx-proxy log level and set it back afterwards:

nsxcli -c get service nsx-proxy logging-level
nsxcli -c set service nsx-proxy logging-level debug
# ... reproduce ...
nsxcli -c set service nsx-proxy logging-level info

If the agent keeps stopping

• Check for core dumps in /var/core: a crashing agent leaves one there each time.
• Check that the host is not short of memory or that its scratch/OSDATA partition is not full; agents fail to start when they cannot write their state.
• Compare the NSX VIB version on the host (esxcli software vib list | grep -i nsx) with the NSX Manager version; a half-finished upgrade often shows up as agents that crash after start.
• Collect a support bundle for the host from NSX Manager before you reboot it, so the logs and cores are kept for Broadcom support.

Further reading

• NSX CLI interactive mindmap: every NSX CLI command per release, filterable by node type
• NSX CLI commands: transport nodes, TEPs, VIBs and logs
• ESXCLI interactive mindmap for the esxcli commands used above