Tag: Homelab

Tesla P40 Passthrough GPU Disappears from the VM: “PCIe link to the device is down” on ESXi

My NVIDIA Tesla P40 in ESXi passthrough suddenly vanished from the VM: nvidia-smi could not talk to the driver and a VM reboot did not help. What the logs show, why only a host reboot fixes it, and how to make a homelab P40 stable (KB 418517).

Short answer: When a GPU in ESXi PCI passthrough suddenly disappears from the virtual machine, nvidia-smi says it couldn’t communicate with the NVIDIA driver and the host’s vmkernel.log says Skipping device reset … because PCIe link to the device is down, the card itself has locked up. Neither the guest nor ESXi can reset it, so rebooting the VM does nothing: only a reboot of the ESXi host brings it back (Broadcom KB 418517). That is exactly what happened to the NVIDIA Tesla P40 in my homelab. The lasting fix is to remove the reason the card drops off the bus: in a homelab that is usually heat, power or the PCIe riser.

What happened in my lab

My Tesla P40 runs in PCI passthrough to a single Linux VM for local LLMs. It sits on a PCIe riser and, because the P40 has no fan of its own, it is cooled by a blower fan on a 3D-printed adapter. After running fine for a long time, the GPU was suddenly gone from the VM in the middle of normal work, without any change on vSphere or in the VM:

$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

Rebooting the VM did not help; the GPU stayed unavailable. The ESXi host told the real story in /var/log/vmkernel.log:

Skipping device reset on <PCIe-ID> because PCIe link to the device is down.

Why a VM reboot does not help

When a VM with a passthrough device powers on or reboots, ESXi resets the PCI device so the guest gets a clean card. That reset needs a working PCIe link. Here the GPU stopped responding at the hardware or firmware level and the link to it went down, so ESXi skips the reset and the guest driver finds nothing it can talk to. The card stays stuck until the host is restarted and the slot is initialized again during boot. Broadcom’s KB describes the same: once a passthrough GPU becomes unresponsive, neither ESXi nor the VM can reset it without a full host reboot.

Confirm it

# in the Linux VM
nvidia-smi
sudo dmesg | grep -iE "nvrm|xid"            # look for Xid errors, e.g. "GPU has fallen off the bus"

# on the ESXi host
grep -iE "PCIe link|device reset" /var/log/vmkernel.log | tail -20
lspci | grep -i nvidia                          # is the card still enumerated?
esxcli hardware pci list | grep -B2 -A12 -i nvidia

Fix: reboot the host

  1. Shut down the VM that uses the GPU (and migrate or shut down the other VMs on that host).
  2. Put the host in maintenance mode and reboot it. If the GPU is still missing after a warm reboot, do a full power cycle so the card loses power completely.
  3. Power the VM on and check nvidia-smi again.

Why it happens, and how to keep a homelab P40 stable

The KB points to GPU hardware or firmware stability and lists what to check with the hardware vendor: card stability, PCIe slot health, GPU firmware or VBIOS, thermal and power conditions, and known reset or firmware lock-up issues. For a data-center card like the P40 running outside its intended server, three of them stand out:

  • Heat. The P40 is passively cooled and expects strong server airflow. A homelab fan adapter that is fine at idle can be too weak under a long inference run. Watch the temperature and throttle reasons during load, and lower the power limit if needed:
    nvidia-smi --query-gpu=temperature.gpu,power.draw,clocks_throttle_reasons.active --format=csv -l 5
    sudo nvidia-smi -pl 200 (default limit 250 W)
  • Power. The P40 draws up to 250 W through an 8-pin CPU (EPS) style connector, not a normal PCIe 8-pin. Use the correct adapter and a power supply rail with enough headroom; a load spike on a weak rail can drop the card.
  • The PCIe path. A riser or extension cable is the first suspect for a link that goes down. Reseat it, try a shorter or better-shielded riser, and if the board allows it, force the slot to PCIe Gen3 (or Gen2 for testing) instead of Auto.

Keep the passthrough settings the large-BAR card needs (pciPassthru.use64bitMMIO = true and pciPassthru.64bitMMIOSizeGB = 64), keep the server BIOS and the NVIDIA guest driver current, and note when the drops happen: always under heavy load points to heat or power, random times at idle point more to the riser or the card itself.

Further reading