vSphere Pod Traffic to ClusterIP Times Out on a VDS Supervisor: Cause and Workarounds

On a vSphere 8 Supervisor with VDS networking, vSphere Pods time out to ClusterIP services: ErrImagePull, CrashLoopBackOff or a stuck Supervisor service install. Why the gateway drops it and how to work around it (KB 312199).

Short answer: On a vSphere 8 Supervisor that uses VDS networking (not NSX), traffic from a vSphere Pod to a Kubernetes ClusterIP is sent to the gateway of the workload network, even when the backend sits in the same subnet. If that gateway or a firewall next to it drops traffic that comes back in on the same interface, or sees only half of a TCP session, the pod times out: DNS lookups fail with ErrImagePull, connections to the API fail with CrashLoopBackOff, and Supervisor services such as Harbor or Contour do not install. Allow the hairpin traffic on the gateway, use a simpler internal gateway, and allow kube-dns to reach the upstream DNS servers. Broadcom fixed the behavior in VCF 9.0 (KB 312199).

Symptoms

vSphere Pods run Supervisor services like Harbor, Contour or the image registry, so the problem usually shows up when you install a Supervisor service or upgrade the Supervisor. Typical signs:

  • ErrImagePull right after deployment. kubectl describe pod shows that the image could not be fetched because the DNS lookup of the registry name timed out (UDP), or the HTTPS connection to the registry timed out (TCP 443).
  • CrashLoopBackOff shortly after start. kubectl logs shows a timeout or a TLS handshake timeout when the pod connects to a ClusterIP, for example the Kubernetes API service.
  • ClusterIP fails, backend works. A connection from a vSphere Pod to a ClusterIP times out, while a direct connection to the backend pod IP works.
  • Supervisor upgrade stuck with an ImageRegistryUpgrade timeout, and the image disks for the service are missing in the Supervisor: kubectl get imagedisks -A -o yaml | grep <service-name> returns nothing.

Why it happens

A ClusterIP is a virtual address; it is not in the pod’s subnet. With VDS networking on vSphere 8, a vSphere Pod therefore sends every packet for a ClusterIP to its default gateway, the gateway of the Supervisor workload network. The backend behind the ClusterIP is often in the very same subnet, so the gateway has to send the packet back out of the interface it came in on. Two things can break this:

  • Hairpin routing is blocked. Routing a packet back out of the same interface can be denied by router security features such as reverse-path (RPF/uRPF) checks or a policy against same-interface forwarding. This hits UDP first, so DNS lookups during image fetching fail.
  • Asymmetric TCP. The request goes through the gateway, but the reply from the backend goes straight to the pod, because both are in the same subnet. A stateful firewall on or next to the gateway sees only the first and third packets of the TCP handshake and drops the session as invalid. The connection opens and then nothing passes.

A related variant: kube-dns forwards name lookups to the upstream DNS servers set in the workload network, and a third-party router or firewall between them does not allow that traffic. vSphere Pods then cannot resolve any external name.

Workarounds for vSphere 8

  1. Allow hairpin traffic on the gateway. Change the router and firewall configuration so that any traffic arriving from the workload network interface may be routed back out of the same interface: relax the RPF check and do not drop asymmetric TCP for that subnet.
  2. Use a simpler gateway for the workload network. Instead of the company edge firewall, point the Supervisor workload network to an internal router one hop before it, without stateful filtering on that segment.
  3. Allow kube-dns to reach the upstream DNS. Find the kube-dns address in the Supervisor context and allow DNS (UDP and TCP 53) between it and the DNS servers configured in the workload network:
# Supervisor context
kubectl get services -n kube-system | grep -i dns

After the change, delete the failing pod or retry the Supervisor service installation, and check that the image disks appear with kubectl get imagedisks -A.

Fixed in VCF 9.0

According to the KB, the underlying behavior of vSphere Pods on VDS networking is fixed in VCF 9.0. After you upgrade to 9.0 you can remove the router and firewall workarounds. The issue does not apply to Supervisors that use NSX or VPC networking.

Further reading

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere.