Tag: TKG

TKG, Supervisor and VKS Nodes Cannot Resolve .local Hostnames: Cause and Workarounds

Kubernetes nodes on vSphere fail to resolve names ending in .local while the DNS server answers fine. Why systemd-resolved sends .local to mDNS, and the lab-only workarounds for TKG, the Supervisor and VKS clusters (KB 317072, KB 323444).

Short answer: Kubernetes nodes on vSphere (Tanzu Kubernetes Grid, Supervisor control plane VMs and VKS cluster nodes) run Linux with systemd-resolved, which by default sends names ending in .local to Multicast DNS, not to your DNS server. Lookups such as vcenter.corp.local therefore fail even though the DNS server knows the name. The workaround is to add the .local domain as a search domain on the nodes, but Broadcom supports it only for labs and proofs of concept: .local is reserved for mDNS by RFC 6762, and production environments should use a different domain (KB 317072, KB 323444).

Symptoms

  • On TKG or VKS nodes, any name that ends in .local fails to resolve: Temporary failure in name resolution. Other names resolve fine.
  • The Supervisor shows an Error state in Workload Management with a message that a host ending in .local cannot be resolved on a control plane VM and that local must be included in the management DNS search domains.
  • The DNS server is correct, reachable and answers when you ask it directly.

A quick test on a node separates the two layers:

resolvectl status                              # which DNS servers and domains each link uses
resolvectl query vcenter.corp.local            # goes through systemd-resolved: fails
dig @<dns-server-ip> vcenter.corp.local        # asks the DNS server directly: works

If the direct query works and the resolver query fails, the DNS server is fine and the node’s resolver is the problem.

Why it happens

This is not a TKG or VKS bug; it is how systemd-resolved works on any modern Linux. RFC 6762 reserves the .local top-level domain for Multicast DNS on the local link. systemd-resolved follows that: names ending in .local go to mDNS on the interfaces where it is enabled, and they are not sent to unicast DNS servers unless .local (or the specific domain, such as corp.local) is configured as a search or routing domain for that DNS server. Older systems and Windows clients simply asked the DNS server, which is why the same names work from your laptop or from older appliances.

Labs are hit most often, because many lab and nested environments still use .local domains.

Workaround for the Supervisor and VKS clusters (lab only)

KB 323444 has three parts. First, add the .local domain to the Supervisor’s management DNS search domains. On the vCenter appliance as root:

# find the Supervisor cluster ID (for example domain-c8)
dcli com vmware vcenter namespacemanagement clusters list

# current search domains
dcli com vmware vcenter namespacemanagement clusters get --cluster domain-c8 | grep -A 1 master_DNS_search_domains

# set the .local search domain
dcli com vmware vcenter namespacemanagement clusters update --cluster domain-c8 --master-dns-search-domains corp.local

Note that the update sets the list, so include any search domains you already had. Second, on each Supervisor control plane VM, make sure /etc/hosts contains the vCenter FQDN with its IP address. Third, on each VKS cluster node, set the search domain for the node’s interface:

resolvectl domain eth0 corp.local
resolvectl status

The node setting survives reboots but not node replacement: every node created by an upgrade, a scale-out or a machine health check needs it again. That alone makes it a lab-only fix.

Workaround for standalone TKG (lab only)

For standalone TKG management and workload clusters, KB 317072 uses the custom nameserver overlays from the TKG documentation (Nameservers on vSphere). Create the overlay files vsphere-overlay-dns-control-plane.yaml and vsphere-overlay-dns-workers.yaml as described there and add a search domain line at the end of both, so they finish like this:

nameservers: ["10.0.0.53"]
searchDomains: ["corp.local"]

Then deploy the clusters with that plan, so every node gets the search domain at creation, including nodes created later by upgrades.

The real fix: do not use .local

The latest versions of both KBs are clear: Broadcom does not recommend .local for any component, including vCenter, ESXi, NSX Manager, NSX Edges, Avi, Harbor and any endpoint that Kubernetes clusters use. The workarounds are strictly for proof of concept and lab use; they are not validated for production. For a new environment, use a domain you own (for example a subdomain of your company domain) or one reserved for private use such as home.arpa. For an existing production environment on .local, plan the rename before you deploy the Supervisor or VKS.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.