Tag: Proxy

Supervisor Stuck in Configuring: “ERROR – Configured Core Supervisor Services” with ReconcileFailed

Supervisor stays in Configuring with “ERROR – Configured Core Supervisor Services” and velero or tkg packages in ReconcileFailed because docker-registry.kube-system.svc returns Service Unavailable. The proxy is the cause (KB 411761).

Short answer: When the Supervisor stays in Configuring in Workload Management with ERROR – Configured Core Supervisor Services, and services such as velero.vsphere.vmware.com or tkg.vsphere.vmware.com show ReconcileFailed because https://docker-registry.kube-system.svc:5000/v2/ returns Service Unavailable, the Supervisor’s HTTPS proxy is blocking the package installation. Broadcom’s workaround (KB 411761): disable the HTTPS proxy on the Supervisor, restart the wcp service on vCenter, let the packages reconcile, then enable the proxy again and restart wcp once more.

Symptoms

In the vSphere Client, Workload Management shows the Supervisor in Configuring with an error that lists the core Supervisor services that failed. Each entry has Reason: ReconcileFailed and a message from vendir saying it could not fetch the package image, because the connection to the internal registry docker-registry.kube-system.svc:5000 returned Service Unavailable.

Core Supervisor services, such as the VKS (TKG) service and Velero, are installed as Carvel packages (PackageInstall objects) in the Supervisor. The latest version of the KB shows how to see the failure directly. In the Supervisor context:

# all core service packages and their state
kubectl get pkgi -n vmware-system-supervisor-services

# details of a failing one
kubectl describe pkgi -n vmware-system-supervisor-services svc-velero.vsphere.vmware.com
kubectl describe pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

The status shows the condition ReconcileFailed, and usefulErrorMessage contains the same Unable to create round tripper … docker-registry.kube-system.svc:5000/v2/ … Service Unavailable text.

Why it happens

The package images for the core services are served from a registry inside the Supervisor, reached by its cluster-internal name docker-registry.kube-system.svc. When an HTTPS proxy is configured for the Supervisor, that internal request can be sent to the proxy instead of directly to the registry. The proxy cannot reach a Kubernetes-internal service name, answers Service Unavailable, and every package that needs an image from the internal registry stays in ReconcileFailed. The KB summarizes it simply: enabling the proxy can block package reconciliation.

Fix: proxy off, reconcile, proxy on

  1. Note the current proxy settings, then disable the HTTPS proxy: select the Supervisor, Configure > Network > Proxy Configuration.
  2. On the vCenter appliance, restart the Workload Management service so it reconciles again: service-control --restart wcp
  3. Wait until the packages reconcile; check with kubectl get pkgi -n vmware-system-supervisor-services that Velero, TKG and the others show Reconcile succeeded.
  4. Enable the HTTPS proxy again with the same settings, and restart wcp once more: service-control --restart wcp
  5. Check that all packages still reconcile and that Workload Management shows the Supervisor as Running.

Restarting wcp does not affect running workloads; it restarts the Workload Management service on vCenter, which drives the Supervisor configuration.

Prevent it: check the no-proxy list

The KB only describes the disable/re-enable cycle. To keep it from coming back after the next upgrade or package update, review the No Proxy list in the Supervisor proxy configuration: internal destinations should not go to the proxy. Typical entries are the Kubernetes service domain (.svc and .svc.cluster.local), the Supervisor service and pod CIDRs, the management and workload networks, vCenter and the internal registry. Check the exact list against the Broadcom documentation for your release before you change it.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.