Deploying Tanzu Kubernetes “guest” cluster in…

Deploying Tanzu Kubernetes “guest” cluster in vSphere with Tanzu

Deploying Tanzu Kubernetes “guest” cluster in…

In this final installment of my “vSphere with Tanzu” posts, we are going to look at how to create our very first Tanzu Kubernetes (TKG) guest cluster. In previous posts, we have compared vSphere with Tanzu to VCF with Tanzu, and covered the prerequisites. Then we looked at the steps involved in deploying the HA-Proxy to provide a load balancer service to vSphere with Tanzu. In my most recent post, we looked at the steps involved in enabling workload management. Now that all of that is in…Read More


VMware Social Media Advocacy

“A Digital Foundation for an Unpredictable World”

“A Digital Foundation for an Unpredictable World”

“A Digital Foundation for an Unpredictable World”

Pat Gelsinger, VMware CEO, explores the impact of technology in a time of disruption and uncertainty worldwide at VMworld 2020. Tap into the full conference experience — visionary keynotes, simulated labs, 900+ sessions and entertainment zone. Experience more at https://www.vmworld.com/en/index.html And read more about how businesses survive disruption – and why some thrive – at https://www.vmware.com/radius/business-disruption-vmworld-2020/?utm_source=youtube&utm_medium=ownedsocial&utm_campaign=digital_foundation-21Q3-corp-ww


VMware Social Media Advocacy

vSphere Cluster Services (vCLS) Explained: External vs Embedded vCLS (7.0 to 9.x)

What vCLS VMs do, how many run per cluster, what changed with embedded vCLS in vSphere 8.0 U3 and 9.x, and what you must not do to them.

vSphere Cluster Services (vCLS) are small agent VMs that vCenter runs in every cluster so that cluster services such as DRS keep a healthy, distributed control plane. vCLS arrived in vSphere 7.0 Update 1 as external vCLS (up to three VMs per cluster). vSphere 8.0 Update 3 replaced it with embedded vCLS (two VMs per cluster, based on vSphere Pod technology). From vSphere 9.0 you can even disable vCLS without affecting DRS or HA. Leave vCLS VMs alone unless you are diagnosing them.

What vCLS does

Starting with vSphere 7.0 Update 1, vCLS is enabled by default and runs in all vSphere clusters. It keeps cluster services available and maintains the resources and health of the workloads in the cluster even if vCenter Server becomes unavailable. The vCLS VMs are created automatically when you add hosts to a cluster, including clusters with only one or two hosts.

External vs embedded vCLS

External vCLSEmbedded vCLS
vSphere versions7.0 Update 1 to 8.0 Update 28.0 Update 3 and later
TechnologyAgent VMs deployed by the ESX Agent Manager (EAM)vCLS VMs based on vSphere Pod technology, managed by the internal vpxd extension instead of EAM
VMs per clusterUp to 3 (1 or 2 in one- and two-host clusters)2, with at least 1 powered on for DRS
NamingvCLS-{UUID}vCLS-{UUID} (UUID of the ESXi host), plus the config option vCLSCRX.agent = true
Retreat ModeSupportedSupported (same per-cluster activate/deactivate)

Sources: TechDocs: Embedded vCLS and Broadcom KB 312147. If embedded vCLS VMs don’t deploy after an upgrade to 8.0 U3, start with KB 378133.

vCLS in vSphere 9.x

In vSphere 7.0 and 8.0, removing vCLS from a cluster (Retreat Mode) stops DRS and makes HA placement suboptimal. Starting with vSphere 9.0, vCLS can be disabled on a cluster without any impact on DRS or HA (KB 316514). The steps for every version are in vCLS Retreat Mode: how to disable vCLS on a vSphere cluster.

Retrieving the password for vCLS VMs (external vCLS)

You can retrieve the password to log in to external vCLS VMs. Use it only for explicit diagnostics; to keep cluster services healthy, avoid logging in to vCLS VMs.

Output of the vCLS password decryption script on vCenter Server Appliance
  1. Log in to the vCenter Server Appliance over SSH.
  2. Run /usr/lib/vmware-wcp/decrypt_clustervm_pw.py
  3. Read the password from the PWD: line of the output.

What not to do with vCLS VMs

These operations can disrupt the healthy functioning of vCLS VMs:

  • Changing their power state
  • Reconfiguring resources: CPU, memory, disk size or disk placement
  • VM encryption, changing the BIOS, configuring PMem, enabling FT
  • Triggering vMotion of vCLS VMs
  • Removing them from the inventory, deleting them from disk, cloning them, or recovering them from a snapshot
  • Renaming the VMs or the vCLS folders, or moving them to another folder or a resource pool
  • Applying DRS rules and overrides, HA admission control policies or HA overrides to vCLS VMs

If you need vCLS out of the way, for example during storage maintenance, use Retreat Mode instead of touching the VMs.

Further reading

NSX Disable Password Expiration: NSX-T 3.x, NSX 4.x and NSX in VCF 9 (Manager and Edge)

How to check, change or disable NSX password expiration for admin, root and audit on Manager and Edge, via CLI or API, and what changes in VCF 9.

Short answer: NSX Manager and NSX Edge local passwords (admin, root, audit) expire after 90 days by default. Log in to the NSX CLI as admin on each appliance and run clear user <user> password-expiration to disable expiration, or set user <user> password-expiration <1-9999> to change it. Verify with get user <user> password-expiration. The API field is password_change_frequency (0 disables). In VCF 9, Broadcom recommends rotation instead of never-expire.

Default NSX password expiration

Password expiration was introduced in NSX-T 2.4.0. Since then, the local accounts on NSX Manager and NSX Edge appliances expire 90 days after the last password change, and users get warning messages 7 days before expiry. When the admin password expires you can no longer log in to the NSX Manager UI, and any task or API call that uses that account fails. That is why an expired NSX password often shows up first as a broken integration (vCenter, backup, SDDC Manager, monitoring), not as a login prompt.

Check NSX password expiration

Connect to the appliance over SSH or the console as admin and check each local account:

nsx-mgr> get user admin password-expiration
Password expires 90 days after last change
User will receive warning messages 7 days before password expires.

nsx-mgr> get user root password-expiration
nsx-mgr> get user audit password-expiration

If the password is already expired, the output says Current password expired ## day ago. The appliance also logs upcoming expiry in /var/log/auth.log (entries such as password for user admin will expire in 1 days), and NSX raises the “Password expired” / “Password about to expire” alarms in the UI.

NSX disable password expiration (CLI)

The same NSX CLI commands work on NSX-T 3.x, NSX 4.x and NSX 9.x, on both Manager and Edge nodes. Run them as admin:

# Disable expiration (password never expires)
nsx-mgr> clear user admin password-expiration
nsx-mgr> clear user root password-expiration
nsx-mgr> clear user audit password-expiration

# Verify
nsx-mgr> get user admin password-expiration
Password expiration not configured for this user

If your security policy does not allow passwords that never expire, set a longer interval instead. The allowed range is 1 to 9999 days:

nsx-mgr> set user admin password-expiration 9999
nsx-mgr> set user root password-expiration 9999
nsx-mgr> set user audit password-expiration 9999

Practical notes:

  • Run the commands on every appliance where the account exists: each NSX Manager node and each NSX Edge node. Then verify with get user … password-expiration on each node rather than assuming it applied everywhere.
  • NSX 4.1.x only: changing a password with set user admin password resets a custom expiration back to 90 days (KB 376166). It is fixed in NSX 4.2.0 and later. On 4.1.x, re-run set user … password-expiration or clear user … password-expiration after every password change.
  • Resetting an expired password does not reset the expiration date. Extend it separately with the CLI or API.

API alternative: password_change_frequency

Password expiration is a property of the node user (NodeUserProperties). The field password_change_frequency holds the number of days; setting it to 0 disables password expiration. Broadcom documents these endpoints:

# Local node (the Manager you call)
PUT /api/v1/node/users/{userid}

# A specific Manager cluster node
PUT /api/v1/cluster/{cluster-node-id}/node/users/{userid}

# An Edge / transport node
PUT /api/v1/transport-nodes/{transport-node-id}/node/users/{userid}

Read the user first (GET on the same path), change password_change_frequency in the body and send it back with PUT. Check the “Update node user” call in the NSX API reference for your release for the exact required fields. The API route is the better choice when you manage many Edge nodes or want the change in an automation pipeline.

NSX password expired: how to recover

Expired admin or audit password

Log in as root, switch to admin with su admin, and set a new password. You need to know the current (expired) password:

nsx-mgr> set user admin password
Current password:
New password:
Confirm new password:

nsx-mgr> clear user admin password-expiration

In the UI, admin can reset expired passwords of other local users (System > User Management or the password-expiry notification), but not its own. Guest users cannot be reset from the CLI.

Expired root password

If you try st en with an expired root password, NSX refuses to open the root shell and tells you to run set user root password as admin. If admin is not accessible but you know the root password, a direct SSH/console login as root forces a password change. If you know neither password, follow “Resetting the Passwords of an Appliance” (GRUB edit during a reboot). KB 408091 also describes how to propagate a manually changed root password to the other Manager nodes in the cluster. Afterwards set the expiration again:

nsx-mgr> set user root password-expiration 9999
nsx-mgr> set user admin password-expiration 9999

NSX in VCF 9: rotate instead of disabling

In VMware Cloud Foundation, NSX Manager credentials belong to VCF password management, so changing them only on the appliance is different from standalone NSX:

  • Broadcom KB 435599 (VCF 9.x) says “Never Expire” is not a supported state for core components in the automated lifecycle engine and recommends configuring automatic rotation instead. Changing account properties directly on NSX with the CLI can cause compliance drift during SDDC Manager inventory syncs or LCM operations.
  • From VCF 9.1, password policies (length, complexity, expiration, lockout) are created in VCF Operations and assigned to the fleet or to a VCF instance. VCF Operations then flags non-compliant components and can remediate them.
  • Known issue in VCF Operations 9.1.x (KB 448813): NSX users set to never expire can still trigger a false “NSX password expired” alert. The KB workaround is to set a finite value above 90 days, e.g. set user admin password-expiration 365.

My recommendation: in a lab or standalone NSX, clear user … password-expiration is fine. In production VCF, keep expiration enabled, rotate passwords from VCF Operations or SDDC Manager, and use the CLI only to recover an account that has already expired.

Sources

Related on vdan.cz

What’s New with VMware Cloud Foundation 4.1

What’s New with VMware Cloud Foundation 4.1

What’s New with VMware Cloud Foundation 4.1

The VMware Cloud Foundation product team is pleased to announce the upcoming release of VMware Cloud Foundation 4.1. This new release builds upon the industry momentum and traction from the March announcement of VMware Cloud Foundation 4.0 with Tanzu by introducing new features and functionality that deliver developer ready infrastructure, allowing customers to scale without The post What’s New with VMware Cloud Foundation 4.1 appeared first on Cloud Foundation.


VMware Social Media Advocacy