Category: VCSA

Introduction to vSphere Clustering Service (vCLS)

The vSphere Clustering Service (vCLS) is a new capability that is introduced in the vSphere 7 Update 1 release. It’s first release provides the foundation to work towards creating a decoupled and distributed control plane for clustering services in vSphere.

More information:

vSphere Cluster Services (vCLS)

Starting with vSphere 7.0 Update 1, vSphere Cluster Services (vCLS) is enabled by default and runs in all vSphere clusters. vCLS ensures that if vCenter Server becomes unavailable, cluster services remain available to maintain the resources and health of the workloads that run in the clusters. vCenter Server is still required in 7.0 update 1 to run DRS and HA.

vCLS uses agent virtual machines to maintain cluster services health. The vCLS agent virtual machines (vCLS VMs) are created when you add hosts to clusters. Up to three vCLS VMs are required to run in each vSphere cluster, distributed within a cluster. vCLS is also enabled on clusters which contain only one or two hosts. In these clusters the number of vCLS VMs is one and two.

How to? Retrieving Password for vCLS VMs

You can retrieve the password to login to the vCLS VMs. To ensure cluster services health, avoid accessing the vCLS VMs. This document is intended for explicit diagnostics on vCLS VMs.

Procedure

  1. Use SSH to login to the vCenter Server Appliance.
  2. Run the following python script:
    /usr/lib/vmware-wcp/decrypt_clustervm_pw.py
  3. Read the output for the password.
    pwd-script-output
    Read key from file
    Connected to PSQL
    PWD: (password displayed here)

Operations that might disrupt the healthy functioning of vCLS VMs:

  • Changing the power state of the vCLS VMs
  • Resource reconfiguration of the vCLS VMs such as changing CPU, Memory, Disk size, Disk placement
  • VM encryption
  • Triggering vMotion of the vCLS VMs
  • Changing the BIOS
  • Removing the vCLS VMs from the inventory
  • Deleting the vCLS VMs from disk
  • Enabling FT of vCLS VMs
  • Cloning vCLS VMs
  • Configuring PMem
  • Moving vCLS VM to a different folder
  • Renaming the vCLS VMs
  • Renaming the vCLS folders
  • Enabling DRS rules and overrides on vCLS VMs
  • Enabling HA admission control policy on vCLS VMs
  • Enabling HA overrides on vCLS VMs
  • Moving vCLS VMs to a resource pool
  • Recovering vCLS VMs from a snapshot

Vulnerability in the VMware Directory Service (vmdir) (CVE-2020-3952) – VMSA-2020-0006

On April 9th, 2020 VMSA-2020-0006 was published. This advisory documents a critical severity sensitive information disclosure vulnerability identified by CVE-2020-3952.

Affected versions

The vulnerability received a CVSSv3 score of 10 out of 10. Which means this is a very serious security issue. Response matrix is VMSA-2020-0006.

How I can check it?

Additional Documentation for VMSA-2020-0006: Determining if a vCenter 6.7 deployment w/embedded or external Platform Services Controller (PSC) is affected by CVE-2020-3952 (78543)

https://kb.vmware.com/s/article/78543

Virtual Appliance Log File Location: /var/log/vmware/vmdird/vmdird-syslog.log or in /var/log/vmware/vmdird/vmdird-syslog.log.*.gz

zgrep "ACL" /var/log/vmware/vmdird/*.gz
/var/log/vmware/vmdird/vmdird-syslog.log.x.gz:2020-xx-xxTxxxxxx+00:00 info vmdird t@xxxxxx: ACL MODE: Legacy

Notes from KB:

  • In order to be affected by CVE-2020-3952, a deployment must meet 2 criteria. First, it must be a 6.7 deployment prior to 6.7u3f. Second, it must be running in legacy ACL mode.
  • Because the ACL MODE: Legacy log entry is only thrown at vmdir startup,  it is possible that it will be absent due to log file rollover even on affected deployments.
  • The ACL MODE: Legacy log entry will still be thrown after upgrading to 6.7u3f and/or 7.0 even though CVE-2020-3952 is resolved in these releases.

Path it NOW ! – PoC was published !

It is recommended to block any access over the LDAP port (389) except for administrative use.

Clean installations of vCenter Server 6.7 (embedded or external PSC) are not affected.

vCenter Server 6.7 (embedded or external PSC) prior to 6.7u3f is affected by CVE-2020-3952 if it was upgraded from a previous release line such as 6.0 or 6.5.

Path it ASAP because:

  • On April 15th, 2020 was relased information about How to reconstructed the faulty code flow that led to this vulnerability.

vSphere 6 Update Manager -> vSphere 7 Lifecycle Manager : upgrade ESXi 6.7 to ESXi 7

vSphere Update Manager – vSphere 6

In vSphere 6 we can use various methods and tools to deploy ESXi hosts and maintain their software lifecycle.

To deploy and boot an ESXi host, you can use an ESXi installer image or VMware vSphere® Auto Deploy™. The availability of choice options results in two different underlying ESXi platforms:

  • Using vSphere Auto Deploy – stateless mode
  • Using an installer ESXi image – statefull mode

vSphere Lifecycle Manager Images: A Single Platform, Single Tool, Single Workflow

By introducing the concept of images, vSphere Lifecycle Manager provides a unified platform for ESXi lifecycle management.
You can use vSphere Lifecycle Manager for stateful hosts only, but starting with vSphere 7.0, you can convert the Auto Deploy-based stateless hosts into stateful hosts, which you can add to clusters that you manage with vSphere Lifecycle Manager images.

How to Upgrade ESXi 6.7 to 7 with vSphere Lifecycle Manager?

After upgrade VCSA 7.0, We prepare upgrade for ESXi 6.7. It is simular logic like in vSphere Update Manager:

IMPORT ISO – We can upload ISO for example VMware-VMvisor-Installer-7.0.0-15843807.x86_64.iso
Step 1 of 2 – Uploading file to server
Step 2 of 2 – Adding to repository
After ISO upload We could check ISO image context
Next step is Create Baseline
Create Baseline – for ISO image
Select uploaded ISO image
Check summary
On Targer Cluster We attach our Baseline
Select ESXi 7 Baseline
– Check Compliance
– We can see Non-compliant for 3x ESXi host
REMEDIATE will start upgrade dialoge
It is necessary accept EULA
REMEDIATE will start ESXi 7 upgrade
In Recent Task We caould chek progress.
We can check ESXi 7.0 upgrade result.

vCenter Appliance (VCSA) root Partition full

Symptons: VCSA cannot provide an update or Unable to connect to the vCenter Server as services are not started.

Validate filesystem

Check disk space

root@vcsa [ /var/spool/clientmqueue ]# df -h
 Filesystem                                     Size  Used Avail Use% Mounted on
 /dev/sda3                                       11G  8.2G  1.9G  82% /

Check INODES

root@vcsa [ ~ ]# df -i
 Filesystem                   Inodes IUsed    IFree IUse% Mounted on
 /dev/sda3                     712704 97100   615604   14% /

Problem is with disk space.

How to resize partition

It is possible to increase the disk space of a specific VMDK , according KB. But After some time You could have the same issues.

https://kb.vmware.com/s/article/2126276

How to cleanup partition

It is necessary find where is a problem:

root@vcsa [ ~ ]# cd /var
root@vcsa [ /var ]# du -sh *
 2.1G    log
 5.2G    spool

clientmqueue

Problem with clientmqueue directory could be related with config for SMTP relay. It is possible to cleanup easily:

find /var/spool/clientmqueue -name "*" -delete

audit.log

Problem with audit.log is describe in KB. Size of audit.log file is very large and /var/log/audit folder consumes majority of the space.

https://kb.vmware.com/s/article/2149278

root@vcsa [ /var/log/audit ]# ls -l
 total 411276
 -rw------- 1 root root 420973104 Mar 31 00:53 audit.log
 truncate -s 0 audit.log

How to reset the Update Manager Database VCSA 6.5 or 6.7

After upgrading vCSA from 6.5 to 6.7, I reached error “Scan for Updates” .

Error:There are errors during the scan operation. Check the events and log files for details.

Cautions: It is a destrictive task. Ensure you have a backup or snapshots.

  • Stop the VMware Update Manager Service:

service-control --stop vmware-updatemgr 

  • Run the following command to reset the VMware Update Manager Database:

vCenter Server Appliance 6.5 : /usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db
vCenter Server Appliance 6.7: /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db

  • Run the following Command to delete the contents of the VMware Update Manager Patch Store

rm -rf /storage/updatemgr/patch-store/* 

  • Start the VMware Update Manager Service:

service-control --start vmware-updatemgr 

https://kb.vmware.com/s/article/67771

https://kb.vmware.com/s/article/2147284