Month: December 2021

Add F1705 Alert to Cisco UCS Manager Plugin

Add Cisco UCS fault F1705 (Rank VLS) as a custom fault in UCS Manager Plugin 3.0(6) for the vSphere HTML client, for Proactive HA.

New Cisco UCS firmware brings possibility to have notification about F1705 Alerts – Rank VLS.

In latest version of Cisco UCS Manager Plugin for VMware vSphere HTML Client (Version 3.0(6)) we could add Custom fault addition for proactive HA monitoring. How to do it?

Cisco UCS / Proactive HA Registration / Fault monitoring details / Add / ADDDC_Memory_Rank_VLS
Cisco UCS / Proactive HA Registration / Fault monitoring details / Add
Cisco UCS / Proactive HA Registration / vCenter server credentials / Register
Cisco UCS / Proactive HA Registration / Register
How Could I check it? Edit Proactive HA / Providers
It is better use Name “ADDDC_Memory_Rank_VLS” without spaces. On my picture I used “My F1705 Alerts”

Adding Custom Alert is only possible with unregistered Cisco UCS Provider, it is better to do it immediatly after Cisco UCS Manager Plugin instalation.

Now I can deceided If I will block F1705 or NOT. I personaly preffer to have F1705 Alert under Proactive HA. Then I only restart Blades with F1705. During reboot Hard-PPR permanently remaps accesses from a designated faulty row to a designated spare row.

Links:

Fastest workaround instructions to address CVE-2021-44228 (log4j) in vCenter Server

The fastest workaround for Log4j CVE-2021-44228 (VMSA-2021-0028) on vCenter Server 7.0, 6.7 and 6.5, and why patching is the real fix.

https://logging.apache.org/log4j/2.x/

Apache Log4j open source component has security bug (CVE-2021-44228 – VMSA-2021-0028). It is neccesary to fix vCenter Server 7.0.x, vCenter 6.7.x & vCenter 6.5.x.

Fastest and recommended is workaround with KB 87081 script (vc_log4j_mitigator.py).

Run ssh and create script via vim
Connected to service

    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

Command> shell
Shell access is granted to root
root@localhost [ ~ ]# cd /tmp
root@localhost [ /tmp ]# vim vc_log4j_mitigator.py
Run script python vc_log4j_mitigator.py 
root@localhost [ /tmp ]# python vc_log4j_mitigator.py
2021-12-21T10:38:20 INFO main: Script version: 1.6.0
2021-12-21T10:38:20 INFO main: vCenter type: Version: 7.0.2.00500; Build: 18455184; Deployment type: embedded; Gateway: False; VCHA: False; Windows: False;
A service stop and start is required to complete this operation.  Continue?[y]y
2021-12-21T10:38:23 INFO stop: stopping services
2021-12-21T10:38:46 INFO process_jar: Found a VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar
2021-12-21T10:38:46 INFO backup_file: VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar backed up to /tmp/tmpxi89fco8/opt/vmware/lib64/log4j-core-2.13.0.jar.bak
2021-12-21T10:38:47 INFO process_jar: VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar backed up to /tmp/tmpxi89fco8/opt/vmware/lib64/log4j-core-2.13.0.jar.bak
2021-12-21T10:39:03 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar
2021-12-21T10:39:03 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:04 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:04 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar
2021-12-21T10:39:04 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:04 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:06 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar
2021-12-21T10:39:06 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar.bak
2021-12-21T10:39:06 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar.bak
2021-12-21T10:39:07 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar
2021-12-21T10:39:07 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak
2021-12-21T10:39:07 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak
2021-12-21T10:39:08 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:08 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:08 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:14 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:14 INFO backup_file: VULNERABLE FILE: /tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:15 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war
2021-12-21T10:39:15 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war.bak
2021-12-21T10:39:15 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war.bak
2021-12-21T10:39:15 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:15 INFO backup_file: VULNERABLE FILE: /tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:16 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war
2021-12-21T10:39:16 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war.bak
2021-12-21T10:39:16 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war.bak
2021-12-21T10:39:16 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:16 INFO backup_file: VULNERABLE FILE: /tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:17 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war
2021-12-21T10:39:17 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war.bak
2021-12-21T10:39:18 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war.bak
2021-12-21T10:39:21 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:21 INFO backup_file: VULNERABLE FILE: /tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:22 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war
2021-12-21T10:39:22 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war.bak
2021-12-21T10:39:24 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war.bak
2021-12-21T10:39:25 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:25 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:26 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:28 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar
2021-12-21T10:39:28 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:29 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:32 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:32 INFO backup_file: VULNERABLE FILE: /tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:33 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-lookupsvc/webapps/ROOT.war
2021-12-21T10:39:33 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT.war.bak
2021-12-21T10:39:34 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT.war.bak
2021-12-21T10:39:34 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:35 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:35 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:37 INFO _patch_file: Found VULNERABLE FILE: /usr/lib/vmware-vmon/java-wrapper-vmon
2021-12-21T10:39:37 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-vmon/java-wrapper-vmon backed up to /tmp/tmpxi89fco8/usr/lib/vmware-vmon/java-wrapper-vmon.bak
2021-12-21T10:39:37 INFO patch_vum: Found a VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/start.ini
2021-12-21T10:39:37 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/start.ini backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak
2021-12-21T10:39:37 INFO print_summary:
=====     Summary     =====
Backup Directory: /tmp/tmpxi89fco8
List of processed java archive files:

/opt/vmware/lib64/log4j-core-2.13.0.jar
/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar
/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar
/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar
/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar
/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar
/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war
/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war
/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war
/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war
/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar
/usr/lib/vmware-lookupsvc/webapps/ROOT.war
/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar

List of processed configuration files:

/usr/lib/vmware-vmon/java-wrapper-vmon
/usr/lib/vmware-updatemgr/bin/jetty/start.ini

Total fixed: 16

    NOTE: Running this script again with the --dryrun
    flag should now yield 0 vulnerable files.

Log file: /var/log/vmsa-2021-0028_2021_12_21_10_38_20.log
===========================
2021-12-21T10:39:37 INFO start: starting services
2021-12-21T10:52:47 INFO main: Done.
Verify python vc_log4j_mitigator.py -r
root@localhost [ /tmp ]# python vc_log4j_mitigator.py -r
2021-12-21T11:10:01 INFO main: Script version: 1.6.0
2021-12-21T11:10:01 INFO main: vCenter type: Version: 7.0.2.00500; Build: 18455184; Deployment type: embedded; Gateway: False; VCHA: False; Windows: False;
2021-12-21T11:10:01 INFO main: Running in dryrun mode.
2021-12-21T11:11:01 INFO print_summary:
=====     Summary     =====

No vulnerable files found!

Total found: 0
Log file: /var/log/vmsa-2021-0028_2021_12_21_11_10_01.log
===========================
2021-12-21T11:11:01 INFO main: Done.

vc_log4j_mitigator.py [-h] – helps and more options

root@localhost [ /tmp ]# python vc_log4j_mitigator.py -h
usage: vc_log4j_mitigator.py [-h] [-d dirnames [dirnames ...]] [-a] [-r] [-b BACKUP_DIR] [-l LOG_DIR]

VMSA-2021-0028 vCenter tool; Version: 1.6.0 This tool deletes the JndiLookup.class file from *.jar and *.war files. On Windows systems the tool will by default traverse the folders identified by the VMWARE_CIS_HOME, VMWARE_CFG_DIR, VMWARE_DATA_DIR and VMWARE_RUNTIME_DATA_DIR
variables. On vCenter Appliances the tool will search by default from the root of the filesystem. All modified files are backed up if the process needs to be reversed due to an error.

optional arguments:
  -h, --help            show this help message and exit
  -d dirnames [dirnames ...], --directories dirnames [dirnames ...]
                        space separated list of directories to check recursively for CVE-2021-44228 vulnerable java archive files.
  -a, --accept-services-restart
                        accept the restart of the services without having manual prompt confirmation for the same
  -r, --dryrun          Run the script and log vulnerable files without mitigating them. The vCenter services are not restarted with this option.
  -b BACKUP_DIR, --backup-dir BACKUP_DIR
                        Specify a backup directory to store original files.
  -l LOG_DIR, --log-dir LOG_DIR
                        Specify a directory to store log files.

Links:

NSX Edge on Cisco UCS: Design for NSX 4.x and VCF 9 (Edge VM vs Bare Metal, vNICs, MTU, Geneve Offload)

NSX Edge design on Cisco UCS for NSX 4.x and VCF 9: Edge VM vs bare metal, vNIC pairs without fabric failover, TEP MTU and VIC Geneve offload.

Short answer: On Cisco UCS, NSX Edge VMs usually go on a small, dedicated ESX edge cluster. Give each Edge a pair of vNICs, one pinned to Fabric A and one to Fabric B, with UCS fabric failover disabled. Redundancy then comes from the Edge’s fp-eth active/standby mapping, not from the Fabric Interconnect. Keep TEP MTU consistent: 1700 or more, ideally 9000 end to end. Enable Geneve offload in the VIC adapter policy for the host transport nodes. For VCF 9, follow the VCF Edge Cluster design library.

I wrote this in 2021 for NSX-T 3.x, because Cisco’s design guides did not cover Edge placement. This September 2026 refresh covers NSX 4.2 and VCF 9.0/9.1, checked against Broadcom TechDocs and Cisco documentation. Parts that apply only to the 2021 design are labelled.

Where is the NSX-T design guide now?

VersionDesign reference
NSX-T 3.x (2021, this post’s original basis)VMware NSX-T Reference Design Guide. The old communities.vmware.com links no longer work.
NSX 4.2 standalone / VCF 5.2NSX Reference Design Guide 4.2 (linked from Broadcom KB 412786)
VCF 9.xVCF design library: NSX Edge Cluster Detailed Design, plus the VCF Virtual Networking Technical Reference, which replaces the NSX Design Guide from VCF 9.0

NSX Edge VM vs bare-metal Edge on Cisco UCS

Edge VMBare-metal Edge
Deployment in VCF 9From vCenter / NSX. Passwords managed by VCF Operations.Installed from ISO or PXE, then registered manually with NSX Manager
Data path interfaces4 datapath vNICsUp to 16 physical interfaces
StrengthsEasy to scale up, vSphere HA recovery, enough performance for most casesBetter small-packet performance, can use several 100G links
UCS specificsRuns on any UCS ESX host. Uses VIC vNICs through the VDS.The VCF 9.0 NIC list includes Cisco VIC 1300 and 1400 series (VIC 15000 is not listed). You must configure enough RX/TX queues in UCS Manager for one queue per datapath core. The CPU needs AES-NI and 1 GB huge pages.

Edge VM sizes (VCF 9.0 docs): Small 2 vCPU / 4 GB (proof of concept only), Medium 4 vCPU / 8 GB (L2 to L4 under 2 Gbps), Large 8 vCPU / 32 GB (2 to 10 Gbps, L7 load balancing, TLS inspection), Extra Large 16 vCPU / 64 GB (VPN, IDS/IPS, URL filtering, malware prevention). All sizes use a 200 GB disk.

NSX Edge design on Cisco UCS: uplinks and TEPs on vNICs

The VCF 9.1 edge cluster design recommends a dedicated vSphere cluster for Edges. It limits Edges per host to the number of available physical NICs and gives each Edge its own pair of pNICs. fp-eth0 maps to pNIC 1 active / pNIC 2 standby, and fp-eth1 maps to pNIC 2 active / pNIC 1 standby. Each Edge peers BGP only with the ToRs in its own rack, with BFD. On an FI-attached UCS server, “pNIC 1 / pNIC 2” means a Fabric A vNIC and a Fabric B vNIC:

  1. Create a vNIC pair per Edge (for example edge1-A on Fabric A and edge1-B on Fabric B), plus separate vNICs for host management, vMotion and storage.
  2. Disable fabric failover on these vNICs. Cisco’s teaming guidance for FI-attached servers says to use either fabric failover or OS/vSwitch teaming, never both. The VDS/Edge active/standby mapping is the teaming here.
  3. On the VDS, use two trunk port groups: one with the Fabric A uplink active and Fabric B standby, and one with the reverse. Connect fp-eth0 and fp-eth1 to them. Cisco supports only “Route based on originating port ID” and “source MAC hash” on FI-attached VICs. IP hash is not supported.
  4. In the Edge uplink profile, use two TEPs (multi-TEP) and add named teaming policies that pin each Tier-0 uplink VLAN to one uplink. The Fabric Interconnects run in end-host mode, so the BGP peers are the upstream switches, not the FIs.
NSX Edge VM with single N-VDS multi-TEP design: fp-eth0 and fp-eth1 on two VDS trunk port groups, TEP VLAN 25, BGP to TOR-LEFT on VLAN 27 and TOR-RIGHT on VLAN 28
NSX-T 3.0 single N-VDS multi-TEP Edge VM design (source: VMware NSX-T 3.0 Edge Design Step-by-Step). The topology is still valid in NSX 4.x and VCF 9.

The 2021 named teaming example still applies: two policies, PIN-TO-TOR-LEFT (“uplink-1” active) and PIN-TO-TOR-RIGHT (“uplink-2” active), one per external VLAN segment. On UCS, left and right become Fabric A and Fabric B.

MTU for NSX on UCS

NSX requires an MTU of at least 1600 bytes and recommends 1700 to allow for Geneve header growth. For better throughput, set the global TEP MTU, uplink profile MTU and VDS MTU to 9000 when the underlay supports jumbo frames. VCF also requires the Edge TEP MTU to match the host TEP MTU in the same overlay transport zone. On UCS, set the vNIC MTU to 9000, as Cisco’s VIC best practices recommend. The QoS system class and the upstream switches must also carry jumbo frames. Guest VMs can use up to 8800 on a 9000 underlay.

VIC adapter policy and Geneve offload

  • Supported on VIC 1400 from UCSM/IMC 4.1(2) (NSX standard mode) and 4.1(3) (Enhanced Datapath, nenic-ens driver). Supported on VIC 15000 from 4.2(3). Not supported on VIC 1200/1300. Geneve offload is disabled by default.
  • Recommended Ethernet adapter policy with Geneve offload (UCSM 4.3 guide): TX queues 1, TX ring 4096, RX queues 8, RX ring 4096, completion queues 16, interrupts 32, RSS on, GENEVE offload enabled. RoCEv2 cannot be used with it. Remove the Geneve setting before downgrading firmware.
  • Geneve offload is used by ESX host transport nodes (the nenic driver). Check it on the host per Broadcom KB 397644:
vsish -e get /net/pNics/vmnic0/properties | grep "Driver\|Device Hard\|Device Soft"
# look for VMNET_CAP_GENEVE_OFFLOAD in "Device Hardware Cap Activated"
# vmkernel.log: "nenic: ... Geneve: Overlay offload enabled, udp_port = 6081"

Use the nenic driver and VIC firmware pair listed in the Broadcom Compatibility Guide for your ESX release. The history (nenic 1.0.35.0 on ESXi 7.0) is covered in my nenic post linked below.

2021 design: dedicated ESXi edge host on UCS C220 M5 (NSX-T era)

The 2021 Dell EMC VxBlock NSX-T design put the edge on a C220 M5 host. vmnic0/1 were a Cisco VIC 1457 to the FIs for management, vmnic2/3 an Intel XXV710 for TEP/overlay, and vmnic4/5 a second XXV710 for BGP peering, cabled directly to the Nexus 9000 pair. The pattern is still valid when Edge traffic should bypass the FI domain, and it fits VCF’s dedicated-host, dedicated-pNIC recommendation.

Cisco UCS C220 M5 NSX edge host cabling: VIC ports to Fabric Interconnect A and B, two Intel XXV710 adapters to Nexus 9K-A and 9K-B
C220 M5 edge host cabling, after the Dell EMC VxBlock NSX-T design (2021)

PoC or lab: FI uplink Ethernet interfaces

For a PoC or homelab, configure FI uplink Ethernet ports for the edge VLANs and bind a vNIC template to them. For production, use the vNIC pairs described above.

Cisco UCS Manager Configure Uplink Ports dialog for Fabric Interconnect A fixed module

Sources

Related on vdan.cz

VMSA-2021-0028 & Log4j: What You Need to Know

VMSA-2021-0028 & Log4j: What You Need to Know

VMware has released a new critical security advisory, VMSA-2021-0028, in response to the industry-wide issue regarding the open source Apache Software Foundation log4j Java logging component, which was discovered to have a critical vulnerability (CVE-2021-44228). Because the log4j component […]

VMware Social Media Advocacy

Announcing NSX-T 3.2: Innovations in…

Announcing NSX-T 3.2: Innovations in Multi-Cloud Security, Networking, and Operations

Announcing NSX-T 3.2: Innovations in…

We’re excited to announce the general availability of VMware NSX-T 3.2, one of the largest NSX releases so far. NSX-T 3.2 includes key innovations across multi-cloud security, scale-out networking for containers, VMs, and physical workloads. It also delivers simplified operations that help enterprises achieve a one-click, public cloud experience wherever their workloads are deployed. Strong Multi-Cloud Security NSX-T 3.2 provides strong, multi-cloud, easy-to-operationalize network defenses…Read More

VMware Social Media Advocacy

My journey to Certified Kubernetes Administrator (CKA)

Tips from passing the Certified Kubernetes Administrator (CKA) exam: kubectl cheat sheet and autocomplete, Vim and tmux shortcuts, and time management.

The Journey Is More Important Than the Destination.

CKA: Certified Kubernetes Administrator
Certified Kubernetes Administrator (CKA)
  • CKA is the most touching exam I have passed.
  • The Journey Is More Important Than the Destination.
  • Fastest practice skills are crutial.
  • Have fun.

My tips

Use kubectl Cheat Sheet

Kubectl autocomplete – BASH

source <(kubectl completion bash) # setup autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubectl completion bash)" >> ~/.bashrc # add permanently

You can also use a shorthand alias for kubectl that also works with completion:

alias k=kubectl
complete -F __start_kubectl k

Kubectl apply 

kubectl apply -f ./my-manifest.yaml            # create resource(s)

Interacting with running Pods

# Run pod nginx and write its spec into a file called pod.yaml
kubectl run nginx --image=nginx --dry-run=client -o yaml > pod.yaml

#Better usage with $do
export do="–dry-run=client -o yaml"
kubectl run nginx --image=nginx $do > pod.yaml

Great Vim

vi ~/.vimrc
set ts=2 sw=2 expandtab ruler

Use tmux

A Ctrl+B Cheat Sheet

Session Commands
  • S: List sessions.
Window Commands
  • C: Create a new window.
  • N: Move to the next window.
  • P: Move to the previous window.
Pane Commands
  • %: Create a horizontal split.
  • “: Create a vertical split.
  • X: Close the current pane.

Cisco UCS Manager Plugin for VMware vSphere HTML Client (Version 3.0(6))

What’s in version 3.0(6) of the Cisco UCS Manager plugin for the vSphere HTML client: view, manage and monitor UCS hardware from vCenter.

Cisco has released the 3.0(6) version of the Cisco UCS Manager VMware vSphere HTML client plugin. The UCS Manager vSphere HTML client plugin enables a virtualization administrator to view, manage, and monitor the Cisco UCS physical infrastructure. The plugin provides a physical view of the UCS hardware inventory on the HTML client.

I notify BUG “Host not going into monitoring state vCenter restart”. Thank You for fix.

Release 3.0(6)

Here are the new features in Release 3.0(6):

  • Custom fault addition for proactive HA monitoring
  • Resolved host not going into monitoring state vCenter restart
  • Included defect fixes

VMware vSphere HTML Client Releases

Cisco UCS Manager plug-in is compatible with the following vSphere HTML Client releases:

VMware vSphere HTML Client VersionCisco UCS Manager Plugin for VMware vSphere Version
6.73.0(1), 3.0(2), 3.0(3), 3.0(4), 3.0(5), 3.0(6)
7.03.0(4), 3.0(5), 3.0(6)
7.0u1, 7.0u23.0(5), 3.0(6)

Note		VMware vSphere HTML Client Version 7.0u3 is not supported.
More info here.