vRealize/vROps events stop arriving after a patch or upgrade with an authorization error. How to identify it and fix it by enabling basic authentication.
(Authorization Error on vRealize Host)
After applying a patch or performing a version upgrade of vRealize Operations, some customers notice that events stop appearing in ServiceNow, specifically in the em_event table.
At first glance, the integration looks healthy: the MID Server is up, credentials haven’t changed, and the connection test may even pass. Yet no new events are flowing.
This post explains why this happens, how to recognize it quickly, and how to fix it safely.
Symptoms
After upgrading vRealize Operations, you may observe:
No new events in ServiceNow (em_event)
Existing integrations appear configured correctly
No obvious UI errors in vRealize Operations
However, the vRealize MID Server logs reveal the real problem.
How to Identify the Issue
Worker-Interactive:ConnectorProbe-) [MIDSystem:35]
*** Script: vRealize Payload : Response Body of Resource Properties :
<head><title>Not Authorized</title></head>
<body>
We apologize. You are not authorized for the request
</body>
This message is the key indicator that authentication between ServiceNow and vRealize Operations is failing at the API level.
Root Cause
Starting with vRealize Operations 8.1 and later, Basic Authentication is required for API access used by the MID Server.
vROps stuck on “Data Retriever is not initialized yet” or admin login fails? Check for an expired internal certificate and replace it (KB 71018 PAK).
Short answer: if the vRealize Operations (vROps / Aria Operations) login page hangs on “Data Retriever is not initialized yet. Please wait …” or the admin login returns “Incorrect user name/password”, check for an expired internal certificate first. Grep the logs for CertificateExpiredException (command below). If it’s there, replace the internal certificate with the Certificate Renewal PAK from KB 71018, then bring the cluster online again. Resetting the admin password won’t help until the certificate is fixed.
Data Retriever is not initialized yet. Please wait … What is WRONG?
OK, I tries to login with vRops admin login but another message “Incorrect user name/password”.
Note: The file name indicates 8.0.0 but will work for vRealize Operations 6.x and 7.x. The original ftpsite.vmware.com download links have been retired; download the current PAK file from KB 71018 on the Broadcom support portal.
Copy the vRealize Operations Certificate Renewal PAK file to the /tmp/ directory on all nodes in the vRealize Operations cluster using an SCP utility.
Log into all nodes in the vRealize Operations cluster as root via SSH or Console.
Run the following command on all nodes in the vRealize Operations cluster to make the necessary directories:
First, run the command on all Remote Collector nodes (if present) in the cluster, and wait for the task to complete. Continue to step 7.2.
Next, run the command on all Data nodes, the Witness node (if present), and the Primary Replica node (if present) in the cluster; do not wait for each node to complete, just start the command on all nodes. Once Waiting for certificate generation to complete appears on the last node, wait roughly 60 seconds, and continue to step 7.3.
Finally, run the command on the Primary node.
The expected behavior is for the command to finish, then shortly afterwards the pending tasks on the Data nodes and Primary Replica node (if present) will complete. To ensure that the command completes successfully check:
ls -l /var/vmware/_cert_generation_completed
Run the following commands on all nodes in the vRealize Operations cluster:
chown admin:admin -R /storage/vcops/user/conf/ssl/ /storage/vcops/user/conf/ssl_bak/ /storage/db/casa/webapp/hsqldb/
chmod guo+r -R /storage/vcops/user/conf/ssl/
service vmware-casa restart
service vmware-vcops stop
sed -i 's/sliceonline\ \=\ true/sliceonline\ \=\ false/g' /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/data/roleState.properties
Run the following commands on the Primary node, and Primary Replica node (if present):
service vmware-casa stop
sed -i -e 's/\"onlineState\"\:\"GOING\_OFFLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"GOING\_OFFLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"GOING\_ONLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"GOING\_ONLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"ONLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"ONLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"FAILURE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"FAILURE\"/\"online\_state\"\:\"OFFLINE\"/g' /data/db/casa/webapp/hsqldb/casa.db.script
service vmware-casa start
service vmware-vcops-web restart
/etc/init.d/apache2 restart
Log into the vRealize Operations Admin UI as the local admin user.
Click Bring Online under Cluster Status.
Summary:
When I can see message “Data Retriever is not initialized yet. Please wait …” or “Incorrect user name/password”.
Try to check and fix for expired internal certificate in vRealize Operations.
Reset the admin password in vROps, Aria Operations or VCF Operations with vcopsSetAdminPassword.py, and fix the CERTIFICATE_VERIFY_FAILED error when it fails.
Short answer: to reset the admin password in vRealize Operations, VMware Aria Operations or VCF Operations, log in to the primary (master) node as root and run $VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsSetAdminPassword.py --reset. The new password applies across all nodes in the cluster. If you get a Code 500 error, the password doesn’t meet the complexity or history rules. If the reset fails with CERTIFICATE_VERIFY_FAILED, the internal certificate has expired and must be replaced first (see below). Current reference: Broadcom KB 326391 (formerly VMware KB 2078313).
One day I reach vRops admin login message “Incorrect user name/password”. Here is a short info How to reset the admin password in vRealize Operations according KB 2078313. But I was unable to reset. My problem was with expired internal certificate in vRealize Operations.
Reset the OS admin Password
Note: Before resetting the admin password, ensure that the account is not locked out by this command:
pam_tally2 --user admin --reset
Note: If the admin password expires for the local OS, update the local OS version of the password before completing the below steps to reset the password across the Application:
Log into the Master node as root via SSH or Console.
Run the following command to reset the password, and password age of the local admin account:
passwd admin
Repeat steps 1 and 2 on all nodes in the vRealize Operations Manager cluster, including Remote Collectors.
Reset the vRops admin Password
To reset the admin password, follow the steps below:
Log in to the Master node as root via SSH or Console.
