After applying a patch or performing a version upgrade of vRealize Operations, some customers notice that events stop appearing in ServiceNow, specifically in the em_event table.
At first glance, the integration looks healthy: the MID Server is up, credentials haven’t changed, and the connection test may even pass. Yet no new events are flowing.
This post explains why this happens, how to recognize it quickly, and how to fix it safely.
Symptoms
After upgrading vRealize Operations, you may observe:
No new events in ServiceNow (em_event)
Existing integrations appear configured correctly
No obvious UI errors in vRealize Operations
However, the vRealize MID Server logs reveal the real problem.
How to Identify the Issue
Worker-Interactive:ConnectorProbe-) [MIDSystem:35]
*** Script: vRealize Payload : Response Body of Resource Properties :
<head><title>Not Authorized</title></head>
<body>
We apologize. You are not authorized for the request
</body>
This message is the key indicator that authentication between ServiceNow and vRealize Operations is failing at the API level.
Root Cause
Starting with vRealize Operations 8.1 and later, Basic Authentication is required for API access used by the MID Server.
First, run the command on all Remote Collector nodes (if present) in the cluster, and wait for the task to complete. Continue to step 7.2.
Next, run the command on all Data nodes, the Witness node (if present), and the Primary Replica node (if present) in the cluster; do not wait for each node to complete, just start the command on all nodes. Once Waiting for certificate generation to complete appears on the last node, wait roughly 60 seconds, and continue to step 7.3.
Finally, run the command on the Primary node.
The expected behavior is for the command to finish, then shortly afterwards the pending tasks on the Data nodes and Primary Replica node (if present) will complete. To ensure that the command completes successfully check:
ls -l /var/vmware/_cert_generation_completed
Run the following commands on all nodes in the vRealize Operations cluster:
chown admin:admin -R /storage/vcops/user/conf/ssl/ /storage/vcops/user/conf/ssl_bak/ /storage/db/casa/webapp/hsqldb/
chmod guo+r -R /storage/vcops/user/conf/ssl/
service vmware-casa restart
service vmware-vcops stop
sed -i 's/sliceonline\ \=\ true/sliceonline\ \=\ false/g' /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/data/roleState.properties
Run the following commands on the Primary node, and Primary Replica node (if present):
service vmware-casa stop
sed -i -e 's/\"onlineState\"\:\"GOING\_OFFLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"GOING\_OFFLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"GOING\_ONLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"GOING\_ONLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"ONLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"ONLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"FAILURE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"FAILURE\"/\"online\_state\"\:\"OFFLINE\"/g' /data/db/casa/webapp/hsqldb/casa.db.script
service vmware-casa start
service vmware-vcops-web restart
/etc/init.d/apache2 restart
Log into the vRealize Operations Admin UI as the local admin user.
Click Bring Online under Cluster Status.
Summary:
When I can see message “Data Retriever is not initialized yet. Please wait …” or “Incorrect user name/password”.
Try to check and fix for expired internal certificate in vRealize Operations.
One day I reach vRops admin login message “Incorrect user name/password”. Here is a short info How to reset the admin password in vRealize Operations according KB 2078313. But I was unable to reset. My problem was with expired internal certificate in vRealize Operations.
Reset the OS admin Password
Note: Before resetting the admin password, ensure that the account is not locked out by this command:
pam_tally2 --user admin --reset
Note: If the admin password expires for the local OS, update the local OS version of the password before completing the below steps to reset the password across the Application:
Log into the Master node as root via SSH or Console.
Run the following command to reset the password, and password age of the local admin account:
passwd admin
Repeat steps 1 and 2 on all nodes in the vRealize Operations Manager cluster, including Remote Collectors.
Reset the vRops admin Password
To reset the admin password, follow the steps below:
Log in to the Master node as root via SSH or Console.
