Hey all,
Been a little while since my last post been busy finishing up some side projects one of which is sort of relevant in this blog post which is a soon to be released book on vROps available for pre order HERE .
Today im putting this in my learning vCO (vRO) series of posts as most of the magic is in vRO not vROps.
There is a number of ways this can be done but today I thought I would look at Emails and how we can use this notification to trigger an action in vRO. Today we will look at putting a host in maintenance mode when network redundancy is lost and to exit maintenance mode when the network redundancy is restored.
The trigger for this will be the alerts in vROps 6.x.
1) We need to first configure the outbound notifications for vROps, this is done by navigating to the admin section and selecting “outbound alert settings” Here we will create an outbound alert instancing pointing it at a mail system. should have something similar to the image below.
2) Now that the alert instance has been configured we now need to create the alert notification based on the type of alert we want to target, with vROps 6.0 it is now extremely easy to alert based on exactly what you like, if it doesnt exist then we can make it but today we are just using the out of the box network redundancy lost, Navigate to the content section (little notepad icon) and select notifications. in here create a new notification based on the alert you want and the outbound notification configuration. Should have it looking similar to the below image.
take note of the Description field, this can be very handy when it comes to processing the email on the vRO side as you can put text in here unique to the type of alert or something else to validate what the email is for.
3) The alert and notification has been configured its time to do some work in vRO. we will take advantage of the a similar flow as the one we previously covered which was “Learning vCO Part 3 – bulk vm build via email“. The easiest thing to do is to take a copy of the out of the box workflow “Retrieve messages (via mail client)” as this will have allot of the attributes needed already configured.
4) Now we have this workflow which is just a script its time to make the modifications to the script so it does what we want. the following is the script I used to go through the email.
var myMailClient = new MailClient();
var tag1 = "Resource Name : "
myMailClient.setProtocol(mailProtocol);
if(useSSL){
myMailClient.enableSSL();
}
myMailClient.connect( mailServer, mailPort, mailUsername, mailPassword);
System.log("Successfully login!");
try {
myMailClient.openFolder("Inbox");
var messages = myMailClient.getMessages();
System.log("Reading messages...!");
if ( messages != null && messages.length > 0 ) {
System.log( "You have " + messages.length + " email(s) in your inbox" );
var i = 0;
var messageid = i + 1;
System.log("");
System.log("-----MSG-------");
System.log("Headers: ");
var headerProp = messages[i].getHeaders();
for each(key in headerProp.keys){
System.log(key+": "+headerProp.get(key));
}
System.log("");
System.log( "Message["+ i +"] with from: " + messages[i].from + " to: " + messages[i].to);
System.log( "Message["+ i +"] with subject: " + messages[i].subject);
var content = messages[i].getContent();
var subjectvar = messages[i].subject;
System.log("Msg content as string: " + content);
var hostname = content.match(/([A-Za-z0-9-]+(-[A-Za-z0-9-]+)*\.virtualiseme.com.au)/);
//var temphostname = [];
var temptemp = ""+hostname+"";
var temphostname = temptemp.split(",");
System.log("hostname is: " + temphostname[0]);
hostname = temphostname[0];
System.log("host name MATCHED : " + hostname);
var alerttype = subjectvar.match(/(new alert)/);
System.log(" pre alert: " + alerttype);
if (alerttype == null) {
var alerttype = subjectvar.match(/(cancelled alert)/);
}
var temptemp2 = ""+alerttype+"";
var tempalert = temptemp2.split(",");
alerttype = tempalert[0];
System.log("alert MATCHED : " + alerttype);
var match = content.match(/(vCO Network Loss)/);
var temptemp3 = ""+match+"";
var tempmatch = temptemp3.split(",");
match = tempmatch[0];
System.log("string MATCHED : " + match);
if ((match != "vCO Network Loss" && alerttype != "new alert")||(match != "vCO Network Loss" && alerttype != "cancelled alert")){
System.log("Email Message not vailid");
myMailClient.deleteMessage(messageid);
}
else {
System.log("Email Message is valid");
if (alerttype == "cancelled alert"){
var choice = "exit";
} else {
var choice = "enter";
}
System.log("Choice is; " + choice);
email = "yes";
myMailClient.deleteMessage(messageid);
}
} else {
System.warn( "No messages found" );
email = "no";
}
} finally {
myMailClient.closeFolder();
myMailClient.close();
}
Quickly running through the above code, we read the first email in the queue, and check to see if it has a valid host name in it. We then look at the subject for the type of alert, is it new or canceled and also looks at that description I was talking about in this case “vCO Network Loss” We then check to see if it is a valid email with:
if ((match != "vCO Network Loss" && alerttype != "new alert")||(match != "vCO Network Loss" && alerttype != "cancelled alert")){
System.log("Email Message not vailid");
myMailClient.deleteMessage(messageid);
}
The we look at if we want to exit maintenance mode or enter it, with:
if (alerttype == "cancelled alert"){
var choice = "exit";
} else {
var choice = "enter";
}
System.log("Choice is; " + choice);
This code is not that robust and look at doing only one thing, but this is showing a very basic implementation but as we can see here though we could have it handle allot of different actions and putting more check in etc.
The mapping for this script is shown in the below image:
5) So hopefully the script works give it some runs to validate it. Now we drag on a custom decision and exit if the there is no valid email.
6) Next we use the getallhostsystems action and drag it onto the workflow as shown in the below image.
7) We then use the result of this action and find the host we pulled out of the email by adding in a script-able task as shown below. We output the object of the type of hostsystem.
8) Next we place a custom decision in to choose if we enter or exit maintenance mode based on the subject of the email we pulled out in the first step.
9) The next part is simple drag and drop the enter maintenance mode and exit maintenance mode out of the box workflows. imputing all the required data.
That is it. We should now have a flow looking similar to the image below:
now it time to test, I used virtual ESXi hosts as I can simply turn a NIC on or off externally to simulate a NIC outage.
I triggered a nic failure and we can see the alert come through to vROPS as shown below
Now I check to see if the email comes through which is has, shown below
Now We run the work flow obviously if this was an automated response we would have the workflow set on a schedule to run very regularly
Awesome worked like a charm, Next I fix the NIC issue and vROPS should now send an alert saying that its been canceled.
Awesome looks to work well.
Trigger flows from vROps using SMTP Package
This should give a foundation to build up some alerting responses you would find useful in your environment
Cheers
Hi Scott,
I developped few weeks ago an outbound rest notification plugin that is able to call VRO worklows from VROPS UI. If you interested in cluster auto redmediation, you should take a look at this: http://pierrelx.com/?p=373
Hey Pierre,
I have seen that on your blog, it is very neat I like it allot!!
For some of my clients it not being supported limits my ability to implement it.
Cheers
Hi,
Yes I know, I did talk to VROPS PM to implement this in the next release. We will see
Would you know if VMware already released the management pack for vROPs? I can’t find it on Solution Exchange.
Hey Pierre,
Thanks for your information. I am also finding for the solution on how to generate SNMP trap from vROPs 6.x ? Is there any update so that we can generate the SNMP trap from there and after getting that trap in vRO (i am using vRO7.x) I would like to implement the below use cases:
1. vCenter sends trap to vROPS.
2. vROPS reports the error to vRO via SNMP trap.
3. vRO scales up the resources as per requirement.
4. End user is notified after the scale up.
Thanks,
Koushik
Hi Koushik,
you can configure the outbound alert settings to use SNMP. this is unfiltered though and the filtering would need to done on the receiving end example vRO.
But vROps currently can not receive SNMP, I think there was a plugin but it lacked the ability to add MIBs,
From the 4 points there all are doable apart from the first one but as vROps has native connectivity with vCenter you shouldn’t have to send snmp traps for the alert.
Cheers
Scott
Hi Scott,
Thanks for the detailed blog on this topic.
I am trying to do the same as Koushik . So We dont really have to install SNMP plugin with vROPS and my vRO workflow can be triggered directly from the notification that we configure in vROPS?
Thanks,
Hi Sam,
Out side of the ootb actions or custom solutions like Pierre showed, you can configure outbound notifications like smtp or snmp then have vRO or another external system interpret those and do the action.
I think we will start to see more tighter integration on this front going forward where this would be more ootb functionality rather than working with whats available and coupling up pieces of functionality.
Thanks Scott !!
Hey Scott,
I’d like to make use of this:
https://solutionexchange.vmware.com/store/products/vro-solution-and-workflow-package-for-vrealize-operations-manager
However I can’t seem to find any instructions on how to configure it once it has been installed. I assume it is similar to what you are doing here but not sure what I have to configure in terms of outbound and alert notifications.
What I would like to achieve is powering on an additional VM via VCO workflow when vROPS triggers an alert (i.e high CPU) on a particular VM.
Any ideas on how I need to go about this?
Cheers
Hey GD,
I remember installing that solution but never actually used it. But all it adds is a couple of additional actions, while you could modify the vRO workflows and change what its doing still limited.
Think we should start seeing some better ootb integration to achieve what you are looking for down the track. but for now there is a number of ways.
Can use the smtp like I have or can setup vRO to accept snmp traps to kick off a workflow based on that. Either way it will involve a bit of smarts in the vRO flows.
Hi, i have vRops 6.3 and vRO 6.0.3, i installed the integration package in vRO and configured it to capture traps sent by vRops. The traps are caught using a vRO policy, the problem i am having is during filtering of the traps, i notice the OIDs that vRops sends for the alerts don’t match the OIDs defined in the vRO policy. the vCOPs SNMP trap policy *installed on vRO expects this OID > “1.3.6.1.4.1.6876.0.19”: “healthWorkloadNew”,
but the traps that vRO is catching are only this>
“1.3.6.1.4.1.6876.4.5.1.0.46”, “1.3.6.1.4.1.6876.4.5.1.0.47”
as a workaround i edited the processsnmpresult action in vRO that gets the server name from this
else if (item.oid == “1.3.6.1.4.1.6876.2.1”) {
vcopsProp.put(“serverName”, item.stringValue);
to the OID value that is sent by VROps>
else if (item.oid == “1.3.6.1.4.1.6876.4.5.1.2.2.0”) {
vcopsProp.put(“serverName”, item.stringValue);
that way it gets now the server name from the trap, but i would need to do all this changes to the rest of the script. could it be i need to install something else so that the OIDs match?
Can someone direct me to the docs outlining how to install the “integration package” that will allow me to configure vRO to receive and trigger flows based on SNMP traps?
Also, I don’t think Pierre’s nifty code was ever adopted by VMware, is that a fair statement?
Thanks!
Hey Tim,
https://pubs.vmware.com/orchestrator-plugins/index.jsp#com.vmware.using.snmp.plugin.doc_10/GUID-996B84E8-369C-462B-ADC1-657E961E9446.html is the documentation on the default snmp plugin, it appears to have all the information you should need.
Yeah I think that is a fair statement.
Cheers