SNMP, likely one of the oldest protocols in OSI Layer 7, is still used today in the monitoring environments and offers a very simple way, with SNMP traps, to send messages from one system to another to trigger an action. This action could involve executing an automated workflow.
In this post, I describe how the VMware Aria Operations SNMP Trap Plugin can be used to execute automated workflows in Aria Automation Orchestrator in response to alarms triggered in VMware Aria Operations.
In Aria Operations you can utilize the Outbound Settings to configure your communication preferences, allowing you to send information to users or applications external to Aria Operations. In addition to the most well-known plugin here, the Standard Email Plugin, we also have access to other plugins like the SNMP Trap Plugin.
Before we proceed with configuring in Aria Operations, we need an SNMP receiver to receive our traps, evaluate them, and trigger an automated action. We’ll build this receiver in VMware Aria Automation Orchestrator. The official documentation can be found here: https://docs.vmware.com/en/VMware-Aria-Automation/8.16/Using-Automation-Orchestrator-Plugins/GUID-B54F7000-D3F8-4315-8DC9-23485B2BDBF5.html
The SNMP plugin in Aria Orchestrator provides two methods of communication with the SNMP devices.
- Queries for the values of specific SNMP variables.
- Listening for events (SNMP traps) that are generated from the devices and pushed to the registered SNMP managers.
For our use case, the second method is what we need, receiving SNMP traps.
Aria Orchestrator Basic Setup
The first step is to configure the port for the SNMP receiver and expose it for the outside world. In my example I am using UDP and the default port number 162. On the Aria Automation console following commands are needed.
SNMP_PORT=162
vracli network ports expose --proto udp --targetPort $SNMP_PORT $SNMP_PORT vco-appIn the next step we need to run the Set the SNMP trap port workflow to set the port on which Automation Orchestrator will start listening for incoming SNMP traps – in my case, 162 as specified in the previous step.
Using netstat on the command line we can test if the kube-apiserver is now waiting for incoming messages on UDP port 162.
root@ariaauto [ ~ ]# netstat -tulpena | grep :162
udp 0 0 0.0.0.0:162 0.0.0.0:* 0 28271956 4643/kube-proxy
Aria Automation Orchestrator will show us the newly configured Trap Host in the inventory.
SNMP Trap Host Policy Configuration
The next step is to configure and run a Aria Automation Orchestrator Policy.
The SNMP Trap Host policy in Aria Orchestrator basically listens for SNMP traps and according to its configuration it executes scripts or workflows. In my environment I have cloned the existing SNMP Trap Host policy and configured my own policy to meet my requirements, tk-SNMP Trap Host.
This customized policy will execute a script on OnTrapAll event. The script itself will execute a workflow and pass the SNMP trap data. The workflow configured in the script has to have an input parameter trapData (this is how I have specified it in my script) of the type Array/Properties. The workflowID variable contains the ID of the Orchestrator workflow to execute.
And this is the script code to make it easy to copy and paste:
// Replace the workflowID value with the ID of your workflow
// The workflow need an Input parameter "trapData"
var workflowID = "504f26ea-0ae6-426f-82d1-5fe332228cb6";
// process SNMP trap
var key = event.getValue("key");
var snmpResult = SnmpService.retrievePolicyData(key);
var trapData = System.getModule("com.vmware.library.snmp").processSnmpResult(snmpResult);
var myWorkflow = Server.getWorkflowWithId(workflowID);
if (myWorkflow == null) {
throw "Workflow not found";
}
var myWorkflowParams = new Properties();
myWorkflowParams.put("trapData",trapData);
var wfToken = myWorkflow.execute(myWorkflowParams);Running the tk-SNMP Trap Host will create a new Policy Run.
Every Policy Run can be reconfigured in stopped mode, for example to ensure that the Policy Run will be started on every server start-up.
Note: If you change anything in the Policy itself, you will need to create a entirely new Policy Run based on the altered Policy to take effect.
The Workflow
To finally be able to create and test a suitable SNMP Trap Plugin Outbound instance on the Aria Operations side, and to verify which data a trap contains in this specific use case, I have created a very simple Orchestrator Workflow. As mentioned earlier, this workflow includes a single input parameter to capture the SNMP trap data.
Using this simple code, which I will extend later on, we can check the structure of a test SNMP trap send by Aria Operations. Interesting parts are the OIDs and the actual values.
And this is the initial code for copy and paste:
System.log("Trap Data: " + trapData);
for (var i = 0; i < trapData.length; i++) {
var prop = trapData[i];
System.log(prop.get("oid") + " = " + prop.get("value"));
}Aria Operations Setup
With this preliminary work, we can finally create, configure, and test the necessary components in Aria Operations. The first step is to create a SNMP Trap Plugin Outbound Instance within the Configure Alerts section.
The only options we need to specify when creating the instance are the destination host, so in this case my Aria Automation Orchestrator FQDN and the SNMP Trap receiver port I have specified in the first step.
Figure 10: Configuring SNMP Trap Outbound Instance.
Now we can hit the Test button in the newly created Outbound Instance and check if our Workflow is getting started. If everything has been created and configured correctly you should see this in the Workflow log section as log output:
2024-05-13 13:45:18.512 +02:00INFO1.3.6.1.2.1.1.3.0 = 1908767417
2024-05-13 13:45:18.513 +02:00INFO1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.4.1.6876.4.50.1.0.200
2024-05-13 13:45:18.514 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.1 = localhost
2024-05-13 13:45:18.515 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.2 = Test
2024-05-13 13:45:18.516 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.3 = Test
2024-05-13 13:45:18.517 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.4 = 1715600718121
2024-05-13 13:45:18.518 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.5 = Info
2024-05-13 13:45:18.519 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.6 = Test trap
2024-05-13 13:45:18.520 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.7 =
2024-05-13 13:45:18.521 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.8 = 0
2024-05-13 13:45:18.522 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.9 = Test info message
2024-05-13 13:45:18.523 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.10 = Test type
2024-05-13 13:45:18.524 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.11 = Test subtype
2024-05-13 13:45:18.525 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.12 = Test Health
2024-05-13 13:45:18.526 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.13 = Test Risk
2024-05-13 13:45:18.527 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.14 = Test Efficiency
2024-05-13 13:45:18.528 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.15 = Test Metric Name
2024-05-13 13:45:18.529 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.16 = Test Object Type
2024-05-13 13:45:18.530 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.17 = Test Alert Definition Name
2024-05-13 13:45:18.531 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.18 = Test Alert Definition Description
2024-05-13 13:45:18.532 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.19 = Test Alert Impact
2024-05-13 13:45:18.533 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.20 = Test Notification RuleWith the knowledge of the SNMP Trap structure we can start modifying our actual Aria Automation Orchestrator Workflow and add some more sophisticated logic to it to do some actual work. In the new code I have added checks for the host name and the criticality of the alert. Feel free to use your own logic and extend this workflow basic code.
System.log("Trap Data: " + trapData);
for (var i = 0; i < trapData.length; i++) {
var prop = trapData[i];
System.log(prop.get("oid") + " = " + prop.get("value"));
}
for (var i = 0; i < trapData.length; i++) {
var prop = trapData[i];
if (prop.get("oid") == "1.3.6.1.4.1.6876.4.50.1.2.1") {
System.log("Host name in the message: " + prop.get("value"))
}
}
for (var i = 0; i < trapData.length; i++) {
var prop = trapData[i];
if (prop.get("oid") == "1.3.6.1.4.1.6876.4.50.1.2.5.0") {
if (prop.get("value") == "critical") {
System.log("CRTITICAL ALERT OCCURED! Do something:-)");
}
}
}The last step in Aria Operations is to create and configure a Notifications which will use the SNMP Trap Plugin Outbound Instance.
And this is how a real SMTP Trap message looks like when used in the Aria Operations Notification.
2024-05-14 11:54:03.470 +02:00INFO1.3.6.1.2.1.1.3.0 = 1988492837
2024-05-14 11:54:03.474 +02:00INFO1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.4.1.6876.4.50.1.0.46
2024-05-14 11:54:03.478 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.1.0 = vrops01.mydomain.eu
2024-05-14 11:54:03.481 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.2.0 = esxi07.mydomain.eu
2024-05-14 11:54:03.485 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.3.0 = General
2024-05-14 11:54:03.491 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.4.0 = 1715680441062
2024-05-14 11:54:03.496 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.5.0 = critical
2024-05-14 11:54:03.500 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.6.0 = New alert by id 71dd412d-de66-453f-8b12-dcf8b66a054b is generated at Tue May 14 09:54:01 UTC 2024;
2024-05-14 11:54:03.504 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.7.0 = https://192.168.0.28/ui/index.action#environment/object-browser/hierarchy/64ac1118-826c-4470-9f9d-ad24e5757eab/alerts-and-symptoms/alerts/71dd412d-de66-453f-8b12-dcf8b66a054b
2024-05-14 11:54:03.511 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.8.0 = 71dd412d-de66-453f-8b12-dcf8b66a054b
2024-05-14 11:54:03.514 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.9.0 = symptomSet: f9f06558-8c14-4f65-a0c8-bf5307bc6765
relation: self
totalObjects: 1
violatingObjects: 1
symptom: Host disconnected from vCenter
active: true
obj.1.name: esxi07.mydomain.eu
obj.1.id: 64ac1118-826c-4470-9f9d-ad24e5757eab
obj.1.metric:
obj.1.info: Property not equal notResponding != connected
symptom: SymptomDefinition-VMWARE-HostIsPlacedInStandbyMode
active: false
obj.1.name: esxi07.mydomain.eu
obj.1.id: 64ac1118-826c-4470-9f9d-ad24e5757eab
2024-05-14 11:54:03.517 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.10.0 = Virtualization/Hypervisor
2024-05-14 11:54:03.520 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.11.0 = Availability
2024-05-14 11:54:03.523 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.12.0 = critical
2024-05-14 11:54:03.526 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.13.0 =
2024-05-14 11:54:03.529 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.14.0 =
2024-05-14 11:54:03.532 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.15.0 =
2024-05-14 11:54:03.535 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.16.0 = HostSystem
2024-05-14 11:54:03.538 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.17.0 = Host has lost connection to vCenter Server
2024-05-14 11:54:03.541 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.18.0 = The host has been unexpectedly disconnected from vCenter Server. The virtual machines on the host have also lost connectivity to vCenter Server. The host might not have a problem and the virtual machines on the host might be running as expected, but any configuration, data, and events from the host and its virtual machines are lost.
2024-05-14 11:54:03.544 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.19.0 = health
2024-05-14 11:54:03.547 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.20.0 = tk-GenericAllEventsToSNMP
2024-05-14 11:54:03.551 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.21.0 =
2024-05-14 11:54:03.555 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.22.0 =
2024-05-14 11:54:03.559 +02:00INFO1.3.6.1.4.1.6876.4.50.1.2.23.0 =
2024-05-14 11:54:03.565 +02:00INFOCRTITICAL ALERT OCCURED! Do something:-)Now have fun coding your own automation solutions based on SNMP traps sent by Aria Operations and received by Aria Automation Orchestrator.
Stay safe.
Thomas – https://twitter.com/ThomasKopton