Part 1 and Part 2 of the “Maintenance Mode in vRealize Operations” blog series leveraged features and functionalities of vROps and vSphere which are directly related to ESXi maintenance to fulfil the requirements.
What if you cannot use “Enter Maintenance Mode” in vCenter simply because there is no such option for e.g. Virtual Machines, Datastores or Clusters but your use case requires some kind of maintenance mode in vROps?
In this part we focus on this kind of scenario. In case you do not have to deal with such scenarios, no worries, your time won’t be wasted, you are going to learn how to use vCenter Tags and Custom Attributes in vROps.
The use case for this blog post is:
“In case of VM/Datastore/Cluster maintenance (whatever “maintenance” might be) I do not want to receive any alerts for the affected vROps objects.”
As always during assessment we collected following additional
information:
There might be severalobjects in maintenance at the same time
The team doing the maintenance has read-only access to vROps UI
The team doing the maintenance is able to edit vSphere Tags or Custom Attributes for the objects in maintenance
Automation could be used but is not mandatory
Metrics and properties need to be collected during the maintenance.
Obviously, we cannot use the OOB vROps Maintenance feature described in Part 1 as this would violate the last requirement and the vCenter team would need additional permissions in vROps. The vROps property “Maintenance State” we already used in Part 2 is not an option here, as this property is available only for Host System objects.
We need another property to designate objects in maintenance. This is where vSphere Tags and/or Custom Attributes come in handy.
The ingredients for one possible solution are:
vSphere Tags and Custom Attributes – Interface for the vCenter team to start the host maintenance
vROps Policy – place where we modify the behavior of vROps with regard to distinct objects
vROps Custom Group – place where we group objects to apply a vROps policy to them
In Part 2 I have described (simplified) the model of Objects, Custom Groups and Policies in vROps.
Step 1 – Create a vROps
Policy which implements the requirements.
In our case the requirement is to disable all alerts
for Virtual Machines / Datastores / Clusters etc. during maintenance hence we
need a vROps Policy in which we are going to disable the corresponding alert
definitions.
A new policy is being created based on an existing one, so only few changes will be needed to tweak the default behavior of vROps. In the following picture we see alert definitions for Virtual Machines as an example. Previous parts describe the procedure of creating and modifying the vROps policy.
Figure 1: Disabling vROps Alert Definitions in the new vROps Policy
Step 2 – Create vROps Custom Group which will contain objects being in maintenance.
Again, “maintenance” describes here any work being
done on objects which requires certain alert definitions being disabled temporarily.
In Part 2 I have already described how to create a vROps Custom Group. I will skip the common details here and focus on the membership criteria which is different in this scenario.
There are also differences in vROps when it comes to the properties reflecting both concepts.
vCenter Custom Attributes
Custom
Attributes are easy to use in vROps but the handling in vCenter might be
cumbersome.
Let’s assume we want to conduct some kind of maintenance on certain VMs and a Datastore. In vCenter we assign an appropriate attribute to the objects we want to set into maintenance. Type “Global” allows us to use the same attribute on different object types.
Figure 2: vCenter Custom Attributes
After the next vROps collection cycle has been completed, this custom attribute is available as property.
Figure 3: vCenter Custom Attribute in as vROps Property
And this is where the confusion begins, vCenter Custom Attributes are vROps Custom Tags properties.
These properties are easy to use in vROps Custom Groups membership criteria definitions. Extra tip, if you cannot find the property in the list presented by vROps as “common denominator”, please use the object picker to select an object which is having the required property (or metric).
Figure 4: Find newly created properties – extra tip
My sample group containing both, VMs and Datastores has the following membership criteria configuration.C
Figure 5: Custom Group membership criteria – Custom Attributes
And automatically contains my two objects.
Figure 6: Group members
The members of the group now receive the new policy with disabled alert definitions.
Figure 7: vROps Policy assigned through Custom Group membership
To remove
objects from the new custom group and let them receive the old policy, we
simply set the vCenter attribute to no.
As already mentioned in Part 2 of this blog series, it may take up to 20 minutes until the membership has be re-evaluated and the objects have been removed from the group.
vSphere Tags
In contrast to vCenter Custom Attributes vSphere Tags are easy to use in vCenter, assigning tags to multiple objects can be done in the UI with few clicks.
This is a simple tag to categorize objects in maintenance.
Figure 8: vSphere Tag exampleFigure 9: Assigning vSphere Tags in vCenter
Again, the
next vROps collection cycle needs to complete to collect the new value.
The corresponding property is vROps differs from the property reflecting vCenter Custom Attributes. There is only one property called “vSphere Tag” and the value of this property contains all tags assigned to the particular object, following the format:
[<CategoryX-TagX>,<CategoryY-TagY>,…]
Figure 10: vSphere Tags collected in vROps as one property
That means, that we need to use the “contains” operator in the membership criteria instead of “is” as for custom attributes. That also means, that you need to be carful with naming your tags to avoid overlaps which may result in unwanted behavior of you group membership criteria.
Figure 11: Custom Group membership criteria – vSphere Tags
As expected, the group contains three objects which got the maintenance vSphere Tag assigned.
Figure 12: vROps Custom Group members
As with
custom attributes, remove the tag after the maintenance has been completed,
wait few minutes and the objects leave the custom group automatically.
This is the last part of the “Maintenance Mode for vRealize Operations Objects” blog series.
I wish you and your loved ones a a healthy start into the New Year 2020.
The vRealize Operations REST API allows using RegEx expressions in various GET methods.
Sometimes it is not clear how to use the expressions. Here a very simple example of using RegEx to retrieve vROps Virtual Machine objects, which have VM names starting with certain strings.
Part 1 of the “Maintenance Mode in vRealize Operations” blog series focused on a very basic scenario.
Simplified, we turned off data collection and made the object basically not existent from vROps point of view as the requirements were straight forward. To accomplish this, we used the vROps Maintenance feature.
In this part we consider a slightly more complex scenario. The use case for this blog post is:
“In case of ESXi host maintenance mode I do not want to receive any alerts for the affected Host System objects.” As always during assessment we collected following additional information:
There are only few ESXi hosts in maintenance at the same time
The team doing the maintenance in vCenter has read-only access to vROps UI
Automation could be used but is not mandatory
Metrics and properties need to be collected during the maintenance
Obviously,
we cannot use the vROps Maintenance feature as this would violate the last
requirement and the vCenter team would need additional permissions in vROps.
The ingredients for one possible solution are:
vCenter Maintenance Mode – Interface for the vCenter team to start the host maintenance
vROps Policy – place where we modify the behavior of vROps with regard to distinct objects
vROps Custom Group – place where we group objects to apply a vROps policy to them
The following picture describes (simplified) the model of Objects, Custom Groups and Policies in vROps:
Figure 1: Simplified object model – vROps objects and policies
Step 1 – Create a vROps Policy which implements the requirements.
In our case
the requirement is to disable all alerts for Host Systems during ESXi
maintenance hence we need a vROps Policy in which we are going to disable the
corresponding alert definitions.
A new policy is being created based on an existing one, so only few changes will be needed to tweak the default behavior of vROps.
Figure 2: Creating new vROps Policy
In my test environment I am using my default policy as the base line for the new policy.
Figure 3: Setting the policy baseline
We go to the Alert Definitions section and disable the corresponding alert definitions for the Host System objects.
To reduce the list of alert definitions some filtering options may be applied. In the following picture you see filters for object type and the current state of the alert definitions. We do not care about disabled alert definitions as this is the is the desired state.
Figure 4: Alert definitions – display filter
The easiest way to get rid of all vROps Host System related alerts is to select all alert definitions in the list and disable them locally.
NOTE: This is a very simplified approach. In real life scenarios the requirements may be more complex and could result in alert definitions left enabled even during ESXi maintenance operations.
Figure 5: Disabling all Alert Definitions – 1Figure 6: Disabling all Alert Definitions – 2
Step 2 – Create vROps Custom Group which will contain ESXi hosts being in Maintenance Mode
Following the simplified object model we need a dedicated vROps Custom Group to pool the objects we want to receive the new policy.
Figure 7: Creating a vROps Custom Group
“Group Type” is just an arbitrary categorization to reflect your environment, once you have created a custom group, you cannot change the “Group Type” (at least not using the UI).
In Policy we select the newly created vROps Policy which disables the alert definitions.
“Keep group membership up to date” ticked ensures that the membership will be re-validated every 20 minutes.
To tell vROps which object should be members of the group we need to specify appropriate membership criteria.
Figure 8: Specifying the vROps Custom Group – 1
In the previous figure we select the “Maintenance Mode” property of the Host System object type as the criteria to determine if a certain ESXi host system should become member of the custom group or not.
In the following picture we see a host being in vCenter Maintenance Mode.
Figure 10: vCenter – ESXi in Maintenance Mode
The “Preview” button let us check the configured membership criteria.
Figure 11: Host in Maintenance Mode and member of the custom group
Now the ESXi host system is member of the custom group and receives the new policy which disables all alert definitions for this particular object. In vROps one can see the current policy for any selected object in the upper right corner.
Figure 12: Host in Maintenance Mode receiving the new vROps Policy
Once the maintenance has been completed, vCenter admin disables the Maintenance Mode in vCenter and after up to 20 minutes the ESXi host is no longer member of our custom group and the previous (could be default) policy is being applied again.
Figure 13: Maintenance Mode ended – ESXi host receiving the original vROps Policy
As you probably noticed, one drawback of this method is a possible gap of max. 20 minutes between entering the vCenter Maintenance Mode for a host and vROps completing the re-evaluation of the custom group membership. That fact should be taken into account while designing the maintenance procedure.
In case automation plays a role in your environment and the policy needs to be applied immediately after entering the vCenter Maintenance Mode the corresponding vROps REST API call can be leveraged to programatically retrieve a list of ESXi hosts from vCenter and populate the custom group with the appropriate vROps objects. But this is another story…
Figure 14: vROps Custom Group REST API method
In Part 3 I will focus on use cases comprising additional objects types.
In this and following posts I will show you few different ways of putting vROps objects into maintenance.
Objects in vROps – short intro
The method used to mark an object as being in the maintenances state depends on the actual use case. As usually, the use case itself is being defined by:
Requirements – what does “maintenance” mean from technical perspective, what exactly needs to be achieved?
Constraints – is there any automation in place, which team is involved, what and how many objects are involved?
Assumptions – can we assume that especially the number and kind of objects will stay the same or what is the expected growth?
Risks – what are the risks of using a certain method?
Let us assume our first use case is:
“In case of ESXi host maintenance mode I will to stop collecting any data for this host and disable all alerts.”
As always, before we start any design and implementation, we did a proper assessment and collected further information:
There are only few ESXi hosts in
maintenance at the same time
The team doing the maintenance in
vCenter can also access and use the vROps UI
Automation could be used but is not mandatory
All implications from stopping
metrics and property collection for a given object, like ESXi host are known
and accepted
Let us first look at one specific vROps object, in that case a Host System (ESXi host), using the vROps inventory option:
Host System object in vROps Inventory – “Show Detail” option
We see that the object is properly collecting metrics and properties according to both indicators. The details of the selected object can be checked by clicking the “Show Detail” button. This redirects you to the Summary page of the object. The currently collected metrics and properties can be checked by activating the “Show collecting metrics” option:
Collected metrics for the selected object
Activating the maintenance mode – the UI-way
The easiest way to put an object into maintenance mode is to use the “Start Maintenance” button in the Inventory overview:
Enabling Maintenance Mode in vROps UI
In the following dialog you can specify how long the object should be put into maintenance:
Specifying the Maintenance duration
After starting the Maintenance, you can again check the new status of the object in the Inventory view:
Object status during vROps Maintenance
Now, if you use the same “Show collecting metrics” option in the metrics tab of the object you can see that there are no metrics or properties collecting data. The object stopped the data collection entirely:
No collected metrics and/or properties during vROps Maintenance for an object
At this point you need to know that from the monitoring perspective this object is still in the inventory but the is no single data point being collected, stored and calculated in any way. Any calculations relying on data points coming in for that particular object will not provide new data or calculate nor entirely correct data. What “correct” means, depends on the actual metric, dashboard, view etc.
Deactivating the maintenance mode – the UI-way
As easy as we started the maintenance as easy it can be stopped again using the UI:
Disabling Maintenance Mode in vROps UI
After clicking on the “End Maintenance” button, vROps will start collecting all data for the object again.
Activating the maintenance mode – the REST-API-way
Starting and ending the Maintenance Mode using the UI is easy and convenient if you have to deal with a small number of objects and there are no other constraints like complying with e.g. change management process which may require automation.
If you need to deal with a large number of objects or if the vROps Maintenance Mode should be part of an automated process, leveraging the vROps API is the best way to implement it.
As always when using the REST API, the first step is to obtain the Access Token. To acquire the token, following POST method needs to be used:
POST /api/auth/token/acquire
aquireToken REST API call
Once we have a valid token, we can call the Maintenance Mode related operations. Following REST operation starts the Maintenance Mode for a given object:
REST API method to start the maintenance mode
As you can see, you will need to determine the vROps Object ID of the object(s) you need to put into maintenance before you can call the actual Maintenance Mode calls.
Once you have the ID(s) the method can be used:
REST API Call – Start Maintenance Mode
Deactivating the maintenance mode – the REST-API-way
To end the maintenance following REST API method has to be used:
REST API Call – End Maintenance Mode
Again, you will need the vROps Object ID to call this method.
Part 2 – Outlook
In the upcoming Part 2 of this post, I will describe other methods which may be used in cases when the requirements differ from the use case described in this post.
But what if you would like to use the outstanding vRealize Operations engine to manage and visualize objects which cannot be collected using the rich Management Pack ecosystem?
Let’s imagine you have a cool Smart Home system and you would like to get it integrated into your vRealize Operations. You would like to have all the various elements as objects in vRealize Operations to push metrics and properties to those objects.
In this post I will show you how to create your own custom environments in vRealize Operations using REST and vRealize Orchestrator.
Of course, this is just an example and the environment, and the corresponding inputs are “virtual”. The used vRealize Orchestrator workflows are examples and there are different ways to achieve the same outcome.
The vRealize Orchestrator workflows, actions etc. related to this post can be found here:
There are several REST API calls available to create new objects in vRealize Operations. In this example we will use the method “createResourceUsingAdapterKind”:
POST /api/resources/adapterkinds/{adapterKindKey}
A sample URL body of the call can be as complex as in the official documentation or as easy as this one:
This Entry will initially create our OPENAPI adapter instance for the new custom environment using the REST API. This environment will reflect a Smart Home installation containing various devices. This call will create a lightning device located in the living room. How to execute that call using vRealize Orchestrator will be described later on.
Input data
If we are going to use automation to create our objects it would be not very sophisticated to enter every value manually. Therefor as first step we design and create a JSON file describing the model of our Smart Home. This is an example how a very simple model may look like. The included properties and metrics will play a role in some subsequent blog posts.
In this JSON example our Smart Home consists of three types of devices: Climate, Door and Lightning.
The instances of those devices can be located in different rooms and have various properties and metrics.
vRealize Orchestrator Preparation – Resource Element
To consume our JSON file in a vRealize Orchestrator workflow we need to import that file as “Resource Element”.
JSON file as vRealize Orchestrator Resource Element
After importing the JSON file into vRealize Orchestrator it can be used as an attribute in any vRealize Orchestrator workflow.
vRealize Orchestrator Preparation – Configuration Element
Acquiring a valid vRealize Operations authentication token is one part of our workflow. To make the process as automated as possible, we will store frequently used values in a vRealize Orchestrator “Configuration Element”. Such values are for example user credentials and token related information.
vRealize Orchestrator Configuration Element
A vRealize Orchestrator configuration element is a kind of dictionary data structure storing key:value pairs for ease of use in vRealize Orchestrator workflows.
vRealize Orchestrator Preparation – REST Endpoint and Operations
The last pre-requisite is a working REST endpoint including REST operations configured in our vRealize Orchestrator instance. vRealize Orchestrator provides appropriate workflows in the out-of-the-box library to add new REST hosts and operations offered by those hosts. The following figure shows the location of the workflows which can be used to configure a REST API provider.
The actual job is being done in the CreateCustomObjects workflow:
CreateCustomObjects Workflow
After some URL-encoding for inputs containing e.g. white spaces, we collect information about the new objects using the JSON file (newObjecstInfo), the code snippet here has been shortened and includes only one object category. All workflows can be .
template = newObjectsJSONFile.getContentAsMimeAttachment();
jsonObject = JSON.parse(template.content);
var objects1 = jsonObject.SmartHome["Climate"];……
newResources1 = new Array();
for (object1 in objects1) {
newResources1.push(objects1[object1].Name);
}
numberResource1 = newResources1.length;
Code 5: newObjectsinfo Snippet
What we are doing here is basically creating an array of strings containing the names of our new custom objects. At this point the code needs to be adopted to the model defined in the JSON file to parse the data correctly.
The next steps are fairly simple, we check the validity of the token which we saved in a configuration element and acquire a new token in case our token is valid for less than 10 minutes:
var vropsTokenValidity = vropsConfig.getAttributeWithKey("tokenValidity");
if (vropsTokenValidity.value != null) {
var dateNow = new Date();
var diff = vropsTokenValidity.value - dateNow.getTime();
tokenRemainigVaidity = diff / 1000 / 60;
} else {
tokenRemainigVaidity = 0;
}
Code 6: checkToken Snippet
The following scripting elements take care of looping through the arrays of strings to create all object one by one.
The main part related to creating new vR Ops objects consists of creating the JSON body:
The very last step is to execute the REST API call to create a custom object:
var params = [encodedAdapterKind];
var token = vropsConfig.getAttributeWithKey("token").value;
var request = createResourceCall.createRequest(params, JSON.stringify(jsonBody));
request.contentType = "application/json";
request.setHeader("accept", "application/json");
request.setHeader("Authorization", "vRealizeOpsToken " + token);
var response = request.execute();
if (response.statusCode >= 400) {
System.log(response.contentAsString);
throw "HTTPError: status code: " + response.statusCode;
}
Code 8: callREST Snippet
If everything worked out as expected, we will see our new object types and instances of those types in vRealize Operations.
After importing the vRealize Orchestrator package you will need to configure the REST operation according to your environment:
Workflow Attributes – REST Operation
Our Smart Home in vRealize Operations
The Environment view in vRealize Operations 7.0 including our custom objects:
vRealize Operations Environment View
In some subsequent posts I am going to explain how to push metrics and properties to such custom objects.
Some other interesting object types which may become relevant in the future, where vRealize Operations instances will be found even on space ships use the following JSON as input:
As many of you are probably aware, vROps provides an easy and comprehensive way to monitor and manage a vRealize Automation (vRA) environment through utilizing the Native Management Pack (NMP) for vRealize Automation.
The MP gives you a variety of pre-configured dashboards, view, alerts etc. which all work perfectly fine and give you a sufficient view of the environment if your vRA is dealing with only one tenant.
Of course are vROps and the Management Pack capable of collecting and displaying information for multiple tenants, the problem is, that some content aggregates numbers across all tenants. You can see this in the following screenshot:
OOB vRA Environment Overview Dashboard
But what if you have multiple tenants and you would like to see e.g. environment numbers for each and every tenant separately? How many deployments have been done by tenant X and how many Linux VMs have been deployed by tenant Y?
In this blog I will show you the basic techniques you can utilize to create your own content focusing on a per tenant view.
At the end of this blog post you will find a link to VMware Code where you can download sample content.
The foundation of the solution described in this post are Custom Groups.To have all the needed groups in one place, I have created a distinct Group Type:
Custom Group Type
Since I cannot describe all possible requirements this are my assumptions:
vRA configured with two tenants (vsphere.local, corporation.local in my environment)
requirement for having a Dashboard displaying how many Linux and Windows VMs have been deployed per tenant
requirement for having a Dashboard displaying number of Blueprints per tenant
requirement for having a Dashboard displaying number of Deployments per tenant
Based on these requirements we need following Custom Groups in our vROps:
Custom Groups
How to populate those groups dynamically with the corresponding objects?
Actually, this is pretty easy as the Management Pack provides the required Navigation Trees:
OOB vRA MP Navigation Trees
Let’s see how to create the Custom Groups.
As an example you see here the Custom Group for Windows VMs deployed by the corporation-Tenant. The most important part is bordered in red, it is the relationship of an given object in a navigation tree.
The “contains” statement has to reflect the actual name of the tenant.
Custom Group for Windows VMs per Tenant
Obviously, for the Linux group only the Properties part looks a bit different and needs to reflect the actual environment:
Custom Group Settings – Linux VMs
Now, let’s inspect the group configuration for Deployments and Blueprints:
Custom Group Settings – Deployments per Tenant
For the Blueprints Custom Group we need to use another Navigation Tree:
Custom Group Settings – Blueprints per Tenant
As there is no metric or property reflecting the count of objects belonging to a Custom Group, appropriate Super Metrics need to be created, associated with the corresponding Custom Group Type and enabled in their respective Policy:
Super Metrics
Super Metrics enabled in the policy:
Super Metrics Enabled in the Policy
Here an example of the actual formula for the first Super Metric:
After few collection cycles the new Super Metric will show up on our new Custom Groups and the values can be used in the content.
Now we have all ingredients to create our own custom dashboard displaying the various numbers separately for all our vRA Tenants.
This is how my sample dashboard looks like:
Sample Dashboards for vRA – Tenant focused View
What is happening behind the scenes is fairly easy. The Scoreboards are configured as self-provider and are using previously created Custom Groups as source of the objects and the respective Super Metrics to show the correct numbers.
At the bottom of Dashboard some additional information regarding the selected Blueprint is displayed. To achieve this correct wiring of the sources and destinations is required:
Wiring the Components
A zip file containing all elements can be found here:
vRealize Operations Super Metrics are a very flexible and powerful way to extend the capabilities of the product way beyond the OOB content. There are many blog articles out there explaining how to basically use super metrics but only very few sources gives some examples how to put logical expressions into your formulas. So the question is, how dos this work?
vROps Super Metric
Using some simple examples I am going to explain how the magic of logical expressions work in vROps Super Metrics.
First of all some fundamentals:
Super Metric working on a selected object itself, like ESXi cluster in this example, which is just showing the actual metric (we will need soon):
avg(${this, metric=summary|total_number_hosts})
Super Metric working on direct descendants of a selected object, in this case ESXi hosts in a cluster, which is counting the powered on hosts:
One could translate this formula into that statement:
If all ESXi hosts in a given cluster are powered on AND clusters average CPU usage is greater than 40 THEN show me the average CPU usage, ELSE show me 5
Working vROps Super Metric
Have fun creating your own Super Metrics based on logical expressions.
vRealie Operations Super Metrics are a very flexible and powerful way to extend the capabilities of the product way beyond the OOB content.
There are many blog articles out there explaining how to basically use super metrics but only very few sources gives some examples how to put logical expressions into your formulas. So the question is, how dos this work?
Using some simple examples I am going to explain how the magic of logical expressions work in vROps Super Metrics.
First of all some fundamentals:
Super Metric working on a selected object itself, like ESXi cluster in this example, which is just showing the actual metric (we will need soon):
avg(${this, metric=summary|total_number_hosts})
Super Metric working on direct descendants of a selected object, in this case ESXi hosts in a cluster, which is counting the powered on hosts:
One could translate this formula into that statement:
If all ESXi hosts in a given cluster are powered on AND clusters average CPU usage is greater than 40 THEN show me the average CPU usage, ELSE show me 5
