December 2019 – TOMsOps

Maintenance Mode for vRealize Operations Objects (Tags, Custom Attributes), Part 3

Part 1 and Part 2 of the “Maintenance Mode in vRealize Operations” blog series leveraged features and functionalities of vROps and vSphere which are directly related to ESXi maintenance to fulfil the requirements.

What if you cannot use “Enter Maintenance Mode” in vCenter simply because there is no such option for e.g. Virtual Machines, Datastores or Clusters but your use case requires some kind of maintenance mode in vROps?

In this part we focus on this kind of scenario. In case you do not have to deal with such scenarios, no worries, your time won’t be wasted, you are going to learn how to use vCenter Tags and Custom Attributes in vROps.

The use case for this blog post is:

“In case of VM/Datastore/Cluster maintenance (whatever “maintenance” might be) I do not want to receive any alerts for the affected vROps objects.”

As always during assessment we collected following additional information:

There might be several objects in maintenance at the same time
The team doing the maintenance has read-only access to vROps UI
The team doing the maintenance is able to edit vSphere Tags or Custom Attributes for the objects in maintenance
Automation could be used but is not mandatory
Metrics and properties need to be collected during the maintenance.

Obviously, we cannot use the OOB vROps Maintenance feature described in Part 1 as this would violate the last requirement and the vCenter team would need additional permissions in vROps. The vROps property “Maintenance State” we already used in Part 2 is not an option here, as this property is available only for Host System objects.

We need another property to designate objects in maintenance. This is where vSphere Tags and/or Custom Attributes come in handy.

The ingredients for one possible solution are:

vSphere Tags and Custom Attributes – Interface for the vCenter team to start the host maintenance
vROps Policy – place where we modify the behavior of vROps with regard to distinct objects
vROps Custom Group – place where we group objects to apply a vROps policy to them

In Part 2 I have described (simplified) the model of Objects, Custom Groups and Policies in vROps.

Step 1 – Create a vROps Policy which implements the requirements.

In our case the requirement is to disable all alerts for Virtual Machines / Datastores / Clusters etc. during maintenance hence we need a vROps Policy in which we are going to disable the corresponding alert definitions.

A new policy is being created based on an existing one, so only few changes will be needed to tweak the default behavior of vROps. In the following picture we see alert definitions for Virtual Machines as an example. Previous parts describe the procedure of creating and modifying the vROps policy.

*Figure 1: Disabling vROps Alert Definitions in the new vROps Policy*

Step 2 – Create vROps Custom Group which will contain objects being in maintenance.

Again, “maintenance” describes here any work being done on objects which requires certain alert definitions being disabled temporarily.

In Part 2 I have already described how to create a vROps Custom Group. I will skip the common details here and focus on the membership criteria which is different in this scenario.

Now we have two options:

vCenter Custom Attributes
vSphere Tags

To understand the differences please see:

https://www.virtuallyghetto.com/2015/01/custom-attributes-vsphere-tags.html

There are also differences in vROps when it comes to the properties reflecting both concepts.

vCenter Custom Attributes

Custom Attributes are easy to use in vROps but the handling in vCenter might be cumbersome.

Let’s assume we want to conduct some kind of maintenance on certain VMs and a Datastore. In vCenter we assign an appropriate attribute to the objects we want to set into maintenance. Type “Global” allows us to use the same attribute on different object types.

After the next vROps collection cycle has been completed, this custom attribute is available as property.

*Figure 3: vCenter Custom Attribute in as vROps Property*

And this is where the confusion begins, vCenter Custom Attributes are vROps Custom Tags properties.

These properties are easy to use in vROps Custom Groups membership criteria definitions. Extra tip, if you cannot find the property in the list presented by vROps as “common denominator”, please use the object picker to select an object which is having the required property (or metric).

*Figure 4: Find newly created properties – extra tip*

My sample group containing both, VMs and Datastores has the following membership criteria configuration.C

*Figure 5: Custom Group membership criteria – Custom Attributes*

And automatically contains my two objects.

The members of the group now receive the new policy with disabled alert definitions.

*Figure 7: vROps Policy assigned through Custom Group membership*

To remove objects from the new custom group and let them receive the old policy, we simply set the vCenter attribute to no.

As already mentioned in Part 2 of this blog series, it may take up to 20 minutes until the membership has be re-evaluated and the objects have been removed from the group.

vSphere Tags

In contrast to vCenter Custom Attributes vSphere Tags are easy to use in vCenter, assigning tags to multiple objects can be done in the UI with few clicks.

This is a simple tag to categorize objects in maintenance.

*Figure 9: Assigning vSphere Tags in vCenter*

Again, the next vROps collection cycle needs to complete to collect the new value.

The corresponding property is vROps differs from the property reflecting vCenter Custom Attributes. There is only one property called “vSphere Tag” and the value of this property contains all tags assigned to the particular object, following the format:

[<CategoryX-TagX>,<CategoryY-TagY>,…]

*Figure 10: vSphere Tags collected in vROps as one property*

That means, that we need to use the “contains” operator in the membership criteria instead of “is” as for custom attributes. That also means, that you need to be carful with naming your tags to avoid overlaps which may result in unwanted behavior of you group membership criteria.

*Figure 11: Custom Group membership criteria – vSphere Tags*

As expected, the group contains three objects which got the maintenance vSphere Tag assigned.

As with custom attributes, remove the tag after the maintenance has been completed, wait few minutes and the objects leave the custom group automatically.

This is the last part of the “Maintenance Mode for vRealize Operations Objects” blog series.

I wish you and your loved ones a a healthy start into the New Year 2020.

Quick Tip – vROps REST API requests using RegEx expressions

The vRealize Operations REST API allows using RegEx expressions in various GET methods.

Sometimes it is not clear how to use the expressions. Here a very simple example of using RegEx to retrieve vROps Virtual Machine objects, which have VM names starting with certain strings.

Encoded URL – example:

https://vrops-test.comp01.local/suite-api/api/resources?resourceKind=VirtualMachine&adapterKind=VMWARE&regex=s030v.*%7Cs030e.*%7CS030E.*

Example in Postman (of course, you need to replace the toke with a valid one):

https://vrops-test.comp01.local/suite-api/api/resources resourceKind=VirtualMachine&adapterKind=VMWARE&regex=s030v.*|s030e.*|S030E.*|S030V.*

And in Curl (of course, you need to replace the toke with a valid one):

curl -X GET \
 'https://vrops-test.comp01.local/suite-api/api/resources?resourceKind=VirtualMachine&adapterKind=VMWARE&regex=s030v.*%7Cs030e.*%7CS030E.*%7CS030V.*' \
 -H 'accept: application/json' \
 -H 'authorization: Basic xxxxxxxxxx \
 -H 'cache-control: no-cache' \
 -H 'content-type: application/json' \
 -H 'postman-token: yyyyyyyyyyyy'

it works perfectly and it is case sensitive.

Maintenance Mode for vRealize Operations Objects, Part 2

Part 1 of the “Maintenance Mode in vRealize Operations” blog series focused on a very basic scenario.

Simplified, we turned off data collection and made the object basically not existent from vROps point of view as the requirements were straight forward. To accomplish this, we used the vROps Maintenance feature.

In this part we consider a slightly more complex scenario. The use case for this blog post is:

“In case of ESXi host maintenance mode I do not want to receive any alerts for the affected Host System objects.” As always during assessment we collected following additional information:

There are only few ESXi hosts in maintenance at the same time
The team doing the maintenance in vCenter has read-only access to vROps UI
Automation could be used but is not mandatory
Metrics and properties need to be collected during the maintenance

Obviously, we cannot use the vROps Maintenance feature as this would violate the last requirement and the vCenter team would need additional permissions in vROps.

The ingredients for one possible solution are:

vCenter Maintenance Mode – Interface for the vCenter team to start the host maintenance
vROps Policy – place where we modify the behavior of vROps with regard to distinct objects
vROps Custom Group – place where we group objects to apply a vROps policy to them

The following picture describes (simplified) the model of Objects, Custom Groups and Policies in vROps:

*Figure 1: Simplified object model – vROps objects and policies*

Step 1 – Create a vROps Policy which implements the requirements.

In our case the requirement is to disable all alerts for Host Systems during ESXi maintenance hence we need a vROps Policy in which we are going to disable the corresponding alert definitions.

A new policy is being created based on an existing one, so only few changes will be needed to tweak the default behavior of vROps.

In my test environment I am using my default policy as the base line for the new policy.

We go to the Alert Definitions section and disable the corresponding alert definitions for the Host System objects.

To reduce the list of alert definitions some filtering options may be applied. In the following picture you see filters for object type and the current state of the alert definitions. We do not care about disabled alert definitions as this is the is the desired state.

*Figure 4: Alert definitions – display filter*

The easiest way to get rid of all vROps Host System related alerts is to select all alert definitions in the list and disable them locally.

NOTE: This is a very simplified approach. In real life scenarios the requirements may be more complex and could result in alert definitions left enabled even during ESXi maintenance operations.

*Figure 5: Disabling all Alert Definitions* – 1

*Figure 6: Disabling all Alert Definitions* – 2

Step 2 – Create vROps Custom Group which will contain ESXi hosts being in Maintenance Mode

Following the simplified object model we need a dedicated vROps Custom Group to pool the objects we want to receive the new policy.

*Figure 7: Creating a vROps Custom Group*

“Group Type” is just an arbitrary categorization to reflect your environment, once you have created a custom group, you cannot change the “Group Type” (at least not using the UI).

In Policy we select the newly created vROps Policy which disables the alert definitions.

“Keep group membership up to date” ticked ensures that the membership will be re-validated every 20 minutes.

To tell vROps which object should be members of the group we need to specify appropriate membership criteria.

*Figure 8: Specifying the vROps Custom Group* – 1

In the previous figure we select the “Maintenance Mode” property of the Host System object type as the criteria to determine if a certain ESXi host system should become member of the custom group or not.

In the following picture we see a host being in vCenter Maintenance Mode.

*Figure 10: vCenter – ESXi in Maintenance Mode*

The “Preview” button let us check the configured membership criteria.

*Figure 11:* Host in Maintenance Mode and member of the custom group

Now the ESXi host system is member of the custom group and receives the new policy which disables all alert definitions for this particular object. In vROps one can see the current policy for any selected object in the upper right corner.

*Figure 12:* Host in Maintenance Mode receiving the new vROps Policy

Once the maintenance has been completed, vCenter admin disables the Maintenance Mode in vCenter and after up to 20 minutes the ESXi host is no longer member of our custom group and the previous (could be default) policy is being applied again.

*Figure 13:* Maintenance Mode ended – ESXi host receiving the original vROps Policy

As you probably noticed, one drawback of this method is a possible gap of max. 20 minutes between entering the vCenter Maintenance Mode for a host and vROps completing the re-evaluation of the custom group membership. That fact should be taken into account while designing the maintenance procedure.

In case automation plays a role in your environment and the policy needs to be applied immediately after entering the vCenter Maintenance Mode the corresponding vROps REST API call can be leveraged to programatically retrieve a list of ESXi hosts from vCenter and populate the custom group with the appropriate vROps objects. But this is another story…

*Figure 14:* vROps Custom Group REST API method

In Part 3 I will focus on use cases comprising additional objects types.

Maintenance Mode for vRealize Operations Objects, Part 1

In this and following posts I will show you few different ways of putting vROps objects into maintenance.

Objects in vROps – short intro

The method used to mark an object as being in the maintenances state depends on the actual use case. As usually, the use case itself is being defined by:

Requirements – what does “maintenance” mean from technical perspective, what exactly needs to be achieved?
Constraints – is there any automation in place, which team is involved, what and how many objects are involved?
Assumptions – can we assume that especially the number and kind of objects will stay the same or what is the expected growth?
Risks – what are the risks of using a certain method?

Let us assume our first use case is:

“In case of ESXi host maintenance mode I will to stop collecting any data for this host and disable all alerts.”

As always, before we start any design and implementation, we did a proper assessment and collected further information:

There are only few ESXi hosts in maintenance at the same time
The team doing the maintenance in vCenter can also access and use the vROps UI
Automation could be used but is not mandatory
All implications from stopping metrics and property collection for a given object, like ESXi host are known and accepted

Let us first look at one specific vROps object, in that case a Host System (ESXi host), using the vROps inventory option:

Host System object in vROps Inventory – “Show Detail” option

We see that the object is properly collecting metrics and properties according to both indicators. The details of the selected object can be checked by clicking the “Show Detail” button. This redirects you to the Summary page of the object. The currently collected metrics and properties can be checked by activating the “Show collecting metrics” option:

Collected metrics for the selected object

Activating the maintenance mode – the UI-way

The easiest way to put an object into maintenance mode is to use the “Start Maintenance” button in the Inventory overview:

In the following dialog you can specify how long the object should be put into maintenance:

After starting the Maintenance, you can again check the new status of the object in the Inventory view:

Now, if you use the same “Show collecting metrics” option in the metrics tab of the object you can see that there are no metrics or properties collecting data. The object stopped the data collection entirely:

No collected metrics and/or properties during vROps Maintenance for an object

At this point you need to know that from the monitoring perspective this object is still in the inventory but the is no single data point being collected, stored and calculated in any way. Any calculations relying on data points coming in for that particular object will not provide new data or calculate nor entirely correct data. What “correct” means, depends on the actual metric, dashboard, view etc.

Deactivating the maintenance mode – the UI-way

As easy as we started the maintenance as easy it can be stopped again using the UI:

After clicking on the “End Maintenance” button, vROps will start collecting all data for the object again.

Activating the maintenance mode – the REST-API-way

Starting and ending the Maintenance Mode using the UI is easy and convenient if you have to deal with a small number of objects and there are no other constraints like complying with e.g. change management process which may require automation.

If you need to deal with a large number of objects or if the vROps Maintenance Mode should be part of an automated process, leveraging the vROps API is the best way to implement it.

As always when using the REST API, the first step is to obtain the Access Token. To acquire the token, following POST method needs to be used:

POST /api/auth/token/acquire

Once we have a valid token, we can call the Maintenance Mode related operations. Following REST operation starts the Maintenance Mode for a given object:

REST API method to start the maintenance mode

As you can see, you will need to determine the vROps Object ID of the object(s) you need to put into maintenance before you can call the actual Maintenance Mode calls.

Once you have the ID(s) the method can be used:

Deactivating the maintenance mode – the REST-API-way

To end the maintenance following REST API method has to be used:

Again, you will need the vROps Object ID to call this method.

Part 2 – Outlook

In the upcoming Part 2 of this post, I will describe other methods which may be used in cases when the requirements differ from the use case described in this post.