vRealize Operations, vRO, vROps

Self-Healing with vRealize Operations and vRealize Orchestrator

The vRealize Operations Management Pack for vRealize Orchestrator provides the ability to execute vRO workflows as part of the alerting and remediation process in vROps.

The vRO workflows can be executed manually or automatically.

With this solution it is easy to implement sophisticated self-healing workflows for your vROps managed environment.

In this blog post I will show you step by step how to get from a use case to auto-remediation with vRealize Operations and vRealize Orchestrator Management Pack.

Infrastructure Components

To start with your own self-healing capabilities, you will need:

  • vRealize Orchestrator instance. You can use a stand alone vRealize Orchestrator or the vRO instance deployed as part of the vRealize Automation deployment. For this post I am using the vRA internal vRO 8.1.
  • vRealize Operations, for this post I am using vROps 8.1.1
  • vRealize Operations Management Pack for vRealize Orchestrator installed and configured. I am using the latest version 3.1.1 which can be downloaded from the VMware Solution Exchange for free:

https://marketplace.vmware.com/vsx/solutions/management-pack-for-vrealize-orchestrator

In this post I am not describing the actual installation and initial configuration of the MP. The process is straight forward and described in the official documentation:

https://docs.vmware.com/en/Management-Packs-for-vRealize-Operations-Manager/3.1.1/vrealize-orchestrator/GUID-0096BB63-D347-4777-AAC2-CE0119A53B95.html

Use case

The story begins with a use case. Since I am focusing on the actual generic procedure my use case is very simple and there are for sure already other ways to remediate the issue.

My use case is:

“If a VM (the OS) crashes I will reset that VM in vCenter”

As “seems to be crashed” or the exact description how to determine if a VM really crashed is not object of this post. We assume we have appropriate symptom and alert definitions in place.

Generic Recipe

The procedure from use case to auto-remediation is always the same:

  1. Create a vRO workflow for your use case
  2. Create or modify a vRO package
  3. Discover or re-discover vRO package and included workflows in vROps
  4. Optional – Configure workflow in vROps – in our case it is mandatory
  5. Create or edit vROps recommendation
  6. Add vROps recommendation to an alert definition
  7. Optional – Manual remediation
  8. Optional – Enable automatic remediation

Create Workflow

I am not going to describe the content of the workflow itself or how to code in vRO in this post.

The focus is how to integrate any given workflow in vROps and let it execute manually and automatically as part of the alert remediation.

For our use case the vRO workflow needs at least one input parameter to pass the vCenter VM object reference from vROps to vRO. In the following picture you see a second input string parameter, vrops_alert_id. If this parameter is available, vROps will pass the internal alert ID to vRO. This ID can be used for callbacks to retrieve more information from vROps.

Figure 1: vRO workflow and its input parameters

In this case the inputs are:

  • vm as VC:<Datatype> to be populated with the object which triggered the alert
  • vrops_alert_id as String – to be populated with the actual vROps alert ID for further callbacks

Create or Modify vRO Package

Workflows you would like to use in vROps need to be added to a new or existing vRO package. Keep in mind that all workflows in the package which we will import in the next step will be visible in vROps.

I tend to create a distinct package only for workflows I need in vROps.

In the next picture you see my package with its content.

Figure 2: vRO package content

Since we are not using the package to export the workflows and import it in another vRO instance, you do not have to include the dependencies, like actions used in the workflow.

Discover or Re-discover vRO Package in vROps

To make vROps aware of the available workflows we need to discover the package we created or modified in the previous step.

The procedure in vROps is simple. “Environment” –> “VMware vRealize Orchestrator” –> “vRO Workflows” is the path you need to follow as shown in the following picture.

Figure 3: vRO package discovery process

In the next step you select your vRO instance, in my case “vRA-vRO” and run the “Configure Package Discovery” from the “Actions” menu.

Figure 4: Configure package discovery in vROps

To start the discovery, you just add your package to the list of packages to inspect. In my case it is com.vmware.tkopton.vrops.actions. “Begin Action” starts the process.

Figure 5: Running the package discovery

After few minutes you should see your package and the included workflows in the list of available packages. If it does not show up, reload the page in your browser. If it still does not show up, you will need to check the recent tasks in vROps via the “Administration” for any errors.

The next picture shows my package and the including workflows after the successful completion of the discovery process.

Figure 6: Discovered workflows

Configure Workflow in vROps (may be optional)

As for my use case we want the workflow to be executed on a specific vCenter object, a VM, we need to configure the workflow in vROps properly.

We need to let vROps know on what resource types in alert definitions and on what target resource type in vRO a workflow can be executed. The process is to run “Create/Modify Workflow Action on vCenter Resources” on the specific workflow from the “Actions” menu as depicted in the next figure.

Figure 7: Configuring the workflow

As my alert will be triggered on VM resource type and will be executed on a VM resource in vCenter, we specify “Virtual Machine” in the highlighted parameters. “Operation” is “add”, as we are initially configuring the action.

Figure 8: Action parameters configuration

The VMware online documentation explains the properties:

https://docs.vmware.com/en/Management-Packs-for-vRealize-Operations-Manager/3.1.1/vrealize-orchestrator/GUID-7E4B6D42-2A5C-440B-A8B6-3B31AD9AEBEB.html

Figure 9: Action configuration – properties overview

If the action completed successfully, you should see the available resource type and action target type as shown in the next picture if you re-run the configuration process.

Figure 10: Configured action

Create or Edit vROps Recommendation

vROps recommendations are the (optional) parts of an alert definition responsible for the availability of manual or automated actions.

To have my workflow available for alert definitions, we create a new or edit an existing recommendation. As show in the next two figures I am creating a new recommendation and specify my vRO workflow as the action.

Figure 11: Creating new vROps recommendation – step 1
Figure 12: Creating new vROps recommendation – step 2

Add vROps Recommendation to an Alert Definition

As I am not focusing on symptom and alert definitions itself, I assume we have an appropriate alert definition in place.

Now we need to wire that alert definition and our new recommendation.

This is pretty easy in vRealize Operations 8.1, as shown in the following pictures.

Figure 13: Editing alert definition

We simply edit the alert definition and add the recommendation per drag and drop.

Figure 14: Adding recommendation to an alert definition

An alert definition may have multiple recommendation. In auto-remediation scenarios we should be careful with multiple actions here.

Figure 15: Alert definition with configured recommendation

Optional – Manual Remediation

Now we wait for the alert to be triggered.

Figure 16: Triggered alert

Once we see the active alert, we can open it and start the workflow manually.

Figure 17: Available action in the triggered alert

In the next picture you can see, that the vm parameter has been populated by vROps.

The vrops_alert_id is obviously not being populated when the workflow is being started manually.

Figure 18: Action parameters

After starting the action, we can see the corresponding task in vCenter.

Figure 19: Task executed in vCenter

Optional – Enable automatic remediation

Now, as we see that the workflow is working correctly, we are ready to enable automatic action and let vRealize Operations remediate issues without manual involvement.

To enable automatic actions, we need to modify the settings of the alert definition in the policy applied to the object(s) in scope.

In vROps 8.1 with its new alert definition workflow it is just one click.

Figure 20: Enabling automated action

The next triggered alert will start the action (vRO workflow) automatically and we can see the execution in the “Recent Tasks”.

Figure 21: Recent tasks overview in vROps

This time, vRO also receives the vROps alert ID as depicted in the next figure.

Figure 22: Workflow output in vRO

Conclusion

With vRealize Operations Management Pack for vRealize Orchestrator we have almost unlimited possibilities to extend vROps actions and implement real self-healing operations.

Stay safe.

Thomas – https://twitter.com/ThomasKopton

4 Comments

    1. Thank you, Petar.
      Yes, that’s correct, you will need to modify the policy as I have described: “To enable automatic actions, we need to modify the settings of the alert definition in the policy applied to the object(s) in scope.”

      Thomas

  1. Great article. There is only one question. Let’s say I have several vROPs and one vRO with one worker process. My alert fires and the workflow starts on vRO. How can I determine from which vROPs the alert came in order to get complete information on the alert and its symptoms in the future.

    1. Hey, with this solution you need to query every vROps intance. But stay tuned, my next article will probably give you some more ideas.

      Thomas

Leave a Reply to Petar Penev Cancel reply

Your email address will not be published. Required fields are marked *