The method back then was to utilize the EndPoint Operations Agents.
Since vRealize Operations 7.5 new Application Monitoring capabilities have been introduced including a new Telegraf-based agent.
In this blog post I will describe how to use the new agent to implement an easy solution to continuously check the validity of SSL/TLS certificates. The remaining days until expiration will be displayed as a simple dashboards in vROps.
Application Monitoring – Agent Configuration
After deploying the Application Remote Collector (ARC) vRealize Operations is ready to install agents on monitored virtual machines.
Figure 1: Installing Application Monitoring agent
Once the agent has been installed and is running, the actual configuration of the agent becomes available.
The agent is basically doing two jobs. The agent:
discovers supported applications and can be configured to monitor those applications
provide the ability to run remote check, like ICMP or TCP tests
provide the ability to run custom scripts locally
The ability to run scripts and report the integer output as metric back to vROps is exactly what we need to run certificate checks.
The actual script is fairly easy and available, together with the vROps dashboard, via VMware Code:
To let the agent run the script and provide a metric, we configure the agent with few options.
Figure 2: Configure Custom Script
The script itself expects two parameters, the endpoint to check and the port number.
Figure 3: Custom Script options
One agent can run multiple instances of the same script with different options or completely different scripts.
All scripts need to be placed in /opt/vmware and the arcuser (as per default configuration) needs the execute permissions.
Dashboard
The running custom scripts provide a metric per script. The values can be used to populate dashboards or views or serve as metrics for symptoms and alert definitions.
Figure 4: Custom Scripts as metrics
The dashboard showing is very simple but with the color coding if the widget it is easy to spot endpoints with expiring SSL/TLS certificates and take appropriate actions.
Figure 5: SSL/TLS Certificate Validity dashboard
You will need to adjust the widget settings to include your metrics.
vRealize Operations provide a wide range of content OOB. It gives the Ops teams a variety of dashboards, view, alerts etc. to run and manage their environments.
Sooner or later, in most cases rather sooner than later, vROps users will create their own content. It might be completely new dashboards or maybe just adjusted alert definitions.
Whatever content you create in vRealize Operations, you should treat it like every other software development project.
Ideally you have a development, test and production vROps instances. If this footprint is just too big for your environment, you should at least have a single node test/dev for content development and testing before you import that content into your production instance.
Managing content in vROps, exporting dashboards, views, alert definitions etc. and importing the corresponding files into another vROps instance may be very cumbersome and error prone.
This is where vRealize Suite Lifecycle Manager comes into play.
vRSLCM offers all features to make the management of vRealize Operations (and vRA/vRO/vCenter) content an easy task.
In this post I will describe the basics of vROps content management using vRealize Suite Lifecycle Manager and GitLab.
Logical Design
The procedure described in this post is based on the following logical design of the vROps environment including vRSLCM and GitLab.
Figure 1: Logical design
vRSLCM Configuration Overview
In this post I am not going to describe how to configure vRealize Suite Lifecycle Manager, how to add content endpoints is described in detail here:
For the walkthrough presented in this post I have configured following content endpoints in my vRSLCM.
Figure 2: vRSLCM content endpoints
I have a vRSLCM-deployed vROps instance, serving as my Dev/Test and a vROps-P1 which is the production instance.
Important configuration detail here is, that the production vROps, vROps-P1, is set to accept source controlled content only. If you have only one vROps checking in content into vRSLCM repository you probably won’t set that option. If you do, you will need a source control endpoint, like GitLab. I have set that option to showcase the usage of GitLab and how changes to the content source in GitLab itself can be handled.
Walkthrough – Overview
The steps in my walkthrough are:
Have a content in Dev/Test vROps you would like to deploy to Prod vROps.
Capture content from Dev/Test into vRSLCM repo and GitLab (including Git merge).
Try to deploy content to Prod vROps from vRSLCM repo directly.
Capture and deploy content from GitLab to Prod vROps.
Modify content in GitLab.
Re-capture from GitLab and deploy to all vROps endpoints.
Step 1 – vROps Content
For the demo I am using a simple dashboard as shown in the following picture.
Figure 3: vROps dashboard – first version
The goal is to deploy this dashboard into the production vROps environment.
Step 2 – Capture Content
We start the capture process using the “Add Content” feature.
Figure 4: “Add Content” vRSLCM feature
Obviously we select vRealize Operations as the content endpoint type.
Figure 5: Content capture – endpoint type
As we want to capture the content into the internal vRSLCM repository and into the source controlled repo – GitLab in one step, we are selecting both respective options.
Figure 6: Capture and check in
We select the source endpoint and the dashboard itself. I have also set the option to import all dependencies. In this case there are no dependencies like for e.g. views used as widgets, in other scenarios, vRSLCM will resolve the dependencies and import all needed content parts. I have also selected the “mark content as production ready” option to allow that content to be deployed to my production vROps.
Figure 7: Content capture settings
As we are checking in the content into GitLab, we need to specify the endpoint, repo and branch.
Figure 8: Checking in content into GitLab
After few seconds the content pipelines complete.
Figure 9: Content pipelines
Now we see the merge request in GitLab and after merging the request, the dashboard is available in GitLab repo and treated as any other code.
Figure 10: GitLab merge requestFigure 11: Dashboard as code in GitLab
Step 3 – Deploy Content – First Attempt
At this point we have our dashboard as content in vRSLCM repo and in GitLab.
Figure 12: Content in vRSLCM repo
If we now open the tkopton-Dashjboard-01 we will see the details of the first version and the option to deploy the content to another vROps endpoints.
Figure 13: Content details and deployment – first attempt
In the next picture we can see that our vROps-P1 is not listed as an available endpoint. This is because I have configured that endpoint to accept source controlled content only. This version is not source controlled, it has been captured from Dev/Test vROps into vRSLCM repo, not from the GitLab.
That means, we need to capture the content from GitLab first to be able to deploy it into the production vROps.
Step 4 – Capture from GitLab and Deploy Content – Second Attempt
We start another capture and deploy (in one step) process and this time we select our GitLab as the source.
Figure 14: Capture and deploy content from GitLabFigure 15: Capture and deploy in one step
We use the same settings as during the first attempt, the capture endpoint is different.
Figure 16: GitLab as capture endpoint
As we are doing capture and deploy in one step, we need to specify the deployment target and options.
Figure 17: Deployment target settings
And now we can see in the Figure 17 that our production vROps-P1 is available in the list of destination endpoints.
Now the same dashboard is available in the production vROps and in vRSLCM repo we see the second version of the dashboard wich is source controlled.
Figure 18: Source controlled content
Step 5 – Edit Content in GitLab
Let us change the name of one of our widgets. And let us do it in GitLab instead of editing the dashboard in vROps. The dashboard is just another source code from the content repository perspective.
Figure 19: Editing the content source in GitLab
Step 6 – Re-Capture and Re-Deploy Content
After re-capturing the content from GitLab following the same procedure as in step 4 but this time without deploying the content, we see another version in the vRSLCM repo.
Figure 20: GitLab updated version of the dashboard in vRSLCM
After deploying the updated content to our vROps endpoints, we see the dashboard having a new caption for the first widget.
Figure 21: Updated dashboard in vROps
Conclusion
With vRealize Suite Lifecycle Manager and GitLab you have a perfect foundation to create your own CD pipeline for vRealize Operations content.
In the next part I will describe how to extend the pipelines with custom workflows provided by e.g. vRealize Orchestrator.
The vRealize Operations Management Pack for vRealize Orchestratorprovides the ability to execute vRO workflows as part of the alerting and remediation process in vROps.
The vRO workflows can be executed manually or automatically.
With this solution it is easy to implement sophisticated self-healing workflows for your vROps managed environment.
In this blog post I will show you step by step how to get from a use case to auto-remediation with vRealize Operations and vRealize Orchestrator Management Pack.
Infrastructure Components
To start with your own self-healing capabilities, you will need:
vRealize Orchestrator instance. You can use a stand alone vRealize Orchestrator or the vRO instance deployed as part of the vRealize Automation deployment. For this post I am using the vRA internal vRO 8.1.
vRealize Operations, for this post I am using vROps 8.1.1
vRealize Operations Management Pack for vRealize Orchestrator installed and configured. I am using the latest version 3.1.1 which can be downloaded from the VMware Solution Exchange for free:
In this post I am not describing the actual installation and initial configuration of the MP. The process is straight forward and described in the official documentation:
The story begins with a use case. Since I am focusing on the actual generic procedure my use case is very simple and there are for sure already other ways to remediate the issue.
My use case is:
“If a VM (the OS) crashes I will reset that VM in vCenter”
As “seems to be crashed” or the exact description how to determine if a VM really crashed is not object of this post. We assume we have appropriate symptom and alert definitions in place.
Generic Recipe
The procedure from use case to auto-remediation is always the same:
Create a vRO workflow for your use case
Create or modify a vRO package
Discover or re-discover vRO package and included workflows in vROps
Optional – Configure workflow in vROps – in our case it is mandatory
Create or edit vROps recommendation
Add vROps recommendation to an alert definition
Optional – Manual remediation
Optional – Enable automatic remediation
Create Workflow
I am not going to describe the content of the workflow itself or how to code in vRO in this post.
The focus is how to integrate any given workflow in vROps and let it execute manually and automatically as part of the alert remediation.
For our use case the vRO workflow needs at least one input parameter to pass the vCenter VM object reference from vROps to vRO. In the following picture you see a second input string parameter, vrops_alert_id. If this parameter is available, vROps will pass the internal alert ID to vRO. This ID can be used for callbacks to retrieve more information from vROps.
Figure 1: vRO workflow and its input parameters
In this case the inputs are:
vm as VC:<Datatype> to be populated with the object which triggered the alert
vrops_alert_id as String – to be populated with the actual vROps alert ID for further callbacks
Create or Modify vRO Package
Workflows you would like to use in vROps need to be added to a new or existing vRO package. Keep in mind that all workflows in the package which we will import in the next step will be visible in vROps.
I tend to create a distinct package only for workflows I need in vROps.
In the next picture you see my package with its content.
Figure 2: vRO package content
Since we are not using the package to export the workflows and import it in another vRO instance, you do not have to include the dependencies, like actions used in the workflow.
Discover or Re-discover vRO Package in vROps
To make vROps aware of the available workflows we need to discover the package we created or modified in the previous step.
The procedure in vROps is simple. “Environment” –> “VMware vRealize Orchestrator” –> “vRO Workflows” is the path you need to follow as shown in the following picture.
Figure 3: vRO package discovery process
In the next step you select your vRO instance, in my case “vRA-vRO” and run the “Configure Package Discovery” from the “Actions” menu.
Figure 4: Configure package discovery in vROps
To start the discovery, you just add your package to the list of packages to inspect. In my case it is com.vmware.tkopton.vrops.actions. “Begin Action” starts the process.
Figure 5: Running the package discovery
After few minutes you should see your package and the included workflows in the list of available packages. If it does not show up, reload the page in your browser. If it still does not show up, you will need to check the recent tasks in vROps via the “Administration” for any errors.
The next picture shows my package and the including workflows after the successful completion of the discovery process.
Figure 6: Discovered workflows
Configure Workflow in vROps (may be optional)
As for my use case we want the workflow to be executed on a specific vCenter object, a VM, we need to configure the workflow in vROps properly.
We need to let vROps know on what resource types in alert definitions and on what target resource type in vRO a workflow can be executed. The process is to run “Create/Modify Workflow Action on vCenter Resources” on the specific workflow from the “Actions” menu as depicted in the next figure.
Figure 7: Configuring the workflow
As my alert will be triggered on VM resource type and will be executed on a VM resource in vCenter, we specify “Virtual Machine” in the highlighted parameters. “Operation” is “add”, as we are initially configuring the action.
Figure 8: Action parameters configuration
The VMware online documentation explains the properties:
If the action completed successfully, you should see the available resource type and action target type as shown in the next picture if you re-run the configuration process.
Figure 10: Configured action
Create or Edit vROps Recommendation
vROps recommendations are the (optional) parts of an alert definition responsible for the availability of manual or automated actions.
To have my workflow available for alert definitions, we create a new or edit an existing recommendation. As show in the next two figures I am creating a new recommendation and specify my vRO workflow as the action.
Figure 11: Creating new vROps recommendation – step 1Figure 12: Creating new vROps recommendation – step 2
Add vROps Recommendation to an Alert Definition
As I am not focusing on symptom and alert definitions itself, I assume we have an appropriate alert definition in place.
Now we need to wire that alert definition and our new recommendation.
This is pretty easy in vRealize Operations 8.1, as shown in the following pictures.
Figure 13: Editing alert definition
We simply edit the alert definition and add the recommendation per drag and drop.
Figure 14: Adding recommendation to an alert definition
An alert definition may have multiple recommendation. In auto-remediation scenarios we should be careful with multiple actions here.
Figure 15: Alert definition with configured recommendation
Optional – Manual Remediation
Now we wait for the alert to be triggered.
Figure 16: Triggered alert
Once we see the active alert, we can open it and start the workflow manually.
Figure 17: Available action in the triggered alert
In the next picture you can see, that the vm parameter has been populated by vROps.
The vrops_alert_id is obviously not being populated when the workflow is being started manually.
Figure 18: Action parameters
After starting the action, we can see the corresponding task in vCenter.
Figure 19: Task executed in vCenter
Optional – Enable automatic remediation
Now, as we see that the workflow is working correctly, we are ready to enable automatic action and let vRealize Operations remediate issues without manual involvement.
To enable automatic actions, we need to modify the settings of the alert definition in the policy applied to the object(s) in scope.
In vROps 8.1 with its new alert definition workflow it is just one click.
Figure 20: Enabling automated action
The next triggered alert will start the action (vRO workflow) automatically and we can see the execution in the “Recent Tasks”.
Figure 21: Recent tasks overview in vROps
This time, vRO also receives the vROps alert ID as depicted in the next figure.
Figure 22: Workflow output in vRO
Conclusion
With vRealize Operations Management Pack for vRealize Orchestrator we have almost unlimited possibilities to extend vROps actions and implement real self-healing operations.
Without any doubt configuring vRealize Operations to send log messages to a vRealize Log Insightinstance is the best way to collect, parse and display structured and structured log information.
In this post I will explain the major differences between CFAPI and Syslog as the protocol used to forward log messages to a log server like vRealize Log Insight.
The configuration of the log forwarding in vRealize Operations is straight forward. Under “Administration” –> “Management” –> “Log Forwarding” you will find all options to quickly configure vRLI as target for the selected log files.
The following figure shows how to configure vRealize Operations to send all log messages to vRealize Log Insight using the CFAPI protocol via HTTP.
Figure 1: Log Forwarding configuration
The CFAPI protocol, over HTTP or HTTPS, used by the vRealize Log Insight agent provides additional information used by the vROps Content Pack. The extracted information flows into the various dashboards and alert definitions delivered through the Content Pack. Following picture shows one of the available dashboards populated with available data when using CFAPI and vRLI.
Figure 02: vROps Content Pack
In case you (for whatever strange reason) cannot use CFAPI, you can configure vROps to use Syslog. It is as simple as selecting Syslog as the protocol option in the configuration page shown in the following picture.
Figure 03: Syslog as configured protocol
The drawback of using Syslog here is that the additional information parsed by the agent and used by the content pack will no longer be available and you will need to create your own extracted fields in vRLI to parse data from the log messages.
In the next both pictures you can see the empty dashboards and log messages without any vROps specific fields in the interactive analytics .
Figure 04: Empty dashboards when using SyslogFigure 05: Missing vROps specific fields when using Syslog
It is important to know that vROps is using Syslog over TCPwhen configured via UI as shown in figure 03.
But what if you are forced to use Syslog over UDP?
There is no such option in the UI but since vROps is using the regular vRLI agent, there has to be a way to configure it to use UDP instead of TCP.
The vRLI config file explains how to set the according option:
Figure 06: liagent.ini config file
You can just replace
proto = syslog
with
proto = syslog_udp
restart the agent
service liagentd restart
and your vROps nodes starts to forward log messages to your log server using UDP.
I have setup a fake log server listening on 514 UDP using netcat:
Figure 07: Syslog over UDP in NC
If you configure the vRLI agent in vROps directly via the config file, please keep in mind:
that you are using a function which is not officially supported by VMware
you will need to make such manual changes on every node
you will need to monitor any changes to that file which can be triggered via the UI or vROps updates
provides a perfect solution to manage complex SDDC environments.
The “Optimize Performance” part of vRealize Operations provides a wide range of features like workload optimization to ensure consistent performance in your datacenters or VM rightsizing to reduce bottlenecks and ensure best possible performance of your workloads.
The vROps capability to identify over- and undersized VMs and conduct the required operations to adjust the configuration of VMs is one of the well-known features, accessible directly form the UI.
Figure 1: VM Rightsizing
But what if you would like to rightsize your ESXi Clusters? What information and features is vRealize Operations providing in this area?
What-If Analysis for Clusters
The What-If Analysis feature in the “Optimize Capacity” area is a quick and simple way to check the impact additional workloads or removing of workloads will have in the capacity of an ESXi traditional or HCI cluster.
You can also run infrastructure centric scenarios, like removing or adding hosts from/to clusters.
Figure 2: What-If Scenarios
These are all great features supporting proper capacity and performance management.
But how can you determine if your clusters are configured correctly from the available capacity point of view? What if you have a significant number of clusters? You probably do not want to run the scenarios for every and each cluster over and over again to get updated information.
vRealize Operations is providing all needed information to have a quick and up-to-date insight into your environment allowing you take all necessary actions to adjust the sizing of your ESXi clusters and optimize your SDDC.
Recommended CPU, Memory and Disk Space Metrics
vRealize Operations is constantly calculating recommended values for CPU, Memory and Disk Space based on the configured capacity models, Demand and Allocation if activated. The recommended capacity calculation takes into account vROps Buffers, allocation ratios and Admission Control settings giving you a fairly reliable indication on how to size your clusters.
Figure 3: Recommended Size Metrics
These metrics can be used to calculate the actual number of ESXi host which could be safely removed from the cluster or how many hosts need to be added to cope with the projected demand.
Cluster Rightsizing Dashboard
The Dashboard and all required components can be downloaded from VMware Code page:
My simple dashboard will give you detailed insights into the utilization and capacity of your clusters.
It will also provide recommendations regarding the optimal size of the cluster, which will help improve the efficiency of your environment.
This first version of the dashboard is limited to traditional clusters (non-HCI like vSAN clusters).
Even if it shows all clusters (a filter will be added in the next version), please do not shrink vSAN clusters using information provided by this dashboard.
Only CPU and Memory Demand metrics are processed to conduct the rightsizing.
Before removing ESXi host from a cluster I highly recommend putting them into maintenancemode for some period of time and assess performance of the workloads. Additional What-If analysis based on the numbers provided by the dashboard helps get confidence in uncertain situations.
In addition to the metrics provided out-of-the-box we need few Super Metrics to calculate the actual number of hosts to add/remove. It is important to note that the calculation is working properly for uniform clusters. That means same sizing of ESXi host within a cluster, same CPU speed and number of cores, same memory configuration.
Figure 4: Super Metrics
The list view used in the dashboard displays all clusters in the selected vSphere Datacenter.
In the last column you will see the number of ESXi host you either should add to the cluster to ensure sufficient capacity or you could potentially remove from the cluster.
Figure 5: Simple Cluster-Rightsizing Dashboard
Before you start removing hosts from clusters, you can also run a What-If scenario to check the remaining capacity and the capacity projection.
In my example the dashboard is indicating that I could remove one host from the wdcc02 cluster.
Figure 6: What-If scenario settings
If we run the scenario, we see that from the demand perspective the cluster is still providing sufficient capacity to run the current workloads.
Last year I published a blog post focusing on the JSON parser available in vRealize Log Insight (https://thomas-kopton.de/vblog/?p=144) and I believe this could have been a beginning of a series – “vRLI Parsers “.
In this post I will show how to configure and utilize the CSV parser using a very prominent and media-effective use case:
COVID-19
aka Coronavirus!
The needed data is publicly available, e.g. here:
https://github.com/CSSEGISandData/COVID-19 The actual data sources are listed on the GitHub project page and the data itself is already in the format we need to feed it into vRLI – CSV files.
Data Review
The time series data files provided by CSSE are in the following format:
Figure 1: CSSE CSV file format
Or shown as raw data:
Figure 2: CSSE CSV file format – in an editor
vRealize Log Insight processes every single CSV row as one message and if we do not specify our own timestamp as part of the message, vRLI will use the ingestion time as the timestamp for the message. This is clearly nothing we would like to have in order to have some meaningful diagrams showing us e.g. how the virus spreads in various countries. We will change the timestamp accordingly.
We also see that the original data contains “,” signs as part of province or state names, this is not compatible with the vRLI CSV parser as it would change the number and content of the fields. I have used Numbers on my Mac to “clean up” the file and replace the real “,” field separation signs with “;”.
To ingest the data, we will use the vRLI agent and an appropriate configuration of the agent.
vRLI Agent Config
My setup in for this use case is:
vRLI 8.0.0-14870409
vRLI agent 8.0.0.14743436
vRLI agent installed on a Debian OS
Agent successfully connected to vRLI
The configuration of the agent to parse CSV files is general pretty simple.
Let us take the configuration to pieces and analyze what the single parts are doing. The first four lines, 1 – 4, contains the main section telling vRLI where to look for specified files and what parser to use to process the messages stored in those files. The parser is not specified directly as one of the OOB parsers but as a parser section.
Figure 3: Parser configuration – part 1
Lines 6 – 11 contains the actual parser
configuration. In this section we specify the base parser, means one of
the parsers available in vRLI. In our case csv.
Line 10 describes how to name the fields
extracted from every row in a CSV file, separated by “;” in my case. The specification
of fields has to match exactly the number of fields in the file we want
to parse, and every single line of the file needs to have exactly the same
number of fields. The fields itself might be empty. In this case we expect lines
in our csv file containing exactly five values separated by “;”.
We assume the first value is a timestamp, the second one will be a number we will use later on to filter for messages in vRLI in case we ingest the same set of data several times, like every day. Location and country are pretty clear and “confnumber” is the actual number of confirmed cases as in the repository by Johns Hopkins CSSE. Line 11 specifies a special decoder for a field extracted in line 10 – here we will use a special decoder to decode or parse the timestamp.
Figure 4: Parser configuration – part 2
Lines 13 – 14 specifies our decoder for the timestamp field. A decoder is just another term for parser. As for every parser we specify the base parser, line 14. Line 17 tells the timestamp parser how to process the value in the “timestamp” field. In this case we expect the value to be an UNIX epoch timestamp.
Figure 5: Parser configuration – part 3
With this configuration we expect the data in a csv file to have the following format (example):
Figure 6: CSV format for the vRLI parser
I am using the timestamp instead of the date
as in the original file because:
The format used by Johns Hopkins CSSE is not optimal for the vRLI timestamp parser
I want to see all data in a short time frame in vRLI. The goal is to have a fancy diagram to compare the numbers across different countries.
We will replace the original timestamps, which are day per data point, with 10ms steps per data point.
Format Adjustment
As we can see, the original csv file needs
to be “slightly” adjusted to be correctly parsed by the parser we configured.
This is pretty easy to achieve with few lines. PowerShell, Bash or in my case vRO:
Figure 7: vRO Workflow to adjust the csv file
This very simple workflow can be found on VMware Code soon. I will update the link shortly.
Data Import
Now we are ready to let vRealize Log Insight do the job. Just put the new csv file into the specified folder on the box where you have configured the vRLI agent, wait few seconds, every day in the original file is a second in the parsed file, and analyze the numbers.
Figure 7: vRLI analyzing Coronavirus data – 1
Let’s see how we are doing here in Germany compared to Italy and US:
Recently I was asked if it is possible to merge two, or generally speaking n different vRealize Operations Dashboards into one Dashboard.
To answer that question and provide a procedure how to do it we need to inspect the internal structure of a vROps Dashboard.
Anatomy of a vRealize Operations Dashboard
As everyone knows vROps content, in this case dashboards, can be exported and imported. Exported dashboard(s) are saved as zip files and after decompressing the zip file the actual dashboard is just a JSON file.
Figure 1: Exporting and decompressing vROps Dashboards
Let’s see how the structure of an empty dashboard looks like. The most important part is the line 26 as displayed in the following picture. Dashboards contain widgets and we see in the line 26 an array of widgets, still empty here as this dashboard does not contain any widget.
Figure 2: Empty dashboard JSON
Now let’s add some simple and unconfigured widgets without any interactions. This simple dashboard contains two widgets placed into two rows and one column.
Figure 3: Simple dashboard with 2×1 widgets
And this is the corresponding JSON file. I have collapsed some “unimportant” sections for better visibility.
Figure 4: Dashboard JSON including two widgets
Since I did not configure the widgets, the “config” sections are empty. Interesting at this point is the “gridsterCoords” section. This section describes the size and the position of a widget on the dashboard. Here another example with some more widgets.
Figure 5: Dashboard with 2×3 widgets
This widgets are described by the following JSON (snippet):
Figure 6: Dashboard JSON including six widgets
With this information we can describe the coordinates of widgets and place them on a dashboard. A bit unusual is the usage of the “h” and “w” attributes for widgets having their default sizes, maybe someone designed the widgets using a flipped monitor;-)
Figure 7: Widgets size and coordinates
This unusual usage of width and height seems to be “fixed” and appears as expected if a given widget has been resized during the dashboard creation. But that does not prevent us from merging dashboards.
Merging vRealize Operations Dashboards
Merging dashboards is as easy as copy&paste the appropriate widget sections into the widget array in the target JSON file and adjusting the x and y coordinates accordingly.
In my example we are merging the previous 1×2 dashboard into the 2×3 dashboard and place the 1×2 widgets below the 2×3 widgets. In this example the old widget “names” are still the same, 1-1 and 1-2 to easily identify them on the dashboard.
Figure 8: Merged dashboard
Widget Interaction
Out of curiosity, let’s inspect an interaction.
Figure 9: Widgets interaction
The JSON representation of an interaction, multiple interactions are stored in an array, is pretty simple.
Figure 10: JSON representation of widgets interaction
With this information and a little bit editing of JSON files it should be possible to merge vRealize Operations Dashboards.
Applications Monitoring is an exciting functionality in vRealize Operations. To start monitoring one of the twenty services currently supported by vROps, the vRealize Operations Application Remote Collector (ARC) needs to be deployed and configured in vROps.
The Application Remote Collector is provided as OVA and the deployment process runs similarly to any other VMware OVA deployment.
The most important part during the deployment of the ARC is the API Admin user’s password which is needed for the configuration of this particular ARC in vROps.
Figure 1: ARC Admin user password
Adding the Application Remote Collector (ARC) to vROps
From the vROps “Home” screen you start the configuration of the freshly deployed ARC.
Figure 2: vROps Monitor Applications
Click on “Configure Application Monitoring” to start the process.
Figure 3: Configuring Applications Monitoring
In the following step a freshly deployed ARC can be added and configured. There is also an option to download the OVA just by clicking on the green arrow button.
Figure 4: Add and configure ARC
The next dialog expects basically two values, the FQDN or IP of the ARC and the Admin User’s password specified during the OVA deployment.
Figure 5: ARC configuration values
In the next step we need to specify the vCenters which will be managed by this instance of the Application Remote Collector. One ARC can manage multiple vCenter instances.
Figure 6: Managed vCenter(s)
We can test the connection and after accepting the certificate we can click on “Finish” and complete the ARC configuration.
Figure 7: Finishing the ARC configuration
After few minutes, the status of the ARC should show “Healthy” and report back the correct version.
Figure 8: “Healthy” ARC
Managing Agents
Once the ARC has been configured in vROps we can install agents on VMs running in the vCenter(s) managed by this particular ARC.
Figure 9: Managing ARC agents
One click on “Manage Agents” brings uns to the vROps Inventory – “Manage Agents” tab. Here we can select one or multiple VM objects and start the installation of the agent.
Figure 10: Installing agents
We need to enter credentials. For the user account prerequisites please see:
Since I am installing the first agent on a Linux machine, and this is just a test lab, I am using the root account, which should be avoided in production environments.
Figure 12: Installing agents- VM credentials – Linux VM
After clicking on “Install Agent” and waiting several minutes, the status of the agent should be like in the following figure.
Figure 13: Successfully installed agent
As you can see in the Figure 13, the agent discovered Apache HTTPD service running on the VM.
NOTE: This has nothing to do with the vROps Service Discovery functionality.
At this stage there is still no vROps object related to the discovered Apache instance.
Figure 14: No Apache instance
This is because we first need to configure the agent to properly monitor any discovered (supported) service.
My Apache instance has been configured to serve the status page, this does not have to be the same in every Apache installation, so be sure that your Apache is configured accordingly.
Figure 16: Apache HTTPD configuration values
After clicking on “Save” and “Close” and waiting few minutes, we see a successful configuration.
Figure 16: Monitoring Apache HTTPD
After few collection cycles we see the corresponding vROps object and the collected metrics and properties.
Application Monitoring integrated into vRealize Operations opens up new perspectives and contributes to the holistic view into IT environments provided by vROps.
Part 1 and Part 2 of the “Maintenance Mode in vRealize Operations” blog series leveraged features and functionalities of vROps and vSphere which are directly related to ESXi maintenance to fulfil the requirements.
What if you cannot use “Enter Maintenance Mode” in vCenter simply because there is no such option for e.g. Virtual Machines, Datastores or Clusters but your use case requires some kind of maintenance mode in vROps?
In this part we focus on this kind of scenario. In case you do not have to deal with such scenarios, no worries, your time won’t be wasted, you are going to learn how to use vCenter Tags and Custom Attributes in vROps.
The use case for this blog post is:
“In case of VM/Datastore/Cluster maintenance (whatever “maintenance” might be) I do not want to receive any alerts for the affected vROps objects.”
As always during assessment we collected following additional
information:
There might be severalobjects in maintenance at the same time
The team doing the maintenance has read-only access to vROps UI
The team doing the maintenance is able to edit vSphere Tags or Custom Attributes for the objects in maintenance
Automation could be used but is not mandatory
Metrics and properties need to be collected during the maintenance.
Obviously, we cannot use the OOB vROps Maintenance feature described in Part 1 as this would violate the last requirement and the vCenter team would need additional permissions in vROps. The vROps property “Maintenance State” we already used in Part 2 is not an option here, as this property is available only for Host System objects.
We need another property to designate objects in maintenance. This is where vSphere Tags and/or Custom Attributes come in handy.
The ingredients for one possible solution are:
vSphere Tags and Custom Attributes – Interface for the vCenter team to start the host maintenance
vROps Policy – place where we modify the behavior of vROps with regard to distinct objects
vROps Custom Group – place where we group objects to apply a vROps policy to them
In Part 2 I have described (simplified) the model of Objects, Custom Groups and Policies in vROps.
Step 1 – Create a vROps
Policy which implements the requirements.
In our case the requirement is to disable all alerts
for Virtual Machines / Datastores / Clusters etc. during maintenance hence we
need a vROps Policy in which we are going to disable the corresponding alert
definitions.
A new policy is being created based on an existing one, so only few changes will be needed to tweak the default behavior of vROps. In the following picture we see alert definitions for Virtual Machines as an example. Previous parts describe the procedure of creating and modifying the vROps policy.
Figure 1: Disabling vROps Alert Definitions in the new vROps Policy
Step 2 – Create vROps Custom Group which will contain objects being in maintenance.
Again, “maintenance” describes here any work being
done on objects which requires certain alert definitions being disabled temporarily.
In Part 2 I have already described how to create a vROps Custom Group. I will skip the common details here and focus on the membership criteria which is different in this scenario.
There are also differences in vROps when it comes to the properties reflecting both concepts.
vCenter Custom Attributes
Custom
Attributes are easy to use in vROps but the handling in vCenter might be
cumbersome.
Let’s assume we want to conduct some kind of maintenance on certain VMs and a Datastore. In vCenter we assign an appropriate attribute to the objects we want to set into maintenance. Type “Global” allows us to use the same attribute on different object types.
Figure 2: vCenter Custom Attributes
After the next vROps collection cycle has been completed, this custom attribute is available as property.
Figure 3: vCenter Custom Attribute in as vROps Property
And this is where the confusion begins, vCenter Custom Attributes are vROps Custom Tags properties.
These properties are easy to use in vROps Custom Groups membership criteria definitions. Extra tip, if you cannot find the property in the list presented by vROps as “common denominator”, please use the object picker to select an object which is having the required property (or metric).
Figure 4: Find newly created properties – extra tip
My sample group containing both, VMs and Datastores has the following membership criteria configuration.C
Figure 5: Custom Group membership criteria – Custom Attributes
And automatically contains my two objects.
Figure 6: Group members
The members of the group now receive the new policy with disabled alert definitions.
Figure 7: vROps Policy assigned through Custom Group membership
To remove
objects from the new custom group and let them receive the old policy, we
simply set the vCenter attribute to no.
As already mentioned in Part 2 of this blog series, it may take up to 20 minutes until the membership has be re-evaluated and the objects have been removed from the group.
vSphere Tags
In contrast to vCenter Custom Attributes vSphere Tags are easy to use in vCenter, assigning tags to multiple objects can be done in the UI with few clicks.
This is a simple tag to categorize objects in maintenance.
Figure 8: vSphere Tag exampleFigure 9: Assigning vSphere Tags in vCenter
Again, the
next vROps collection cycle needs to complete to collect the new value.
The corresponding property is vROps differs from the property reflecting vCenter Custom Attributes. There is only one property called “vSphere Tag” and the value of this property contains all tags assigned to the particular object, following the format:
[<CategoryX-TagX>,<CategoryY-TagY>,…]
Figure 10: vSphere Tags collected in vROps as one property
That means, that we need to use the “contains” operator in the membership criteria instead of “is” as for custom attributes. That also means, that you need to be carful with naming your tags to avoid overlaps which may result in unwanted behavior of you group membership criteria.
Figure 11: Custom Group membership criteria – vSphere Tags
As expected, the group contains three objects which got the maintenance vSphere Tag assigned.
Figure 12: vROps Custom Group members
As with
custom attributes, remove the tag after the maintenance has been completed,
wait few minutes and the objects leave the custom group automatically.
This is the last part of the “Maintenance Mode for vRealize Operations Objects” blog series.
I wish you and your loved ones a a healthy start into the New Year 2020.
The vRealize Operations REST API allows using RegEx expressions in various GET methods.
Sometimes it is not clear how to use the expressions. Here a very simple example of using RegEx to retrieve vROps Virtual Machine objects, which have VM names starting with certain strings.
