Dashboard

Checking SSL/TLS Certificates using Aria Operations – Update

In my article “Checking SSL/TLS Certificate Validity Period using vRealize Operations Application Monitoring Agents” published in 2020, I have described how to check the remaining validity of SSL/TLS certificates using Aria Operations, or to be more specific, using vRealize Operations 8.1 and 8.2 back in the days.

I did not expect this post to be utilized by some many customers to check their SSL/TLS certificates used to secure specifically non-VMware endpoints.

As things might have changed in latest versions of Aria Operations including the VMware Aria Operations SaaS offering, In this blog post I will describe how to check and adjust the configuration if required.

Application Monitoring – Agent Configuration

The first change is, that there is no Application Remote Collector (ARC) Aria Operations. Its functionality is now included in the Aria Operations Cloud Proxy.

A Cloud Proxy instance has to be deployed to the Aria Operations instance regardless the option being used, on-premises or SaaS. The following picture shows the Cloud Proxy in an on-premises Aria Operations instance.

*Figure 1: Aria Operations Cloud Proxy.*

To deploy and configure the Cloud Proxy please follow the official VMware documentation: https://docs.vmware.com/en/VMware-Aria-Operations/8.12/Getting-Started-Operations/GUID-7C52B725-4675-4A58-A0AF-6246AEFA45CD.html

The installation and configuration of the Aria Operations managed telegraf agent did not change significantly, the screenshots from my old post still apply. VMware documentation describes the installation, configuration and uninstallation process: https://docs.vmware.com/en/VMware-Aria-Operations/8.12/Configuring-Operations/GUID-0C121456-370C-467E-874B-38ACC93E3776.html

*Figure 2: Installing Application Monitoring agent.*

Once the agent has been installed and is running, the actual configuration of the agent becomes available.

The agent basically:

discovers supported applications and can be configured to monitor those applications
provide the ability to run remote check, like ICMP or TCP tests
provide the ability to run custom scripts locally

The ability to run scripts and report the integer output as metric back to Aria Operations is exactly what we need to run certificate checks.

The actual script is fairly easy and available, together with the Aria Operations dashboard, via VMware Code:

https://code.vmware.com/samples?id=7464

To let the agent run the script and provide a metric, we configure the agent with few options, the process has changed slightly in newer versions and you will find it under the Applications section.

The script itself expects two parameters, the endpoint to check and the port number.

One agent, like for example a designated Linux Virtual Machine, can run multiple instances of the same script with different options or completely different scripts.

All scripts need to be placed in /opt/vmware and the arcuser (as per default configuration) needs the execute permissions.

The running custom scripts provide a metric per script. The values can be used to populate dashboards or views or serve as metrics for symptoms and alert definitions.

After downloading and importing the Dashboard into Aria Operations, please do not forget to reconfigure the Scoreboard widget. You will need to remove my custom script metrics and add yours, as shown here.

*Figure 6: Scoreboard widget configuration.*

A nice option is, to retain one of the examples as with one click apply the custom settings to all your custom script metrics as shown in the following picture, obviously you will need to change the Box Label. For some reason it does not copy the unit, it has to be specified on every new metric manually.

*Figure 7: Scoreboard widget configuration – applying custom settings to all metrics.*

The dashboard showing is very simple but with the color coding if the widget it is easy to spot endpoints with expiring SSL/TLS certificates and take appropriate actions.

Of course you can adjust the widget settings to reflect your color coding.

Stay safe.

Thomas – https://twitter.com/ThomasKopt

How to detect Windows Blue Screen of Death using VMware Aria Operations

Problem statement

Recently I was asked by a customer what would be the best way to get alerted by VMware Aria Operations when a Windows VM stopped because of a Blue Screen of Death (BSoD) or a Linux machine suddenly quit working due to a Kernel Panic.

Even if it looks like a piece of cake (we have tons of the metrics and properties collected by Aria Operations), it turns out that it is not that simple to recognize such crashes without looking at the console.

So, challenge accepted:-)

In this blog post I am focusing on Windows BSoD and the overall idea was to figure out the metrics which combined are indicating a BSoD occurrence.

NOTE: Windows as well as Linux will restart a crashed OS with default settings and the restart usually is quick enough to remain “undetected” by Symptom Definitions unless you are using Near-Real Time Monitoring in VMware Aria Operations SaaS. A restart can also be initiated by vCenter HA settings in case of missing heartbeats from the OS.

My “Windows BSoD” Approach

I quickly created a Windows Server 2019 VM with this configuration:

And with the help of tools like Testlimit and DiskSpd plus some usual activities on the VM I created a “quick and dirty” baseline using metrics shown in the next picture (please ignore the color coding in the following picture for a moment). You will notice that in the Scoreboard the VMTools status is missing. I was not sure if I should include it or not as “tools not running” does not necessarily mean that the OS crashed, it could be also a crashed service.

*Figure 02: Windows 2019 Server VM baseline.*

Blue Screen of Death Examples

NotMyFault is a perfect tool to crash Windows in various ways. As I wanted to check if different BSoD types have different symptoms I used that tool to force few crashes and collected a set of metrics for comparison.

First crash type.

I started with probably the most known Blue Screen:

*Figure 03: IRQL not less or equal – BSoD.*

The first surprise was that the CPU Demand of the VM immediately increased to almost 100%. To ensure that this is not related to the fact that Windows is collecting data after the crash for some time, I checked this metric after two collections cycles (10 minutes) and it did not go down. Second finding is that the RAM Usage is only slowly decreasing, I assume this is simply due the the fact that memory is a really virtualized resource whereas CPU cycles inside the VM are actual cycles on the ESXi CPU. I also added the Guest Tools Status to the Scoreboard but I would not use it as symptom in an Alert Definition.

*Figure 04: IRQL not less or equal – metrics.*

Second crash type.

*Figure 05: IRQL gt zero at system service – BSoD.*

As you can see in the next picture, the metrics are behaving similarly to the first crash. Of course no disk and network usage at all was expected but to see that the CPU Demand and RAM Usage are following the same pattern is interesting and very promising symptom.

*Figure 06: IRQL gt zero at system service – metrics.*

This time I waited few more collection cycles to see how far the RAM usage will decrease and apparently it will go down to approx. 0.99% after >4 cycles.

Third crash type.

*Figure 08: Unexpected kernel mode trap – BSoD.*

This BSoD type again resulted in the same metrics values.

*Figure 09: *Unexpected kernel mode trap* – metrics.*

RAM on ESXi as metric seems to be dependent on the memory usage of the VM before the crash. I did not fully test it but Aria Operations Metric Correlation feature shows the same pattern for the respective metrics.

*Figure 10: *Memory metrics correlation*.*

Just to be sure that the RAM Usage metric values do not change with different memory configurations of the VM, I did two more tests, the first one with 4GB RAM configured and the second with 17GB to check the metric with an odd RAM config.

*Figure 11: *4GB VM* metrics after crash.*

And here the 17GB RAM config.

*Figure 12: *17GB VM* metrics after crash.*

Constraints, assumptions and conclusions

Please be aware that I did not test every possible scenario, this is what I used:

Windows 2019 Server Datacenter as OS
VM Version 19
VMware ESXi, 7.0.3, 20328353
3 different BSoD types tested
VMTools not used as symptom
OS Uptime not used as symptom as the metric is not available after OS crashed
No Guest metrics used as such metrics will not be available after OS crashed

With the observations made during the crash tests I created 6 new Symptom Definitions and an Alert Definition using these new symptoms and one Condition for the power state of the VM. In the following two pictures you see the symptom and alert definitions.

DO NOT forget to activate your new symptoms and alert definition in the Aria Operations Policy assigned to your VMs!

This is how the symptoms looks like on a crashed Windows Server 2019 VM.

Please be aware that the highlighted low memory usage symptom requires several collection cycles to become active. If you need fast response, remove it from the Alert Definition.

The small dashboard I created is shown in the next picture.

You can download the content from VMware Code.

Update 03.03.2023

One of my fellow colleagues (thank you Brandon) suggested to test the behavior with VMTools not running at all as it will have impact on memory usage metrics. Brandon also suggested to add or replace CPU Demand with CPU Usage as demand will be affected by high CPU usage on the ESXi host. I have added this metric to the Metric Configuration file and uploaded it to VMware Code.

NOTE: CPU, Disk and Network metrics are basically instantly affected by the crash, whereas memory slowly converges toward 0.

As you can see in the following screenshot, you can use CPU Usage instead of CPU Demand as it will also increase to 98-100% after the BSoD.

*Figure 18: *BSoD Dashboar*d with new metrics.*

I would like to mention once again that the CPU metrics are available basically right after the crash and if you use Aria Operations SaaS, which is definitely the recommended way of using Aria Operations, you will get the symptoms triggered roughly after 40-60 seconds.

The memory metrics, as you can see in the next picture, will need several minutes to decrease to a level near 0.

*Figure 19: Slowly converging memory metrics.*

Stay safe.

Thomas – https://twitter.com/ThomasKopto

VMware Explore Follow-up 2 – Aria Operations Dashboard Permissions Management

Another question I was asked during my “Meet the Expert – Creating Custom Dashboards” session which I could not answer due to the limited time was:

“How to manage access permissions to Aria Operations Dashboards in a way that will allow only specific group of content admins to edit only specific group of dashboards?“

Even if there is no explicit feature providing such functionality, there is a way to implement it using Access Control and Dashboard Sharing capabilities of Aria Operations.

My solution

Assumption is that for example following AD users and groups are available, content admins are responsible for creating dashboards and content users will be consuming their dedicated content.

I have imported the AD groups in Aria Operations Access Control and for the sake of simplicity I have assigned them the predefined roles Content Admin and Read Only respectively and granted access to all objects in Aria Operations.

*Figure 02: AD Groups in Operations Access Control*

I have also created two sample dashboards and two dashboard folders for these two dashboards. This is not really required but it makes it easier to find the dashboards if you have a larger number of them with a more complex categorization.

*Figure 03: Aria Operations dashboard folders*

And the last thing to do is to configure dashboard sharing accordingly using in the dashboard management shown in the next picture.

A dashboard can be shared with multiple user groups. In may example I have shared one sample dashboard with one editor and user group and the other sample dashboard with another editor and another user group. This way only dedicated editors (the members of the AD group) have access only to dashboards shared with them, and of course to any other dashboard shared with the built-in group Everyone. The very same way as regular users get access to their respective content.

*Figure 05: Aria Operations dashboard sharing*

Of course this approach requires a proper user group and dashboard sharing concept but such a concept is recommended anyway.

Stay safe.

Thomas – https://twitter.com/ThomasKopton

How vRealize Operations helps size new vSphere Clusters

In ESXi Cluster (non-HCI) Rightsizing using vRealize Operations I have described how to use vRealize Operations and the numbers calculated by the Capacity Engine to estimate the number of ESXi hosts which might be moved to other clusters or decommissioned. The corresponding dashboard is available via VMware Code.

In this post, I describe the opposite scenario.

Problem Statement

The question I will answer is: “How can I use vRealize Operations to help me size new vSphere clusters using completely new ESXi hosts I plan to purchase?“

With its Capacity Engine and the “What-If Analysis” scenarios, vRealize Operations provides powerful features to help with infrastructure and workload planning. In case you are not familiar with “What-If”, the following picture shows the supported scenarios.

*Figure 01: vRealize Operations “What-If Analysis” supported scenarios.*

What we are missing here is a scenario covering local workload (Virtual Machines) migrations from existing vSphere clusters to new planned and not yet existing clusters. Usually, you know what kind of compute hardware you are planning to buy or at least what choices you have, what you do not know is how many of them you need to run any specific workloads.

Solution

NOTE: I am using the demand model in this use case. The allocation model would be similar to implement.

Using vROps and knowing what type of hardware will be used, we have everything we need to estimate the number of hosts required to migrate all workloads from an “old” vSphere Cluster.

These are the ingredients:

“Recommended Total Capacity (Mhz)” as calculated by the vROps Capacity Engine
“Recommended Total Capacity (GB)” as calculated by the vROps Capacity Engine
total CPU resources (in MHz) provided by the new hardware
total RAM resources (in GB) provided by the new hardware

Now we need to do some simple math:

"Recommended Total Capacity (Mhz)" / "total CPU resources (in MHz) provided by the new hardware"

"Recommended Total Capacity (GB)" / "total RAM resources (in GB) provided by the new hardware"

I use two vROps Super Metrics with as simple as possible formula, a number, to represent the potential new resources.

In this example, it is a Cisco Blade system with a certain CPU and RAM configuration.

*Figure 02: Super Metrics representing the new hardware configuration.*

Another three Super Metrics, attached to Cluster Compute Resource as object type, simply calculate the required number of such new hosts, from the CPU and RAM perspective, and identify the higher one as the number of required hosts.

*Figure 03: Super Metrics calculating the number of required new hosts.*

To make it easier to consume I have created a dashboard similar to the rightsizing one.

You can download the Super Metrics, Views, and Dashboard from VMware Code.

Stay safe.

Thomas – https://twitter.com/ThomasKopton

Monitoring vSphere HA and Admission Control Settings using vRealize Operations

vSphere High Availability (vSphere HA) and Admission Control ensure that sufficient resources are reserved for virtual machine recovery when a host fails. Usually, my customers are running their vSphere clusters in either N+1 or N*2 configurations reflected corresponding Admission Control settings

In one of my previous blog posts, I have described how vRealize Operations helps with capacity management for N+1 and N*2 configured clusters.

Capacity Management for n+1 and n*2 Clusters using vRealize Operations

In this post, I will describe how vRealize Operations helps to monitor the vSphere infrastructure to find any deviations from the desired HA and Admission Control state.

The dashboard and all needed components can be, as always, found on code.vmware.com:

https://code.vmware.com/samples?id=7508

Motivation

Even if you are responsible for a very small environment, just few ESXi clusters, do you have a complete, reliable and current overview of the HA and Admission Control settings on every cluster? Do you know any possible deviations from your desired state?

A few simple vROps Super Metrics, Views, and one Dashboard can help you maintain exactly the state of your vSphere Environment that will ensure sufficient resources for virtual machine recovery when a host or multiple hosts fail(s).

How the Dashboard works

The dashboard will help you answer a few simple questions:

Is HA enabled on my ESXi clusters?
What Admission Control Policy is configured?
What is the current amount (in %) of reserved CPU and memory resources on every single cluster?
Does the current amount (in %) of reserved CPU and memory resources configured through Admission Controlequal the desired amount as intended by the selected capacity model for the cluster, N+1 or N*2?

The base indicator to differentiate between different models is a vSphere tag. To make the vROps views work right after importing them, correct tags need to be assigned to the clusters.

These tags are used as filters in the N+1 and N*2 centric views.

For N+1 clusters we need to calculate the desired value for reserved CPU and memory resources and compare that value with the current value calculated by vSphere. To take any ESXi hosts in maintenance I have also added additional information regarding the count of ESXi hosts in maintenance and the count of hosts contributing to the current pool of compute resources.

To make this dashboard work in your environment you need to set the vSphere tags appropriately. Of course, you can use your own tags and adjust the filters in the views accordingly.

Do not forget to enable the imported Super Metrics in your policies.

Stay safe.

Thomas – https://twitter.com/ThomasKopton

Checking SSL/TLS Certificate Validity Period using vRealize Operations Application Monitoring Agents

In my 2019 article “Checking SSL/TLS Certificate Validity Period using vRealize Operations and End Point Operations Agent” on VMware Cloud Management Blog (https://blogs.vmware.com/management/2019/05/checking-ssl-tls-certificate-validity-period-using-vrealize-operations-and-end-point-operations-agent.html) I have described how to check the remaining validity of SSL/TLS certificates.

The method back then was to utilize the End Point Operations Agents.

Since vRealize Operations 7.5 new Application Monitoring capabilities have been introduced including a new Telegraf-based agent.

In this blog post I will describe how to use the new agent to implement an easy solution to continuously check the validity of SSL/TLS certificates. The remaining days until expiration will be displayed as a simple dashboards in vROps.

Application Monitoring – Agent Configuration

After deploying the Application Remote Collector (ARC) vRealize Operations is ready to install agents on monitored virtual machines.

*Figure 1: Installing Application Monitoring agent*

Once the agent has been installed and is running, the actual configuration of the agent becomes available.

The agent is basically doing two jobs. The agent:

discovers supported applications and can be configured to monitor those applications
provide the ability to run remote check, like ICMP or TCP tests
provide the ability to run custom scripts locally

The ability to run scripts and report the integer output as metric back to vROps is exactly what we need to run certificate checks.

The actual script is fairly easy and available, together with the vROps dashboard, via VMware Code:

https://code.vmware.com/samples?id=7464

To let the agent run the script and provide a metric, we configure the agent with few options.

The script itself expects two parameters, the endpoint to check and the port number.

One agent can run multiple instances of the same script with different options or completely different scripts.

All scripts need to be placed in /opt/vmware and the arcuser (as per default configuration) needs the execute permissions.

Dashboard

The running custom scripts provide a metric per script. The values can be used to populate dashboards or views or serve as metrics for symptoms and alert definitions.

The dashboard showing is very simple but with the color coding if the widget it is easy to spot endpoints with expiring SSL/TLS certificates and take appropriate actions.

You will need to adjust the widget settings to include your metrics.

Stay safe.

Thomas – https://twitter.com/ThomasKopton

ESXi Cluster (non-HCI) Rightsizing using vRealize Operations

vRealize Operations with its four main pillars:

Optimize Performance
Optimize Capacity
Troubleshoot
Manage Configuration

provides a perfect solution to manage complex SDDC environments.

The “Optimize Performance” part of vRealize Operations provides a wide range of features like workload optimization to ensure consistent performance in your datacenters or VM rightsizing to reduce bottlenecks and ensure best possible performance of your workloads.

The vROps capability to identify over- and undersized VMs and conduct the required operations to adjust the configuration of VMs is one of the well-known features, accessible directly form the UI.

But what if you would like to rightsize your ESXi Clusters? What information and features is vRealize Operations providing in this area?

What-If Analysis for Clusters

The What-If Analysis feature in the “Optimize Capacity” area is a quick and simple way to check the impact additional workloads or removing of workloads will have in the capacity of an ESXi traditional or HCI cluster.

You can also run infrastructure centric scenarios, like removing or adding hosts from/to clusters.

These are all great features supporting proper capacity and performance management.

But how can you determine if your clusters are configured correctly from the available capacity point of view? What if you have a significant number of clusters? You probably do not want to run the scenarios for every and each cluster over and over again to get updated information.

vRealize Operations is providing all needed information to have a quick and up-to-date insight into your environment allowing you take all necessary actions to adjust the sizing of your ESXi clusters and optimize your SDDC.

Recommended CPU, Memory and Disk Space Metrics

vRealize Operations is constantly calculating recommended values for CPU, Memory and Disk Space based on the configured capacity models, Demand and Allocation if activated. The recommended capacity calculation takes into account vROps Buffers, allocation ratios and Admission Control settings giving you a fairly reliable indication on how to size your clusters.

These metrics can be used to calculate the actual number of ESXi host which could be safely removed from the cluster or how many hosts need to be added to cope with the projected demand.

Cluster Rightsizing Dashboard

The Dashboard and all required components can be downloaded from VMware Code page:

https://code.vmware.com/samples?id=7407#

My simple dashboard will give you detailed insights into the utilization and capacity of your clusters.

It will also provide recommendations regarding the optimal size of the cluster, which will help improve the efficiency of your environment.

This first version of the dashboard is limited to traditional clusters (non-HCI like vSAN clusters).

Even if it shows all clusters (a filter will be added in the next version), please do not shrink vSAN clusters using information provided by this dashboard.

Only CPU and Memory Demand metrics are processed to conduct the rightsizing.

Before removing ESXi host from a cluster I highly recommend putting them into maintenance mode for some period of time and assess performance of the workloads. Additional What-If analysis based on the numbers provided by the dashboard helps get confidence in uncertain situations.

In addition to the metrics provided out-of-the-box we need few Super Metrics to calculate the actual number of hosts to add/remove. It is important to note that the calculation is working properly for uniform clusters. That means same sizing of ESXi host within a cluster, same CPU speed and number of cores, same memory configuration.

The list view used in the dashboard displays all clusters in the selected vSphere Datacenter.

In the last column you will see the number of ESXi host you either should add to the cluster to ensure sufficient capacity or you could potentially remove from the cluster.

Before you start removing hosts from clusters, you can also run a What-If scenario to check the remaining capacity and the capacity projection.

In my example the dashboard is indicating that I could remove one host from the wdcc02 cluster.

If we run the scenario, we see that from the demand perspective the cluster is still providing sufficient capacity to run the current workloads.

Happy rightsizing and stay safe.

Thomas – https://twitter.com/ThomasKopton

vRA Dashboard – Tenant Focused View How-To

As many of you are probably aware, vROps provides an easy and comprehensive way to monitor and manage a vRealize Automation (vRA) environment through utilizing the Native Management Pack (NMP) for vRealize Automation.

The MP gives you a variety of pre-configured dashboards, view, alerts etc. which all work perfectly fine and give you a sufficient view of the environment if your vRA is dealing with only one tenant.

Of course are vROps and the Management Pack capable of collecting and displaying information for multiple tenants, the problem is, that some content aggregates numbers across all tenants. You can see this in the following screenshot:

But what if you have multiple tenants and you would like to see e.g. environment numbers for each and every tenant separately? How many deployments have been done by tenant X and how many Linux VMs have been deployed by tenant Y?

In this blog I will show you the basic techniques you can utilize to create your own content focusing on a per tenant view.

At the end of this blog post you will find a link to VMware Code where you can download sample content.

The foundation of the solution described in this post are Custom Groups.To have all the needed groups in one place, I have created a distinct Group Type:

Since I cannot describe all possible requirements this are my assumptions:

vRA configured with two tenants (vsphere.local, corporation.local in my environment)
requirement for having a Dashboard displaying how many Linux and Windows VMs have been deployed per tenant
requirement for having a Dashboard displaying number of Blueprints per tenant
requirement for having a Dashboard displaying number of Deployments per tenant

Based on these requirements we need following Custom Groups in our vROps:

How to populate those groups dynamically with the corresponding objects?

Actually, this is pretty easy as the Management Pack provides the required Navigation Trees:

Let’s see how to create the Custom Groups.

As an example you see here the Custom Group for Windows VMs deployed by the corporation-Tenant. The most important part is bordered in red, it is the relationship of an given object in a navigation tree.

The “contains” statement has to reflect the actual name of the tenant.

Obviously, for the Linux group only the Properties part looks a bit different and needs to reflect the actual environment:

Now, let’s inspect the group configuration for Deployments and Blueprints:

Custom Group Settings – Deployments per Tenant

For the Blueprints Custom Group we need to use another Navigation Tree:

Custom Group Settings – Blueprints per Tenant

As there is no metric or property reflecting the count of objects belonging to a Custom Group, appropriate Super Metrics need to be created, associated with the corresponding Custom Group Type and enabled in their respective Policy:

Super Metrics enabled in the policy:

Here an example of the actual formula for the first Super Metric:

count(${adaptertype=VMWARE, objecttype=VirtualMachine, attribute=badge|health, depth=1})

After few collection cycles the new Super Metric will show up on our new Custom Groups and the values can be used in the content.

Now we have all ingredients to create our own custom dashboard displaying the various numbers separately for all our vRA Tenants.

This is how my sample dashboard looks like:

Sample Dashboards for vRA – Tenant focused View

What is happening behind the scenes is fairly easy. The Scoreboards are configured as self-provider and are using previously created Custom Groups as source of the objects and the respective Super Metrics to show the correct numbers.

At the bottom of Dashboard some additional information regarding the selected Blueprint is displayed. To achieve this correct wiring of the sources and destinations is required:

A zip file containing all elements can be found here:

https://code.vmware.com/samples?id=5348

Thomas