{"id":1205,"date":"2023-03-02T16:24:47","date_gmt":"2023-03-02T15:24:47","guid":{"rendered":"https:\/\/thomas-kopton.de\/vblog\/?p=1205"},"modified":"2023-04-15T19:56:40","modified_gmt":"2023-04-15T18:56:40","slug":"how-to-detect-windows-blue-screen-of-death-using-aria-operations","status":"publish","type":"post","link":"https:\/\/thomas-kopton.de\/vblog\/?p=1205","title":{"rendered":"How to detect Windows Blue Screen of Death using VMware Aria Operations"},"content":{"rendered":"\n

Problem statement<\/p>\n\n\n\n

Recently I was asked by a customer what would be the best way to get alerted by VMware Aria Operations<\/mark><\/strong> when a Windows VM stopped because of a Blue Screen of Death (BSoD<\/mark>) or a Linux machine suddenly quit working due to a Kernel Panic.<\/p>\n\n\n\n

Even if it looks like a piece of cake (we have tons of the metrics and properties collected by Aria Operations), it turns out that it is not that simple to recognize such crashes without looking at the console.<\/p>\n\n\n\n

So, challenge accepted:-)<\/mark><\/p>\n\n\n\n

In this blog post I am focusing on Windows BSoD and the overall idea was to figure out the metrics which combined are indicating a BSoD occurrence.<\/p>\n\n\n\n

NOTE<\/strong>: <\/mark>Windows as well as Linux will restart a crashed OS with default settings and the restart usually is quick enough to remain “undetected” by Symptom Definitions unless you are using Near-Real Time Monitoring<\/mark><\/strong> in VMware<\/strong><\/mark><\/strong> Aria Operations SaaS<\/mark><\/strong>. A restart can also be initiated by vCenter HA settings in case of missing heartbeats from the OS.<\/p>\n\n\n\n

My “Windows BSoD” Approach<\/h5>\n\n\n\n

I quickly created a Windows Server 2019<\/strong> VM with this configuration:<\/p>\n\n\n\n

\"\"<\/a>
Figure 01: Windows VM configuration.<\/em><\/figcaption><\/figure>\n\n\n\n

And with the help of tools like Testlimit<\/code> and DiskSpd<\/code> plus some usual activities on the VM I created a “quick and dirty” baseline using metrics shown in the next picture (please ignore the color coding in the following picture for a moment). You will notice that in the Scoreboard the VMTools status is missing. I was not sure if I should include it or not as “tools not running” does not necessarily mean that the OS crashed, it could be also a crashed service.<\/p>\n\n\n\n

\"\"<\/a>
Figure 02: Windows 2019 Server VM baseline.<\/em><\/figcaption><\/figure>\n\n\n\n
Blue Screen of Death Examples<\/h5>\n\n\n\n

NotMyFault<\/a> is a perfect tool to crash Windows in various ways. As I wanted to check if different BSoD types have different symptoms I used that tool to force few crashes and collected a set of metrics for comparison.<\/p>\n\n\n\n

First crash type.<\/h6>\n\n\n\n

I started with probably the most known Blue Screen:<\/p>\n\n\n\n

\"\"<\/a>
Figure 03: IRQL not less or equal – BSoD.<\/em><\/figcaption><\/figure>\n\n\n\n

The first surprise was that the CPU Demand<\/code> of the VM immediately increased to almost 100%. To ensure that this is not related to the fact that Windows is collecting data after the crash for some time, I checked this metric after two collections cycles (10 minutes) and it did not go down. Second finding is that the RAM Usage<\/code> is only slowly decreasing, I assume this is simply due the the fact that memory is a really virtualized resource whereas CPU cycles inside the VM are actual cycles on the ESXi CPU. I also added the Guest Tools Status <\/code>to the Scoreboard but I would not use it as symptom in an Alert Definition<\/strong>.<\/p>\n\n\n\n

\"\"<\/a>
Figure 04: IRQL not less or equal – metrics.<\/em><\/figcaption><\/figure>\n\n\n\n
Second crash type.<\/h6>\n\n\n\n
\"\"<\/a>
Figure 05: IRQL gt zero at system service – BSoD.<\/em><\/figcaption><\/figure>\n\n\n\n

As you can see in the next picture, the metrics are behaving similarly to the first crash. Of course no disk and network usage at all was expected but to see that the CPU Demand<\/code> and RAM Usage<\/code> are following the same pattern is interesting and very promising symptom.<\/p>\n\n\n\n

\"\"<\/a>
Figure 06: IRQL gt zero at system service – metrics.<\/em><\/figcaption><\/figure>\n\n\n\n

This time I waited few more collection cycles to see how far the RAM usage will decrease and apparently it will go down to approx. 0.99% after >4 cycles.<\/p>\n\n\n\n

\"\"<\/a>
Figure 07: RAM and CPU usage pattern.<\/em><\/figcaption><\/figure>\n\n\n\n
Third crash type.<\/h6>\n\n\n\n
\"\"<\/a>
Figure 08: Unexpected kernel mode trap – BSoD.<\/em><\/figcaption><\/figure>\n\n\n\n

This BSoD type again resulted in the same metrics values. <\/p>\n\n\n\n

\"\"<\/a>
Figure 09: Unexpected kernel mode trap<\/em> – metrics.<\/em><\/figcaption><\/figure>\n\n\n\n

RAM on ESXi<\/code> as metric seems to be dependent on the memory usage of the VM before the crash. I did not fully test it but Aria Operations Metric Correlation<\/mark> feature shows the same pattern for the respective metrics.<\/p>\n\n\n\n

\"\"<\/a>
Figure 10: Memory metrics correlation<\/em>.<\/em><\/figcaption><\/figure>\n\n\n\n

Just to be sure that the RAM Usage<\/code> metric values do not change with different memory configurations of the VM, I did two more tests, the first one with 4GB RAM configured and the second with 17GB to check the metric with an odd RAM config.<\/p>\n\n\n\n

\"\"<\/a>
Figure 11: 4GB VM <\/em>metrics after crash.<\/em><\/figcaption><\/figure>\n\n\n\n

And here the 17GB RAM config.<\/p>\n\n\n\n

\"\"<\/a>
Figure 12: 17GB VM <\/em>metrics after crash.<\/em><\/figcaption><\/figure>\n\n\n\n
Constraints, assumptions and conclusions<\/h5>\n\n\n\n

Please be aware that I did not test every possible scenario, this is what I used:<\/p>\n\n\n\n