Aria Operations, Integration SDK, VCF, VCF Operations, vRealize Operations, vROps

Calculating Metric Deltas in VCF Operations with a Small Management Pack

Scenario

Most metrics in VCF Operations answer the question “what is the value right now”. A lot of the questions I get from customers are about something else: “how much did this move since the last collection cycle?”

Think of a counter that only ever goes up, a capacity number that should stay flat, or a metric where the interesting part is the jump and not the level. In all of those cases you do not want the value, you want the difference between the last two values.

That is harder than it sounds. A super metric formula works with the values of the current cycle, across objects, and there is no “the value one cycle ago” to put in the formula. So the difference between the last and the previous collection is not something you can build in the formula editor.

So I built a small management pack with the VCF Operations Integration SDK that does exactly one thing: it calculates that difference and writes it back as a new metric on the same object.

The Metric Delta Management Pack

You give it a list of object types and stat keys in a configuration file:

{
  "metricSets": [
    {
      "adapterKind": "VMWARE",
      "resourceKind": "VirtualMachine",
      "maxSampleAgeMinutes": 15,
      "metrics": ["disk|usage_average", "net|usage_average"]
    }
  ]
}

and after two collection cycles every VM carries the deltas:

Source metricNew metric
disk|usage_averageCustom Metrics|Delta|disk|usage_average
net|usage_averageCustom Metrics|Delta|net|usage_average

A few properties of the pack worth naming up front:

  • It attaches the deltas to the existing objects of the vCenter adapter. No new objects, no second inventory tree.
  • It never connects to vCenter. The only thing it talks to is the Suite API of the local VCF Operations instance.
  • If there is no delta, nothing is written. More on that below, because it is the one design decision in the whole pack.
  • The metric sets are configured in the UI, not in the adapter code, so adding a metric does not mean rebuilding anything.

How It Works

Every collection cycle the adapter:

  1. reads its configuration file with GET api/configurations/files?path=SolutionConfig/<file>,
  2. lists all objects of the configured adapterKind and resourceKind via GET api/resources,
  3. asks for the last two samples of the configured stat keys with POST api/resources/stats/latest/query and maxSamples: 2, in batches of 200 object IDs,
  4. returns last - previous as a metric on those same objects, using the SDK’s key_to_object() so the keys match the existing ones.

The “extending an existing management pack” pattern from the SDK documentation is what makes step 4 work. You rebuild the resource key of an object another adapter owns, and whatever metrics you return land on that object.

Why no value is better than zero

Here is the decision I mentioned. What should the adapter write when an object does not have two samples of a metric? A new VM, a metric that is not collected for that object, a collection that was skipped?

Zero would be wrong. Zero is a perfectly valid delta: it means the value did not change. If you write zero for “I do not know”, every average, every alert threshold and every view column silently mixes “no change” with “no data”, and you will not see the difference.

So the adapter writes nothing in that case. The metric simply does not appear on that object for that cycle. It is one if in the code and it is the single most important line in the whole pack.

The same thinking produced the optional maxSampleAgeMinutes.

What maxSampleAgeMinutes does

It is a freshness check on the newest of the two samples, nothing else. At collection time the adapter compares the timestamp of the newest sample with the current time:

now - timestamp(newest sample) > maxSampleAgeMinutes  ->  no value is written

Without the setting there is no such check: a delta is calculated whenever two samples exist, however old they are.

Why it matters: when a metric stops being collected, because the VM is powered off, the source adapter stopped or the metric was removed from a policy, its last two samples stay in the database. The delta is still calculable, and every cycle would write the same number again with a current timestamp. The delta metric would look alive while its source is dead. With the setting, the delta simply stops, which is the honest answer.

A sensible value is a little more than the collection interval of the source metric, for example 15 minutes for a metric collected every 5 minutes, so one missed cycle does not blank the delta.

One thing it does not do: it does not limit the gap between the two samples. If an object was not collected for a day and comes back, the two newest samples can be a day apart, and the delta covers that whole gap. The newest sample is fresh, so the check passes.

Lessons Learned in the Lab

1. The configuration file does not have to be XML

The files under Configurations > Management Packs Configuration are usually XML, but the content of those files is free format. The UI gives you a text editor and the Suite API hands your adapter back whatever is in the file.

That means you can use JSON, which is both easier to read and a single json.loads() away from a Python dict. The only loss is comments, so the reference file keeps its documentation in a _comment key that the adapter ignores.

Reading the file is one call:

GET api/configurations/files?path=SolutionConfig/<name>.json

2. The Suite API is not available during Test Connection

My first version read and validated the configuration file in the test() method, which is the obvious place for it. On a live VCF Operations instance the test connection then failed with:

TypeError("'NoneType' object does not support the context manager protocol")

The suite_api_client of the adapter instance was None. VCF Operations passes Suite API credentials to an adapter during collection, not during test connection.

So the test connection now validates the adapter instance parameters only, and the configuration file is read and validated at the start of every collection. An error there shows up as a collection error on the adapter instance and in the adapter log. The downside is honest: a typo in the file name is only visible after the first collection, not when you save the account.

3. mp-test and file permissions

Before any of that, my very first mp-test run died with:

/usr/local/bin/python: can't open file '/home/aria-ops-adapter-user/src/app/app/adapter.py': [Errno 13] Permission denied

The container runs as an unprivileged user, and Docker’s COPY keeps the permissions your files have on your machine. My adapter.py was 600, so the container user could not read it. A chmod -R a+rX app content conf and the problem was gone. Worth remembering if your umask is 077.

4. Read your own configuration file before debugging the code

The VM deltas showed up on the first try. The host deltas did not, and the log did not even mention the host metric set. I checked the stat keys, I checked the object type, I was halfway into the Suite API queries when I looked at the configuration file again and found "enabled": false on that metric set, left over from the example file I had cloned.

The pack now logs every metric set it loads and every one it skips. That is a cheap line of code that would have saved me twenty minutes.

Get It and Set It Up

Everything is on GitHub: Metric-Delta-MP

Install the ready-made .pak

Download MetricDeltaAdapter_1.0.0.pak and:

  1. Go to Data Sources > Integrations > Repository > Add and upload the .pak.
  2. Under Configurations > Management Packs Configuration, clone the shipped metric_delta_config.json and edit the clone. Do not edit the shipped file, an upgrade of the pack overwrites it.
  3. Under Accounts > Add Account, choose Metric Delta, enter the name of your configuration file, pick the cloud proxy and save.
  4. Wait two collection cycles, then open one of the configured objects and look under Custom Metrics | Delta.

The .pak itself contains no adapter code. It points to a container image on Docker Hub, which the cloud proxy pulls when the account starts, so the cloud proxy needs access to Docker Hub. In an air-gapped environment, build your own .pak and push the image to a registry your proxy can reach.

Configuration file keys

KeyRequiredDescription
adapterKindyesAdapter kind owning the objects, e.g. VMWARE
resourceKindyesObject type, e.g. VirtualMachine, HostSystem
metricsyesStat keys, case-sensitive, exactly as the source adapter publishes them
maxSampleAgeMinutesnoFreshness check on the newest sample, see above
enablednofalse skips the whole set, default true

Build it yourself

pip install vmware-aria-operations-integration-sdk
cd vcf-operations-content/Metric-Delta-MP
python3 -m venv "venv-Metric Delta Adapter"
. "venv-Metric Delta Adapter/bin/activate"
pip install -r adapter_requirements.txt pytest

python -m pytest -q tests     # unit tests, no VCF Operations needed
mp-test collect               # against a live instance
mp-build                      # build the .pak

For mp-test collect, answer Yes to “Set connection information for SuiteAPI calls?”. Without it the collection has no Suite API client, and you will meet lesson 2 from the other side.

Putting the Deltas to Work

Once the deltas are on the objects they behave like any other metric:

  • as a column in a view or a report,
  • as the input of a symptom definition, so you can alert on a jump instead of a level,
  • in a super metric, where the delta is now just another stat key you can reference,
  • as the metric of a custom group membership rule.

Known Limitations

  • Metrics added to another adapter’s objects cannot carry metadata, so the delta appears with the raw stat key of the source metric, without unit or label.
  • The delta is always one collection cycle behind the source metric.
  • A delta is a difference per collection interval, not a rate. If the collection interval of the source metric changes, deltas from before and after are not comparable.
  • Stat keys are case-sensitive. A wrong key is not an error, it simply produces no value, which looks exactly like “no data”.

Give it a try in your lab and let me know what you would want it to do next. Issues and pull requests are welcome on the GitHub project.

Disclaimer: this is a community project, not a Broadcom product. It is not supported by Broadcom. Test it in your lab before you let it anywhere near production.

Stay safe.

Thomas – https://twitter.com/ThomasKopton

Leave a Reply

Your email address will not be published. Required fields are marked *