{"id":100,"date":"2018-12-24T13:18:01","date_gmt":"2018-12-24T11:18:01","guid":{"rendered":"http:\/\/thomas-kopton.de\/vblog\/?p=100"},"modified":"2018-12-24T13:18:29","modified_gmt":"2018-12-24T11:18:29","slug":"vrealize-log-insight-data-sets-using-custom-filters","status":"publish","type":"post","link":"https:\/\/thomas-kopton.de\/vblog\/?p=100","title":{"rendered":"vRealize Log Insight Data Sets using custom filters"},"content":{"rendered":"\n

IMPORTANT NOTE:<\/strong> All log file examples are not real and exact PCF log files. Since I do not have a running PCF environment while writing this post, I have used some fake examples and modified them to meet my needs.<\/p>\n\n\n\n

Let\u2019s consider following use case.<\/p>\n\n\n\n

You are responsible for operating a Pivotal Cloud Foundry (PCF) environment.<\/p>\n\n\n\n

PCF is sending all log messages for all spaces, orgs and apps etc. to one central vRealize Log Insight cluster.<\/p>\n\n\n\n

Your developers using the PCF services would like to access their log messages. <\/p>\n\n\n\n

Now, you could just grant read access to all developers but doing that you would allow every developer see all messages received by vRLI, not only PCF related.<\/p>\n\n\n\n

The first and pretty easy solution is to leverage the static tag already being used by the PCF Content Pack.<\/p>\n\n\n\n

The installation instruction of the content pack says:<\/p>\n\n\n\n

Log Insight Setup:<\/p>\n\n\n\n

1.      Obtain a Virtual IP from your IT department that will be under the same Subnet as your Log Insight environment. <\/p>\n\n\n\n

2.    Follow the `Procedure` defined in the following link: http:\/\/pubs.vmware.com\/log-insight-33\/topic\/com.vmware.log-insight.administration.doc\/GUID-72CB469A-F276-438D-BC93-77853E2CC0B9.html<\/a><\/p>\n\n\n\n

3.    When prompted for `a list of static tags (key=value)`, enter the following:<\/p>\n\n\n\n

            product=pcf<\/strong><\/p>\n\n\n\n

This static tag can be used for creation of an appropriated data set, which will contain only logs from your PCF environment.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

If you run this settings in interactive analytics you will get only those log messages which are tagged with “product=pcf”:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

But what if you want your developers to access only logs which belongs to their PCF org or\/and space?
Reading the documentation, you could come up with the idea of using extracted fields:<\/p>\n\n\n\n


“Use the first drop-down menu to select a field defined within\u00a0vRealize Log Insight\u00a0to filter on.\u00a0<\/em>
For example,\u00a0hostname<\/strong>.\u00a0<\/em>
The list contains all defined fields that are available statically, in content packs, and in custom content.”<\/em><\/p>\n\n\n\n

But if you try to use your own extracted fields in a data set you will notice that this kind of fields are not available in a data set configuration.
The solution I have used with my last customer was to configure the vRLI agent on the PCF syslog server to set static fields “dynamically” via RegEx expression. These static fields based on app or space IDs can be used in data set filters.\u00a0<\/p>\n\n\n\n


NOTE<\/strong>: This is just an example (fake messages):<\/p>\n\n\n\n


[filelog|pcf-syslog]
directory=\/var\/log\/pcf_sample
include=*.log
tags={\"tkopton_tag01\":\"testing-orgs\"}
parser=pcforg-parser
enabled=yes

[parser|pcforg-parser]
base_parser=regex
format=cf_org_id=(?<tkopton_tag02>\\S+\\b)
debug=no
next_parser=pcfspace-parser

[parser|pcfspace-parser]
base_parser=regex
format=cf_space_id=\"(?<tkopton_tag03>\\S+\\b)
debug=no
next_parser=syslog_parser<\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Now, you can use those fields as filters in data sets, like:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
And the test using “Run in Interactive Analytics”:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
IMPORTANT NOTE:<\/strong> Tested on a PCF Linux syslog server, I do not think, this will work on a Windows system;-)<\/p>\n","protected":false},"excerpt":{"rendered":"
IMPORTANT NOTE: All log file examples are not real and exact PCF log files. Since I do not have a running PCF environment while writing this post, I have used some fake examples and modified them to meet my needs. Let\u2019s consider following use case. You are responsible for operating a Pivotal Cloud Foundry (PCF) environment. …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[8,7],"class_list":["post-100","post","type-post","status-publish","format-standard","hentry","category-vrli","tag-parser","tag-vrli"],"yoast_head":"\nvRealize Log Insight Data Sets using custom filters - TOMsOps<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/thomas-kopton.de\/vblog\/?p=100\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"vRealize Log Insight Data Sets using custom filters - TOMsOps\" \/>\n<meta property=\"og:description\" content=\"IMPORTANT NOTE: All log file examples are not real and exact PCF log files. Since I do not have a running PCF environment while writing this post, I have used some fake examples and modified them to meet my needs. Let\u2019s consider following use case. You are responsible for operating a Pivotal Cloud Foundry (PCF) environment. ...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/thomas-kopton.de\/vblog\/?p=100\" \/>\n<meta property=\"og:site_name\" content=\"TOMsOps\" \/>\n<meta property=\"article:published_time\" content=\"2018-12-24T11:18:01+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-12-24T11:18:29+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/thomas-kopton.de\/vblog\/wp-content\/uploads\/2018\/12\/image.png\" \/>\n<meta name=\"author\" content=\"Thomas Kopton\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Thomas Kopton\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100\"},\"author\":{\"name\":\"Thomas Kopton\",\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/#\\\/schema\\\/person\\\/892d6b96c66b1dd4b75c6e32fdbfea82\"},\"headline\":\"vRealize Log Insight Data Sets using custom filters\",\"datePublished\":\"2018-12-24T11:18:01+00:00\",\"dateModified\":\"2018-12-24T11:18:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100\"},\"wordCount\":489,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/thomas-kopton.de\\\/vblog\\\/wp-content\\\/uploads\\\/2018\\\/12\\\/image.png\",\"keywords\":[\"parser\",\"vRLI\"],\"articleSection\":[\"vRLI\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100\",\"url\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100\",\"name\":\"vRealize Log Insight Data Sets using custom filters - TOMsOps\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/thomas-kopton.de\\\/vblog\\\/wp-content\\\/uploads\\\/2018\\\/12\\\/image.png\",\"datePublished\":\"2018-12-24T11:18:01+00:00\",\"dateModified\":\"2018-12-24T11:18:29+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/#\\\/schema\\\/person\\\/892d6b96c66b1dd4b75c6e32fdbfea82\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100#primaryimage\",\"url\":\"http:\\\/\\\/thomas-kopton.de\\\/vblog\\\/wp-content\\\/uploads\\\/2018\\\/12\\\/image.png\",\"contentUrl\":\"http:\\\/\\\/thomas-kopton.de\\\/vblog\\\/wp-content\\\/uploads\\\/2018\\\/12\\\/image.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?p=100#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"vRealize Log Insight Data Sets using custom filters\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/#website\",\"url\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/\",\"name\":\"TOMsOps\",\"description\":\"Just another VMware Cloud Management Blog\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/#\\\/schema\\\/person\\\/892d6b96c66b1dd4b75c6e32fdbfea82\",\"name\":\"Thomas Kopton\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e746aafbd3733172ceb4d600ba1feda61bc87cd3b70f5a9dfb581907cc7973b1?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e746aafbd3733172ceb4d600ba1feda61bc87cd3b70f5a9dfb581907cc7973b1?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/e746aafbd3733172ceb4d600ba1feda61bc87cd3b70f5a9dfb581907cc7973b1?s=96&d=mm&r=g\",\"caption\":\"Thomas Kopton\"},\"url\":\"https:\\\/\\\/thomas-kopton.de\\\/vblog\\\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"vRealize Log Insight Data Sets using custom filters - TOMsOps","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/thomas-kopton.de\/vblog\/?p=100","og_locale":"en_US","og_type":"article","og_title":"vRealize Log Insight Data Sets using custom filters - TOMsOps","og_description":"IMPORTANT NOTE: All log file examples are not real and exact PCF log files. Since I do not have a running PCF environment while writing this post, I have used some fake examples and modified them to meet my needs. Let\u2019s consider following use case. You are responsible for operating a Pivotal Cloud Foundry (PCF) environment. ...","og_url":"https:\/\/thomas-kopton.de\/vblog\/?p=100","og_site_name":"TOMsOps","article_published_time":"2018-12-24T11:18:01+00:00","article_modified_time":"2018-12-24T11:18:29+00:00","og_image":[{"url":"http:\/\/thomas-kopton.de\/vblog\/wp-content\/uploads\/2018\/12\/image.png","type":"","width":"","height":""}],"author":"Thomas Kopton","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Thomas Kopton","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100#article","isPartOf":{"@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100"},"author":{"name":"Thomas Kopton","@id":"https:\/\/thomas-kopton.de\/vblog\/#\/schema\/person\/892d6b96c66b1dd4b75c6e32fdbfea82"},"headline":"vRealize Log Insight Data Sets using custom filters","datePublished":"2018-12-24T11:18:01+00:00","dateModified":"2018-12-24T11:18:29+00:00","mainEntityOfPage":{"@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100"},"wordCount":489,"commentCount":0,"image":{"@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100#primaryimage"},"thumbnailUrl":"http:\/\/thomas-kopton.de\/vblog\/wp-content\/uploads\/2018\/12\/image.png","keywords":["parser","vRLI"],"articleSection":["vRLI"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/thomas-kopton.de\/vblog\/?p=100#respond"]}]},{"@type":"WebPage","@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100","url":"https:\/\/thomas-kopton.de\/vblog\/?p=100","name":"vRealize Log Insight Data Sets using custom filters - TOMsOps","isPartOf":{"@id":"https:\/\/thomas-kopton.de\/vblog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100#primaryimage"},"image":{"@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100#primaryimage"},"thumbnailUrl":"http:\/\/thomas-kopton.de\/vblog\/wp-content\/uploads\/2018\/12\/image.png","datePublished":"2018-12-24T11:18:01+00:00","dateModified":"2018-12-24T11:18:29+00:00","author":{"@id":"https:\/\/thomas-kopton.de\/vblog\/#\/schema\/person\/892d6b96c66b1dd4b75c6e32fdbfea82"},"breadcrumb":{"@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/thomas-kopton.de\/vblog\/?p=100"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100#primaryimage","url":"http:\/\/thomas-kopton.de\/vblog\/wp-content\/uploads\/2018\/12\/image.png","contentUrl":"http:\/\/thomas-kopton.de\/vblog\/wp-content\/uploads\/2018\/12\/image.png"},{"@type":"BreadcrumbList","@id":"https:\/\/thomas-kopton.de\/vblog\/?p=100#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/thomas-kopton.de\/vblog"},{"@type":"ListItem","position":2,"name":"vRealize Log Insight Data Sets using custom filters"}]},{"@type":"WebSite","@id":"https:\/\/thomas-kopton.de\/vblog\/#website","url":"https:\/\/thomas-kopton.de\/vblog\/","name":"TOMsOps","description":"Just another VMware Cloud Management Blog","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/thomas-kopton.de\/vblog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/thomas-kopton.de\/vblog\/#\/schema\/person\/892d6b96c66b1dd4b75c6e32fdbfea82","name":"Thomas Kopton","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/e746aafbd3733172ceb4d600ba1feda61bc87cd3b70f5a9dfb581907cc7973b1?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/e746aafbd3733172ceb4d600ba1feda61bc87cd3b70f5a9dfb581907cc7973b1?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e746aafbd3733172ceb4d600ba1feda61bc87cd3b70f5a9dfb581907cc7973b1?s=96&d=mm&r=g","caption":"Thomas Kopton"},"url":"https:\/\/thomas-kopton.de\/vblog\/?author=1"}]}},"_links":{"self":[{"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=\/wp\/v2\/posts\/100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=100"}],"version-history":[{"count":2,"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=\/wp\/v2\/posts\/100\/revisions"}],"predecessor-version":[{"id":103,"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=\/wp\/v2\/posts\/100\/revisions\/103"}],"wp:attachment":[{"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=100"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thomas-kopton.de\/vblog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}