{"id":338,"date":"2020-03-05T17:10:37","date_gmt":"2020-03-05T16:10:37","guid":{"rendered":"http:\/\/thomas-kopton.de\/vblog\/?p=338"},"modified":"2020-07-29T12:57:55","modified_gmt":"2020-07-29T11:57:55","slug":"parsing-csv-files-using-vrealize-log-insight","status":"publish","type":"post","link":"https:\/\/thomas-kopton.de\/vblog\/?p=338","title":{"rendered":"Parsing CSV Files using vRealize Log Insight"},"content":{"rendered":"\n

Last year I published a blog post focusing on the JSON parser<\/strong> available in vRealize Log Insight<\/strong> (https:\/\/thomas-kopton.de\/vblog\/?p=144<\/a>) and I believe this could have been a beginning of a series \u2013 \u201cvRLI Parsers<\/strong> \u201c.<\/p>\n\n\n\n

In this post I will show how to configure and utilize the CSV parser<\/strong> using a very prominent and media-effective use case:<\/p>\n\n\n\n

COVID-19\naka Coronavirus!<\/strong><\/strong><\/p>\n\n\n\n

The needed data is publicly available, e.g. here:<\/p>\n\n\n\n

https:\/\/github.com\/CSSEGISandData\/COVID-19<\/a> The actual data sources are listed on the GitHub project page and the data itself is already in the format we need to feed it into vRLI \u2013 CSV files. <\/p>\n\n\n\n

Data Review<\/h4>\n\n\n\n

The time series data files provided by CSSE are in the following format: <\/p>\n\n\n\n

\"\"<\/a>
Figure 1: CSSE CSV file format<\/em><\/figcaption><\/figure>\n\n\n\n

Or shown as raw data:<\/p>\n\n\n\n

\"\"<\/a>
Figure 2: CSSE CSV file format – in an editor<\/em><\/figcaption><\/figure>\n\n\n\n

vRealize Log Insight processes every single CSV row as one message and if we do not specify our own timestamp as part of the message, vRLI will use the ingestion time<\/strong> as the timestamp<\/strong> for the message. This is clearly nothing we would like to have in order to have some meaningful diagrams showing us e.g. how the virus spreads in various countries. We will change the timestamp accordingly.<\/p>\n\n\n\n

We also see that the original data contains \u201c,\u201d signs as part of province or state names, this is not compatible with the vRLI CSV parser as it would change the number and content of the fields. I have used Numbers on my Mac to “clean up” the file and replace the real “,” field separation signs with “;”.<\/p>\n\n\n\n

To ingest the data, we will use the vRLI agent and an appropriate configuration of the agent. <\/p>\n\n\n\n

vRLI Agent Config <\/h4>\n\n\n\n

My setup in for this use case is:<\/p>\n\n\n\n