Even configured as a cluster vRealize Log Insight does not support high availability in terms of availability of data and availability of all functions and configuration data.
What does it mean exactly?
In a vRLI cluster there are basically two types of nodes:
- one master node
- up to 11 worker nodes (as for vRLI 4.5)
What will happen if one (or more) of those nodes fails?
- worker node
In case it is a worker node, the cluster remains fully accessible but we will not be able to access the data which was stored on this particular node. It might also be, that exactly this node was holding the VIP, in this case the cluster will elect a new node to hold the VIP. But what if this node cannot be restored anymore?
- You have a full backup of this node – everything will be fine, just run you restore procedure and your back to business.
- You, for some reason, don’t have any backups. For sure, you will loose the data but “restoring” the node is as easy as removing the failed node from the cluster (https://docs.vmware.com/en/vRealize-Log-Insight/4.3/com.vmware.log-insight.administration.doc/GUID-116098CD-6F9D-4FC9-A037-CB2CAE035B29.html) adding a new node to the cluster, you can just use the same name and IP.
- master node
In case it is a master node, again the same applies: the cluster remains fully accessible but we will not be able to access the data which was stored on this particular node. It might also be, that exactly this node was holding the VIP, in this case the cluster will elect a new node to hold the VIP. BUT you will not be able to access and change the cluster configuration, the status will be unavailable etc.:
But what if this node cannot be restored anymore?
- You have a full backup of this node – everything will be fine, just run you restore procedure and your back to business.
- You, for some reason, don’t have any backups, nothing, not even a single file. You’re screwed!
But there is good news even if you cannot backup the whole node (maybe it is just too big, or for whatever reason) – just backup the right data to make a master node restore as easy as restoring a worker node.
### to edit ###
These are the steps to backup and restore the master node (minimal version), for a regular full backup, please follow the official VMware documentation (vRLI 4.5) https://docs.vmware.com/en/vRealize-Log-Insight/4.5/com.vmware.log-insight.administration.doc/GUID-FB70EF83-7E6B-4AEE-9522-CD6173F52FA0.html:
- Backup following files on your master node on a regular basis:
/storage/core/loginsight/config/loginsight-config.xml#number /storage/core/loginsight/config/node-token
/storage/var/loginsight/apache-tomcat/conf/keystore
which is the target of the symlink:
The restore procedure after the master node is lost:
- Spin up a new log insight server with same IP and FQDN as the old master node
- Stop the loginsight service (connected via ssh): service loginsight stop
- Create the config folder: /storage/core/loginsight/config
- Copy all loginsight-config.xml#number and node-token files to /storage/core/loginsight/config/ (Note: “config” folder has to be manually created)
- Copy keystore file to /storage/var/loginsight/apache-tomcat/conf/
- Check the symlink: /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.15/conf -> /storage/var/loginsight/apache-tomcat/conf
- Reboot the log insight appliance
- Login to UI and checked the cluster status
Disclaimer: This method does not replace a proper backup of the system as described in the VMware documentation!