In a Fusion high-availability cluster on a Linux platform, ensure the live files on each server are copied to static local system files on each server and then backup these static files to an archival storage system like a Virtual Tape Device.

In this example, local system files on each Fusion server are backed up to the fusion user's home folder.

cd /home/fusion/backup

Configuration

The files in ZOOKEEPER_HOME/conf determine how Zookeeper behaves, for example: initLimit, tickTime and other properties regulate how the local zookeeper syncs with its fellow ensemble servers and how it communicates with Solr nodes. Since these files store Fusion and Solr cluster information, don't expect to use these in another Zookeeper cluster without special care or modification.

Manual Backup

Copy local server Zookeeper config files and export config files from all servers in cluster to single JSON file in /home/backup.

Copy Zookeeper config directory
cp -rf $FUSION_HOME/conf/zookeeper ~/backup/zookeeper-config

Export all config files from all servers in cluster to single JSON file
$FUSION_HOME/scripts/zkImportExport.sh -zkhost 10.11.12.13:9983,10.11.12.14:9983,10.11.12.15:9983 -cmd export -path / -filename ~/backup/zookeeper_dump.json

Note: the alternative using Solr "zkcli.sh" client script from a terminal command line to save Zookeeper config files does not appear to work using the command:
sh $FUSION_HOME/apps/solr-dist/server/scripts/cloud-scripts/zkcli.sh -zhost 10.11.12.13:9983,10.11.12.14:9983,10.11.12.15:9983 -cmd downconfig -confdir ~/backup -confname myconf

Automatic backup

Schedule manual backup command to run at 1:01am every day as fusion user (see Summary section below)

Edit crontab
crontab -e

# Zookeeper
1 1 * * * cp -rf /opt/fusion/3.0.1/conf/zookeeper /home/fusion/backup/zookeeper-config

# Config files
1 1 * * * /opt/fusion/3.0.1/scripts/zkImportExport.sh -zkhost 10.11.12.13:9983,10.11.12.14:9983,10.11.12.15:9983 -cmd export -path / -filename /home/fusion/backup/zookeeper_dump.json
Shell

Backup Fusion Snapshots

Zookeeper takes snapshots of data files of all its interactions automatically. The zoo.cfg file specifies where Zookeeper saves these snapshots.

To look at the Zookeeper config file, go to …
nano $FUSION_HOME/conf/zoo.cfg

… which contains

# Snapshot directory relative to the Fusion home directory
dataDir=data/zookeeper
Shell

… which points to the zookeeper directory
$FUSION_HOME/data/zookeeper

Manual Backup

Copy the entire contents of this folder to the home directory on the server. Do this on all servers in the cluster.
cp -rf $FUSION_HOME/data/zookeeper ~/backup/zookeeper-data

Tip: during a restore, be sure to turn Solr off and on surrounding the process. There are ephemeral nodes attached to Solr sessions that  will be in a confused state when Solr is restarted otherwise.

Automatic Backup

Schedule manual backup command to run at 1:01am every day as fusion user (see Summary section below)

Edit crontab
crontab -e

# Snapshots  
1 1 * * * cp -rf /opt/fusion/3.0.1/data/zookeeper /home/fusion/backup/zookeeper-data
Shell

Tip: crontab does not accept path shortcuts like $PATH or ~. Everything needs to be spelled out in full paths