19.18 Highly Available Storage (HAST)

19.18.3.1 Synchronization and Replication Modes

HAST tries to provide fast failure recovery. For this reason, it is very important to reduce synchronization time after a node's outage. To provide fast synchronization, HAST manages an on-disk bitmap of dirty extents and only synchronizes those during a regular synchronization (with an exception of the initial sync).

There are many ways to handle synchronization. HAST implements several replication modes to handle different synchronization methods:

• memsync: report write operation as completed when the local write operation is finished and when the remote node acknowledges data arrival, but before actually storing the data. The data on the remote node will be stored directly after sending the acknowledgement. This mode is intended to reduce latency, but still provides very good reliability. The memsync replication mode is currently not implemented.

• fullsync: report write operation as completed when local write completes and when remote write completes. This is the safest and the slowest replication mode. This mode is the default.

• async: report write operation as completed when local write completes. This is the fastest and the most dangerous replication mode. It should be used when replicating to a distant node where latency is too high for other modes. The async replication mode is currently not implemented.

19.18.4.1 Failover Configuration

The goal of this example is to build a robust storage system which is resistant to the failure of any given node. The scenario is that a primary node of the cluster fails. If this happens, the secondary node is there to take over seamlessly, check and mount the file system, and continue to work without missing a single bit of data.

To accomplish this task, another FreeBSD feature provides for automatic failover on the IP layer — CARP. CARP (Common Address Redundancy Protocol) allows multiple hosts on the same network segment to share an IP address. Set up CARP on both nodes of the cluster according to the documentation available in Section 32.14. After setup, each node will have its own carp0 interface with a shared IP address 172.16.0.254. The primary HAST node of the cluster must be the master CARP node.

The HAST pool created in the previous section is now ready to be exported to the other hosts on the network. This can be accomplished by exporting it through NFS, Samba etc, using the shared IP address 172.16.0.254. The only problem which remains unresolved is an automatic failover should the primary node fail.

In the event of CARP interfaces going up or down, the FreeBSD operating system generates a devd(8) event, making it possible to watch for the state changes on the CARP interfaces. A state change on the CARP interface is an indication that one of the nodes failed or came back online. These state change events make it possible to run a script which will automatically handle the HAST failover.

To be able to catch state changes on the CARP interfaces, add this configuration to /etc/devd.conf on each node:

notify 30 {
	match "system" "IFNET";
	match "subsystem" "carp0";
	match "type" "LINK_UP";
	action "/usr/local/sbin/carp-hast-switch master";
};

notify 30 {
	match "system" "IFNET";
	match "subsystem" "carp0";
	match "type" "LINK_DOWN";
	action "/usr/local/sbin/carp-hast-switch slave";
};

Restart devd(8) on both nodes to put the new configuration into effect:

# /etc/rc.d/devd restart

When the carp0 interface goes up or down (i.e., the interface state changes), the system generates a notification, allowing the devd(8) subsystem to run an arbitrary script, in this case /usr/local/sbin/carp-hast-switch. This is the script which will handle the automatic failover. For further clarification about the above devd(8) configuration, please consult the devd.conf(5) manual page.

An example of such a script could be:

#!/bin/sh

# Original script by Freddie Cash <fjwcash@gmail.com>
# Modified by Michael W. Lucas <mwlucas@BlackHelicopters.org>
# and Viktor Petersson <vpetersson@wireload.net>

# The names of the HAST resources, as listed in /etc/hast.conf
resources="test"

# delay in mounting HAST resource after becoming master
# make your best guess
delay=3

# logging
log="local0.debug"
name="carp-hast"

# end of user configurable stuff

case "$1" in
	master)
		logger -p $log -t $name "Switching to primary provider for ${resources}."
		sleep ${delay}

		# Wait for any "hastd secondary" processes to stop
		for disk in ${resources}; do
			while $( pgrep -lf "hastd: ${disk} \(secondary\)" > /dev/null 2>&1 ); do
				sleep 1
			done

			# Switch role for each disk
			hastctl role primary ${disk}
			if [ $? -ne 0 ]; then
				logger -p $log -t $name "Unable to change role to primary for resource ${disk}."
				exit 1
			fi
		done

		# Wait for the /dev/hast/* devices to appear
		for disk in ${resources}; do
			for I in $( jot 60 ); do
				[ -c "/dev/hast/${disk}" ] && break
				sleep 0.5
			done

			if [ ! -c "/dev/hast/${disk}" ]; then
				logger -p $log -t $name "GEOM provider /dev/hast/${disk} did not appear."
				exit 1
			fi
		done

		logger -p $log -t $name "Role for HAST resources ${resources} switched to primary."


		logger -p $log -t $name "Mounting disks."
		for disk in ${resources}; do
			mkdir -p /hast/${disk}
			fsck -p -y -t ufs /dev/hast/${disk}
			mount /dev/hast/${disk} /hast/${disk}
		done

	;;

	slave)
		logger -p $log -t $name "Switching to secondary provider for ${resources}."

		# Switch roles for the HAST resources
		for disk in ${resources}; do
			if ! mount | grep -q "^/dev/hast/${disk} on "
			then
			else
				umount -f /hast/${disk}
			fi
			sleep $delay
			hastctl role secondary ${disk} 2>&1
			if [ $? -ne 0 ]; then
				logger -p $log -t $name "Unable to switch role to secondary for resource ${disk}."
				exit 1
			fi
			logger -p $log -t $name "Role switched to secondary for resource ${disk}."
		done
	;;
esac

In a nutshell, the script takes these actions when a node becomes master / primary:

• Promotes the HAST pools to primary on a given node.

• Checks the file system under the HAST pool.

• Mounts the pools at an appropriate place.

When a node becomes backup / secondary:

• Unmounts the HAST pools.

• Degrades the HAST pools to secondary.

More detailed information with additional examples can be found in the HAST Wiki page.

19.18.5.1 General Troubleshooting Tips

HAST should generally work without issues. However, as with any other software product, there may be times when it does not work as supposed. The sources of the problems may be different, but the rule of thumb is to ensure that the time is synchronized between all nodes of the cluster.

When troubleshooting HAST problems, the debugging level of hastd(8) should be increased by starting the hastd(8) daemon with the -d argument. Note that this argument may be specified multiple times to further increase the debugging level. A lot of useful information may be obtained this way. Consider also using the -F argument, which starts the hastd(8) daemon in the foreground.

19.18.5.2 Recovering from the Split-brain Condition

Split-brain is when the nodes of the cluster are unable to communicate with each other, and both are configured as primary. This is a dangerous condition because it allows both nodes to make incompatible changes to the data. This problem must be corrected manually by the system administrator.

The administrator must decide which node has more important changes (or merge them manually) and let HAST perform full synchronization of the node which has the broken data. To do this, issue these commands on the node which needs to be resynchronized:

# hastctl role init <resource>
# hastctl create <resource>
# hastctl role secondary <resource>