# Install multiple AxoRouters in high availability mode

Deploy multiple AxoRouter hosts as a high availability cluster using keepalived to manage a shared virtual IP, so sources keep reaching AxoRouter through host failures.

This guide describes how to deploy multiple AxoRouter hosts as a high availability cluster with `keepalived` managing a shared virtual IP across the hosts. For an overview of other approaches, see [AxoRouter high availability](../../../docs/axoflow/deployment-scenarios/axorouter-high-availability/index.md).

![High availability AxoRouter cluster with keepalived](/docs/axoflow/img/axorouter-ha-virtual-ip-keepalived.svg)

The examples in this guide deploy AxoRouter on AlmaLinux 10, but the procedure also works on other supported platforms.

The hosts running AxoRouter can be physical or virtual machines. The recommended layout depends on which you choose:

  * Physical machines: Install two machines with a dedicated HA link between them to keep the HA traffic separate from the production traffic.
  * Virtual machines: Set up three VMs with a dedicated (and guaranteed bandwidth) HA link (HA VLAN) spanning across them, to keep the HA traffic separate from the production traffic.



Note

A cluster targeting highly reliable services should ideally consist of at least three nodes to avoid split-brain situations. In a split-brain situation, both nodes think that they are the last surviving node (for example, during a network failure) and bring up their services, leading to several problems.

The setups described here use fewer than the ideal number of nodes for cost reasons. Dedicated HA links provide an additional channel for HA communication independently of the production traffic, and significantly reduce the risk of split-brain scenarios.

At a high level, setting up the cluster involves the following steps:

  1. [Prepare every cluster host](../../../docs/axoflow/provisioning/axorouter/high-availability/index.md#hosts): deploy AxoRouter, install `keepalived`, and configure SELinux to let `keepalived` query the AxoRouter service status.
  2. [Configure `keepalived` on the primary node](../../../docs/axoflow/provisioning/axorouter/high-availability/index.md#primary) with the `MASTER` state, the shared virtual IP, and the addresses of the peer nodes.
  3. [Configure `keepalived` on each secondary node](../../../docs/axoflow/provisioning/axorouter/high-availability/index.md#secondary) with the `BACKUP` state and a lower priority than the primary.
  4. Open TCP port 112 on every host’s firewall so the nodes can exchange VRRP advertisements, then enable and start the `keepalived` service.



The following sections walk through each step in detail.

## Prepare the hosts

On every AxoRouter host that is a member of the high availability cluster, complete the following steps.

  1. Deploy AxoRouter on every host as described in [Install AxoRouter on Linux](../../../docs/axoflow/provisioning/axorouter/linux/index.md).

  2. Install `keepalived` from the OS repositories.
```
 sudo dnf install keepalived
         
```

  3. Configure SELinux so `keepalived` can monitor the status of AxoRouter. For that, create and load a SELinux policy module that allows `keepalived` to run the `systemctl status` command.

     1. Install the `selinux-policy-devel` package:
```
 sudo dnf install selinux-policy-devel
            
```

     2. Create a policy module file.
```
 cat > keepalived-allow-systemctl-status.te <<EOF
            
            module keepalived-allow-systemctl-status 1.0;
            
            require {
                    type keepalived_t;
                    type syslogd_var_run_t;
                    type systemd_systemctl_exec_t;
                    type systemd_generator_unit_file_t;
                    class file { execute execute_no_trans getattr map open read setattr };
                    class dir read;
                    class service status;
            }
            
            #============= keepalived_t ==============
            allow keepalived_t syslogd_var_run_t:dir read;
            
            #!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
            allow keepalived_t systemd_systemctl_exec_t:file { execute execute_no_trans getattr open read setattr };
            
            #!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
            allow keepalived_t systemd_systemctl_exec_t:file map;
            
            allow keepalived_t systemd_generator_unit_file_t:service status;
            EOF
            
```

     3. Build and load the SELinux module:
```
 checkmodule -M -m -o keepalived-allow-systemctl-status.mod keepalived-allow-systemctl-status.te
            semodule_package -o keepalived-allow-systemctl-status.pp -m keepalived-allow-systemctl-status.mod
            sudo semodule -i keepalived-allow-systemctl-status.pp
            
```

  4. Repeat these steps on the other AxoRouter hosts.




## Set up the primary node

  1. Set up the `keepalived` configuration on the primary node. See the list of main configuration elements after the command.
```
 sudo mv /etc/keepalived/keepalived.conf{,.old} -v
         sudo cat > /etc/keepalived/keepalived.conf <<EOF
         
         vrrp_script check_axorouter {
             script  "/usr/bin/systemctl status axorouter"
             interval 5
             timeout  2
             rise     5
             fall     2
             user     root
         }
         
         vrrp_instance axorouter_vip {
             state MASTER
             interface enp0s8
             virtual_router_id 51
             priority 254
             advert_int 1
             unicast_src_ip 192.168.56.10
             unicast_peer {
                 192.168.56.11
             }
             authentication {
                 auth_type PASS
                 auth_pass Ax0R0ut3r
             }
             virtual_ipaddress {
                 192.168.10.64/25 dev enp0s9
             }
             track_script {
                 check_axorouter
             }
             track_interface {
                 enp0s9                      # Fail over if public interface dies
                 enp0s8                      # Fail over if dedicated HA interface dies
             }
         }
         EOF
         
```

The main configuration elements are:

     * `state`: The default state of the node.
     * `interface`: The interface `keepalived` is listening on (in this case the HA interface).
     * `virtual_router_id`: This ID should match across the nodes belonging to the same cluster.
     * `priority`: An assigned value belonging to the cluster node. During an election, the node with the highest priority wins. The maximum value is 254.
     * `unicast_src_ip`: The source IP address `keepalived` sends its advertisements with (in this case on the HA interface).
     * `unicast_peer`: IP addresses of the other cluster nodes (in this case the IP addresses on their respective HA interfaces).
     * `auth_pass`: The password can have any value. Maximum length is 8 characters; longer values are truncated. All members of the cluster must have the same value.
     * `virtual_ipaddress`: The virtual IP address and the interface it should be configured on (the interface facing the data sources).
     * `track_script`: A shell script that can be used to verify the status of the service.
     * `track_interface`: A list of network interfaces the status of which `keepalived` should monitor (the interface `keepalived` is listening on is in this list by default).
  2. Enable inbound traffic on port 112/TCP on the host’s firewall.
```
 sudo firewall-cmd --permanent --add-port=112/tcp && firewall-cmd --reload
         
```

  3. Enable and start the `keepalived` service.
```
 sudo systemctl enable --now keepalived
         
```




## Set up the secondary nodes

Perform the following steps on the secondary hosts.

  1. Deploy the following `keepalived` configuration.
```
 sudo mv /etc/keepalived/keepalived.conf{,.old} -v
         sudo cat > /etc/keepalived/keepalived.conf <<EOF
         
         vrrp_script check_axorouter {
             script  "/usr/bin/systemctl status axorouter"
             interval 5
             timeout  2
             rise     5
             fall     2
             user     root
         }
         
         vrrp_instance axorouter_vip {
             state BACKUP
             interface enp0s8
             virtual_router_id 51
             priority 253
             advert_int 1
             unicast_src_ip 192.168.56.11
             unicast_peer {
                 192.168.56.10
             }
             authentication {
                 auth_type PASS
                 auth_pass Ax0R0ut3r
             }
             virtual_ipaddress {
                 192.168.10.64/25 dev enp0s9
             }
             track_script {
                 check_axorouter
             }
             track_interface {
                 enp0s9                      # Fail over if public interface dies
                 enp0s8                      # Fail over if dedicated HA interface dies
             }
         }
         EOF
         
```

  2. Enable inbound traffic at port 112/TCP on the host’s firewall.
```
 sudo firewall-cmd --permanent --add-port=112/tcp && firewall-cmd --reload
         
```

  3. Enable and start the `keepalived` service.
```
 sudo systemctl enable --now keepalived
         
```




After this, you should have a functional virtual/service IP address so the virtual IP moves to another AxoRouter host if the active one fails.