High Availability Controller Swiftlet

Overview

The High Availability Controller Swiftlet manages the active/standby state machine for SwiftMQ routers operating in HA mode. It coordinates state transitions, heartbeats, configuration replication, and failover decisions between paired routers to ensure continuous service availability. This Swiftlet handles all negotiation, synchronization, and split-brain resolution logic, as well as the underlying replication channel infrastructure.

Features

Active/Standby State Machine

The HA Controller Swiftlet implements a robust state machine to manage the lifecycle of a router in an HA pair. States include UNKNOWN, INITIALIZE, NEGOTIATE, STANDALONE, ACTIVE-SYNC-PREPARE, ACTIVE-SYNC, ACTIVE, STANDBY-SYNC-PREPARE, STANDBY-SYNC, and STANDBY. The state machine is responsible for orchestrating transitions based on negotiation outcomes, heartbeat monitoring, and failover events. The current state is tracked and can be monitored via the current-instance-state property under the usage entity. The state machine ensures that only one router is active at a time, with the other in standby, and coordinates transitions during startup, failover, or recovery.

Preferred Active Instance

The preferred-active property determines which router in the HA pair should become the active instance if both are eligible. This helps automate failback to a designated primary router after a failover event. The state machine uses this preference during negotiation and split-brain resolution.

Negotiation Timeout

The negotiation-timeout property specifies the maximum time (in milliseconds) the router will wait for HA negotiation to complete before taking recovery action. If negotiation does not complete in time, the router may revert to standalone mode or take other actions based on the last saved state.

Split Brain Instance Action

The split-brain-instance-action property controls how the router responds if a split-brain condition is detected (i.e., both routers believe they are active). Supported actions are keep (keep running), stop (shutdown), or backup-and-standby (backup store and restart as standby). This ensures data consistency and prevents dual-active scenarios.

Configuration Example:

<swiftlet name="sys$hacontroller" preferred-active="true" negotiation-timeout="60000" split-brain-instance-action="backup-and-standby"/>

Replication Channel and Heartbeat Monitoring

The HA Controller Swiftlet establishes a dedicated replication channel between HA routers for state and configuration synchronization. This channel is defined under the replication-channel entity and supports both listener (server) and connector (client) modes. Only one listener and one connector are permitted. Heartbeat messages are exchanged at a configurable interval to monitor peer liveness. If heartbeats are missed beyond the configured threshold, the replication connection is closed and the state machine may initiate failover or recovery.

Heartbeat Interval and Threshold

The heartbeat-interval property sets the frequency (in milliseconds) at which heartbeat messages are sent. The heartbeat-missing-threshold property defines how many consecutive heartbeats can be missed before the connection is considered lost and recovery actions are triggered.

Replication Channel Listener/Connector Settings

The replication channel supports fine-tuned network settings for both listeners and connectors, including bind address, port, TCP no delay, buffer sizes, and retry intervals. Only one listener and one connector may be configured. These settings ensure reliable and performant replication traffic between HA peers.

Configuration Example:

<swiftlet name="sys$hacontroller">
  <replication-channel heartbeat-interval="1000" heartbeat-missing-threshold="5">
    <listeners>
      <listener name="main" port="40000" bindaddress="0.0.0.0"/>
    </listeners>
    <connectors>
      <connector name="to-peer" hostname="ha-peer.example.com" port="40000" retry-time="2000"/>
    </connectors>
  </replication-channel>
</swiftlet>

Configuration Replication and Exclusions

The HA Controller Swiftlet ensures that configuration changes made on the active router are replicated in real time to the standby router. This includes property changes, entity additions/removals, and other dynamic configuration updates. The configuration controller supports property substitution rules and exclusion lists to control which configuration elements are replicated. These lists are maintained under the configuration-controller entity and allow administrators to exclude sensitive or instance-specific settings from replication, or to substitute values as needed for the standby instance.

Property Substitutions

The property-substitutions entity list allows administrators to define rules for substituting property values during replication. This is useful for properties that must differ between active and standby instances (e.g., network addresses, instance IDs). Each substitution specifies the property to substitute and the replacement value.

Replication Excludes

The replication-excludes entity list allows administrators to specify configuration contexts or properties that should not be replicated to the standby instance. This helps prevent accidental overwriting of instance-unique settings.

Configuration Example:

<swiftlet name="sys$hacontroller">
  <configuration-controller>
    <property-substitutions>
      <property-substitution name="/sys$net/listeners/main/port">
        <substitute-with value="40001"/>
      </property-substitution>
    </property-substitutions>
    <replication-excludes>
      <replication-exclude name="/sys$log"/>
    </replication-excludes>
  </configuration-controller>
</swiftlet>

Replication Tunnels (Internal Infrastructure)

Replication tunnels define the logical communication paths between HA routers for configuration and state synchronization. Each tunnel has an address and a set of supported protocol versions. These entities are predefined in the configuration and should not be modified by administrators. The HA Controller uses these tunnels to negotiate protocol compatibility and establish replication sessions.

Spool Management (Internal Infrastructure)

The HA Controller Swiftlet manages spooling of replication items to disk or memory to ensure reliable delivery even in the event of network interruptions or failover. Spool settings, such as directory and cache size, are predefined and should not be changed. The spool infrastructure is used internally for buffering configuration and state updates during synchronization and failover.

Configuration Guide

Configuring the Preferred Active Instance

Use this scenario to designate one router in the HA pair as the preferred active instance. This ensures that after a failover, the system will automatically fail back to the preferred router when it becomes available again.

1. Edit the routerconfig.xml on the preferred router.
2. Set the preferred-active attribute to true on the sys$hacontroller swiftlet entity.
3. Restart the router for the change to take effect.

<swiftlet name="sys$hacontroller" preferred-active="true"/>

Tuning Heartbeat Sensitivity

Adjust the heartbeat interval and missing threshold to detect peer failures more quickly or tolerate longer network interruptions. Lower intervals and thresholds result in faster failover but may increase sensitivity to transient network issues.

1. Edit the routerconfig.xml on both HA routers.
2. Set the heartbeat-interval and heartbeat-missing-threshold attributes under the replication-channel entity.
3. Restart both routers for the changes to take effect.

<swiftlet name="sys$hacontroller">
  <replication-channel heartbeat-interval="1000" heartbeat-missing-threshold="3"/>
</swiftlet>

Customizing Split Brain Handling

Choose the appropriate action for your deployment if a split-brain scenario is detected. For example, to ensure data consistency, you may want the instance to backup its store and restart as standby.

1. Edit the routerconfig.xml on both HA routers.
2. Set the split-brain-instance-action attribute to one of: keep, stop, or backup-and-standby.
3. Restart both routers for the change to take effect.

<swiftlet name="sys$hacontroller" split-brain-instance-action="backup-and-standby"/>

Configuring Replication Channel Listener and Connector

Define the network endpoints for the replication channel between HA routers. Only one listener and one connector are allowed. The listener waits for incoming connections, while the connector initiates connections to the peer.

1. On router A, configure a listener under the replication-channel entity with the desired port and bind address.
2. On router B, configure a connector under the replication-channel entity with the hostname and port of router A.
3. Ensure only one listener and one connector are present in each configuration.
4. Restart both routers for the changes to take effect.

<swiftlet name="sys$hacontroller">
  <replication-channel>
    <listeners>
      <listener name="main" port="40000" bindaddress="0.0.0.0"/>
    </listeners>
    <connectors>
      <connector name="to-peer" hostname="ha-peer.example.com" port="40000"/>
    </connectors>
  </replication-channel>
</swiftlet>

Configuration Reference

The top-level entity in routerconfig.xml is <swiftlet name="sys$hacontroller">.

<swiftlet name="sys$hacontroller"> Properties

These properties are attributes of the <swiftlet name="sys$hacontroller"> entity.

Parameter	Type	Default	Mandatory	Reboot Required	Description
preferred-active	Boolean	false	No	No	States whether this router is the preferred Active Instance
negotiation-timeout	Long	1800000	No	No	Time after which a Negotation must be initiated (min: 1000)
split-brain-instance-action	String	stop	No	No	Action taken on this Instance when a Split Brain is detected (choices: keep, stop, backup-and-standby)

<swiftlet name="sys$hacontroller" preferred-active="false" negotiation-timeout="1800000" split-brain-instance-action="stop"/>

<spool> Entity

Spool Settings

This is a fixed child entity of <swiftlet name="sys$hacontroller">.

Parameter	Type	Default	Mandatory	Reboot Required	Description
directory	String	./	No	Yes	Spool Directory
max-cache-size	Integer	5120	No	Yes	Specifies the size in KB to held in memory. (min: 1024)

<swiftlet name="sys$hacontroller">
  <spool directory="..." max-cache-size="..."/>
</swiftlet>

<configuration-controller> Entity

Tracks and controls configuration replication

This is a fixed child entity of <swiftlet name="sys$hacontroller">.

<swiftlet name="sys$hacontroller">
  <configuration-controller/>
</swiftlet>

<property-substitutions> in <configuration-controller>

Property Substitutions

Each <property-substitution> entry is identified by its name attribute (the Property Substitution).

Parameter	Type	Default	Mandatory	Reboot Required	Description
substitute-with	String	—	No	No	Substitute the Property Value with this Value

<swiftlet name="sys$hacontroller">
  <configuration-controller>
    <property-substitutions>
      <property-substitution name="..."/>
    </property-substitutions>
  </configuration-controller>
</swiftlet>

<replication-excludes> in <configuration-controller>

Replication Excludes

Each <replication-exclude> entry is identified by its name attribute (the Replication Exclude).

<swiftlet name="sys$hacontroller">
  <configuration-controller>
    <replication-excludes>
      <replication-exclude name="..."/>
    </replication-excludes>
  </configuration-controller>
</swiftlet>

<replication-channel> Entity

Replication Channel Listeners and Connectors

This is a fixed child entity of <swiftlet name="sys$hacontroller">.

Parameter	Type	Default	Mandatory	Reboot Required	Description
heartbeat-interval	Long	2000	No	No	Interval for sending Heart Beat Messages (min: 100)
heartbeat-missing-threshold	Integer	10	No	No	Closes Replication Connections after missing this number of Heart Beat Messages (min: 1)
max-packet-size	Integer	1024	No	No	Maximum Packet Size (KB) (min: 1)

<swiftlet name="sys$hacontroller">
  <replication-channel heartbeat-interval="..." heartbeat-missing-threshold="..." max-packet-size="..."/>
</swiftlet>

<listeners> in <replication-channel>

Listener Definitions

Each <listener> entry is identified by its name attribute (the Listener).

Parameter	Type	Default	Mandatory	Reboot Required	Description
bindaddress	String	—	No	No	Listener Bind IP Address
port	Integer	—	Yes	No	Listener Port
use-tcp-no-delay	Boolean	true	No	No	Use Tcp No Delay
router-input-buffer-size	Integer	1048576	No	No	Router Network Input Buffer Size (min: 65536)
router-input-extend-size	Integer	1048576	No	No	Router Network Input Extend Size (min: 65536)
router-output-buffer-size	Integer	131072	No	No	Router Network Output Buffer Size (min: 1024)
router-output-extend-size	Integer	131072	No	No	Router Network Output Extend Size (min: 1024)

<swiftlet name="sys$hacontroller">
  <replication-channel>
    <listeners>
      <listener name="..." port="..."/>
    </listeners>
  </replication-channel>
</swiftlet>

<connectors> in <replication-channel>

Connector Definitions

Each <connector> entry is identified by its name attribute (the Connector).

Parameter	Type	Default	Mandatory	Reboot Required	Description
hostname	String	—	Yes	No	Remote Hostname
port	Integer	—	Yes	No	Remote Port
use-tcp-no-delay	Boolean	true	No	No	Use Tcp No Delay
retry-time	Long	1000	No	No	Retry Time (min: 100)
router-input-buffer-size	Integer	1048576	No	No	Router Network Input Buffer Size (min: 65536)
router-input-extend-size	Integer	1048576	No	No	Router Network Input Extend Size (min: 65536)
router-output-buffer-size	Integer	131072	No	No	Router Network Output Buffer Size (min: 1024)
router-output-extend-size	Integer	131072	No	No	Router Network Output Extend Size (min: 1024)

<swiftlet name="sys$hacontroller">
  <replication-channel>
    <connectors>
      <connector name="..." hostname="..." port="..."/>
    </connectors>
  </replication-channel>
</swiftlet>

<replication-tunnels> in <swiftlet name="sys$hacontroller">

Replication Tunnels

Each <replication-tunnel> entry is identified by its name attribute (the Replication Tunnel).

Parameter	Type	Default	Mandatory	Reboot Required	Description
tunnel-address	Integer	—	No	No	Tunnel Address (min: 0)
versions	String	—	No	No	Supported Tunnel Protocol Versions

<swiftlet name="sys$hacontroller">
  <replication-tunnels>
    <replication-tunnel name="..."/>
  </replication-tunnels>
</swiftlet>

Related Documents

• HA Overview