Replicator Extension Swiftlet

Overview

The Replicator Extension Swiftlet provides file-based replication between SwiftMQ routers by synchronizing directories across routers using a file cache and messaging. It manages replication sources and sinks, monitors file changes, and ensures that updates and deletions are propagated reliably. The Swiftlet is highly configurable and supports event notification for file operations via queues or topics.

Features

Replication Sources

Replication sources define directories on the local router that are monitored for file changes. Each source is associated with a file cache (on a remote or local router), a control file for tracking state, and an update topic for publishing file change events. The source periodically scans the configured directory (optionally recursively), detects new, modified, or deleted files, and synchronizes these changes to the file cache. It uses a registry to track the state of files and emits update messages to the configured topic, ensuring that sinks are notified of changes.

Directory Scanning and Change Detection

The source scans its configured directory at regular intervals, marking all known files as deleted before each scan. It then updates the registry with files found, marking new and modified files accordingly. Deleted files are detected by their absence and are removed from the file cache.

File Cache Synchronization

Files are pushed to the file cache using a Filetransfer mechanism. New and modified files are uploaded, and deleted files are removed from the cache. Each operation is tracked in a control file to ensure consistency and recoverability.

Update Topic Notification

For every file operation (new, modified, deleted), a message is sent to the configured update topic. This message includes a sequence number and control information to ensure sinks can process updates in order.

Configuration Example:

<swiftlet name="xt$replicator">
  <sources>
    <source name="source1" filecache-routername="router2" filecache-name="cache1" controlfile="/var/replication/source1.ctrl" directory="/data/source1" update-topic="replication.updates"/>
  </sources>
</swiftlet>

Replication Sinks

Replication sinks receive file updates from sources and apply them to a local directory. Each sink subscribes to an update topic, listens for file change notifications, and synchronizes its local directory with the file cache. Sinks maintain their own registry and control file to track state and ensure consistency. The sink can also emit events (such as file created or deleted) to user-defined queues or topics for monitoring or integration purposes.

Update Topic Consumption

The sink creates a JMS consumer on the configured update topic and processes incoming messages, which indicate new or deleted files. It ensures that updates are applied in sequence and synchronizes its local directory accordingly.

File Cache Synchronization

Upon receiving an update, the sink retrieves new files from the file cache or deletes files as instructed. The local directory is kept in sync with the remote source, and a registry tracks the current state.

Event Sinks for File Operations

Sinks can be configured with event sinks that emit notifications (as JMS messages) when files are created or deleted. These notifications can be sent to either a queue or a topic, and the delivery mode (persistent or non_persistent) can be configured. The event sinks allow integration with monitoring systems or custom workflows.

Configuration Example:

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="sink1" filecache-routername="router2" filecache-name="cache1" controlfile="/var/replication/sink1.ctrl" directory="/data/sink1" update-topic="replication.updates">
      <event-sinks>
        <event-sink name="notify" destination-name="replication.events" destination-type="topic" delivery-mode="persistent" event-file-created="true" event-file-deleted="true"/>
      </event-sinks>
    </sink>
  </sinks>
</swiftlet>

Event Sink Notification

Event sinks are sub-entities of replication sinks that emit JMS messages when files are created or deleted in the sink's directory. Each event sink can be configured to target a queue or topic, specify the delivery mode, and enable or disable notifications for file creation and deletion events. The event messages are XML-formatted and contain detailed information about the file operation, including the sink, path, and file metadata.

Destination Type and Delivery Mode

Event sinks can send notifications to either a queue or a topic. The delivery mode can be set to persistent or non_persistent, allowing control over message durability.

Selective Event Emission

Notifications for file creation and deletion can be enabled or disabled independently per event sink, providing fine-grained control over which events are reported.

Configuration Example:

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="sink1">
      <event-sinks>
        <event-sink name="notify" destination-name="replication.events" destination-type="queue" delivery-mode="non_persistent" event-file-created="true" event-file-deleted="false"/>
      </event-sinks>
    </sink>
  </sinks>
</swiftlet>

Job Integration for Scheduled Activation

The Swiftlet integrates with the scheduler to allow replication sources and sinks to be started or stopped via scheduled jobs. This enables automated control of replication based on time or external triggers. Job factories for sources and sinks are registered with the scheduler, and jobs can be defined to activate or deactivate replication as needed.

Job Factories and Parameters

Separate job factories are provided for sources and sinks. Each job requires the name of the source or sink to control, and can start or stop the corresponding entity.

Configuration Guide

Basic Directory Replication Between Routers

Use this scenario to synchronize a directory from one router to another, ensuring that all file changes are replicated in near real-time. This is useful for data distribution, backup, or high availability.

1. On the source router, define a replication source specifying the local directory, file cache router and name, control file, and update topic.
2. On the sink router, define a replication sink with the same file cache router and name, a local target directory, control file, and the same update topic.
3. Optionally, add event sinks to the sink for monitoring file operations.
4. Enable the source and sink.

<swiftlet name="xt$replicator">
  <sources>
    <source name="source1" filecache-routername="router2" filecache-name="cache1" controlfile="/var/replication/source1.ctrl" directory="/data/source1" update-topic="replication.updates"/>
  </sources>
  <sinks>
    <sink name="sink1" filecache-routername="router2" filecache-name="cache1" controlfile="/var/replication/sink1.ctrl" directory="/data/sink1" update-topic="replication.updates"/>
  </sinks>
</swiftlet>

Monitoring File Events with Event Sinks

Use this scenario to receive JMS notifications whenever files are created or deleted in a replicated directory. This is useful for integration with monitoring tools or triggering custom workflows.

1. Define a replication sink as usual.
2. Add one or more event sinks under the sink, specifying the destination (queue or topic), delivery mode, and which events to enable.
3. Configure your application to consume messages from the specified queue or topic.
4. Enable the sink.

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="sink1">
      <event-sinks>
        <event-sink name="notify" destination-name="replication.events" destination-type="topic" delivery-mode="persistent" event-file-created="true" event-file-deleted="true"/>
      </event-sinks>
    </sink>
  </sinks>
</swiftlet>

Scheduler Jobs

Sink

Description: Activates an Replication Sink

Source

Description: Activates an Replication Source

Configuration Reference

The top-level entity in routerconfig.xml is <swiftlet name="xt$replicator">.

<sources> in <swiftlet name="xt$replicator">

Replication Source Definitions

Each <source> entry is identified by its name attribute (the Replication Source).

Parameter	Type	Default	Mandatory	Reboot Required	Description
filecache-routername	String	—	Yes	No	File Cache Router Name
filecache-name	String	—	Yes	No	File Cache Name
ivm-username	String	—	No	No	IntraVM Connection User Name
ivm-password	String	—	No	No	IntraVM Connection Password
enabled	Boolean	false	No	No	Enabled
controlfile	String	—	Yes	No	Control File
directory	String	—	Yes	No	Directory
recursive	Boolean	true	No	No	Recursive
check-interval	Long	60000	Yes	No	Check Interval (ms) (min: 1000)
update-topic	String	—	Yes	No	Update Topic

<swiftlet name="xt$replicator">
  <sources>
    <source name="..." filecache-routername="..." filecache-name="..." controlfile="..." directory="..." check-interval="..." update-topic="..."/>
  </sources>
</swiftlet>

<sinks> in <swiftlet name="xt$replicator">

Replication Sink Definitions

Each <sink> entry is identified by its name attribute (the Replication Sink).

Parameter	Type	Default	Mandatory	Reboot Required	Description
filecache-routername	String	—	Yes	No	File Cache Router Name
filecache-name	String	—	Yes	No	File Cache Name
ivm-username	String	—	No	No	IntraVM Connection User Name
ivm-password	String	—	No	No	IntraVM Connection Password
enabled	Boolean	false	No	No	Enabled
controlfile	String	—	Yes	No	Control File
directory	String	—	Yes	No	Directory
check-interval	Long	60000	Yes	No	Check Interval (ms) (min: 1000)
update-topic	String	—	Yes	No	Update Topic

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="..." filecache-routername="..." filecache-name="..." controlfile="..." directory="..." check-interval="..." update-topic="..."/>
  </sinks>
</swiftlet>

<event-sinks> in <sinks>

Event Sinks

Each <event-sink> entry is identified by its name attribute (the Event Sink).

Parameter	Type	Default	Mandatory	Reboot Required	Description
destination-name	String	—	Yes	No	Name of the Queue/Topic
destination-type	String	queue	Yes	No	Destination Type (choices: queue, topic)
delivery-mode	String	non_persistent	Yes	No	Delivery Mode (choices: non_persistent, persistent)
event-file-created	Boolean	true	No	No	Event File Created
event-file-deleted	Boolean	true	No	No	Event File Deleted

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="...">
      <event-sinks>
        <event-sink name="..." destination-name="..." destination-type="..." delivery-mode="..."/>
      </event-sinks>
    </sink>
  </sinks>
</swiftlet>