Replicator Extension Swiftlet

Overview

The Replicator Extension Swiftlet provides file replication capabilities between SwiftMQ routers. It enables the definition of replication sources and sinks, allowing files from specified directories to be synchronized across routers using a file cache and messaging topics. The Swiftlet supports monitoring, event notification, and flexible configuration for both source and sink replication endpoints.

Features

Replication Sources

Replication sources define directories on the local file system whose contents are to be replicated to other routers. Each source is associated with a file cache (by router name and cache name), a control file for state tracking, and an update topic for change notifications. The source scans its directory (optionally recursively), detects new, modified, or deleted files, and synchronizes these changes to the file cache. Changes are published as update messages to the configured topic, which are then consumed by replication sinks.

Directory Scanning and Change Detection

Each replication source periodically scans its configured directory. It maintains a registry of known files (persisted in the control file) and detects new, modified, or deleted files. Recursive scanning can be enabled or disabled per source.

File Cache Synchronization

When new or modified files are detected, they are pushed to the file cache using the configured router and cache name. Deleted files are removed from the cache. The source sends update messages to the update topic to inform sinks of changes.

Update Topic Publishing

For every file change, the source sends a MapMessage to the update topic, containing control information (such as NEW or DELETE), a sequence number, and a link to the file in the cache. This ensures that sinks can track and apply changes in order.

Configuration Example:

<swiftlet name="xt$replicator">
  <sources>
    <source name="mySource" filecache-routername="router2" filecache-name="cacheA" controlfile="/var/replicator/mySource.xml" directory="/data/export" update-topic="replication.updates"/>
  </sources>
</swiftlet>

Replication Sinks

Replication sinks receive file change notifications from sources and synchronize a local directory to match the file cache. Each sink subscribes to the update topic, retrieves new or deleted files from the cache, and maintains a local registry for state tracking. Sinks can be enabled or disabled, and each maintains its own control file for tracking replicated files.

Update Topic Consumption

The sink establishes a JMS consumer on the update topic and processes incoming update messages. It applies file additions, modifications, or deletions as indicated by the control property in each message.

Local Directory Synchronization

Upon receiving a NEW control message, the sink retrieves the file from the cache and stores it in the configured directory. For DELETE messages, the sink removes the corresponding file and cleans up empty directories as needed.

State Tracking with Control File

The sink uses a control file to persist the state of replicated files. This allows for recovery and resynchronization in case of failures or restarts.

Configuration Example:

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="mySink" filecache-routername="router2" filecache-name="cacheA" controlfile="/var/replicator/mySink.xml" directory="/data/import" update-topic="replication.updates"/>
  </sinks>
</swiftlet>

Event Sinks and Notifications

Replication sinks can be configured with event sinks to emit notifications about file creation and deletion events. Each event sink specifies a destination (queue or topic), delivery mode (persistent or non_persistent), and which events to emit (file created, file deleted). When a file is created or deleted in the sink's directory, an event message is sent to the configured destination, containing details about the event in XML format.

Destination Type and Name

Each event sink can target either a queue or a topic, specified by the destination-type and destination-name properties. If the destination does not exist, it is created automatically.

Event Selection and Delivery Mode

Event sinks can be configured to emit only file created events, only file deleted events, or both, using the event-file-created and event-file-deleted properties. The delivery-mode property controls whether event messages are sent as persistent or non_persistent JMS messages.

Event Message Format

Event messages are sent as text messages containing XML describing the event, including the event type, sink name, sink path, file path, filename, and modification time.

Configuration Example:

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="mySink" filecache-routername="router2" filecache-name="cacheA" controlfile="/var/replicator/mySink.xml" directory="/data/import" update-topic="replication.updates">
      <event-sinks>
        <event-sink name="notifyQueue" destination-name="notify" destination-type="queue" delivery-mode="persistent" event-file-created="true" event-file-deleted="false"/>
      </event-sinks>
    </sink>
  </sinks>
</swiftlet>

Job Integration for Automation

The Replicator Swiftlet integrates with the Scheduler Swiftlet, providing job factories for both sources and sinks. This allows replication sources and sinks to be started and stopped automatically according to scheduled jobs. Jobs can be defined to activate or deactivate specific sources or sinks by name.

Sink Job Factory

The SinkJobFactory provides a job type named Sink, which can enable or disable a replication sink by name.

Source Job Factory

The SourceJobFactory provides a job type named Source, which can enable or disable a replication source by name.

Configuration Guide

Replicating a Directory from One Router to Another

Use this scenario to synchronize a directory from a source router to a sink router. Configure a replication source on the source router and a corresponding sink on the sink router, both referencing the same file cache and update topic.

On the source router, define a replication source with the local directory to export, the file cache router and name, a control file path, and the update topic.
On the sink router, define a replication sink with the local directory to import into, the same file cache router and name, a control file path, and the same update topic.
Enable both the source and sink.

<swiftlet name="xt$replicator">
  <sources>
    <source name="exportSource" filecache-routername="router2" filecache-name="sharedCache" controlfile="/var/replicator/exportSource.xml" directory="/srv/export" update-topic="replication.updates" enabled="true"/>
  </sources>
  <sinks>
    <sink name="importSink" filecache-routername="router2" filecache-name="sharedCache" controlfile="/var/replicator/importSink.xml" directory="/srv/import" update-topic="replication.updates" enabled="true"/>
  </sinks>
</swiftlet>

Notifying Applications of File Events

Configure an event sink on a replication sink to notify an application via a queue when files are created in the import directory. This is useful for triggering downstream processing.

Define a replication sink as usual.
Within the sink, add an event sink with destination-type set to queue, destination-name set to the application's queue, delivery-mode set to persistent, and event-file-created set to true.
Ensure the application consumes messages from the specified queue.

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="importSink" filecache-routername="router2" filecache-name="sharedCache" controlfile="/var/replicator/importSink.xml" directory="/srv/import" update-topic="replication.updates">
      <event-sinks>
        <event-sink name="notifyApp" destination-name="appNotify" destination-type="queue" delivery-mode="persistent" event-file-created="true" event-file-deleted="false"/>
      </event-sinks>
    </sink>
  </sinks>
</swiftlet>

Scheduler Jobs

Sink

Description: Activates an Replication Sink

Source

Description: Activates an Replication Source

Configuration Reference

The top-level entity in routerconfig.xml is <swiftlet name="xt$replicator">.

`<sources>` in `<swiftlet name="xt$replicator">`

Replication Source Definitions

Each <source> entry is identified by its name attribute (the Replication Source).

Parameter	Type	Default	Mandatory	Reboot Required	Description
`filecache-routername`	String	—	Yes	No	File Cache Router Name
`filecache-name`	String	—	Yes	No	File Cache Name
`ivm-username`	String	—	No	No	IntraVM Connection User Name
`ivm-password`	String	—	No	No	IntraVM Connection Password
`enabled`	Boolean	`false`	No	No	Enabled
`controlfile`	String	—	Yes	No	Control File
`directory`	String	—	Yes	No	Directory
`recursive`	Boolean	`true`	No	No	Recursive
`check-interval`	Long	`60000`	Yes	No	Check Interval (ms) (min: 1000)
`update-topic`	String	—	Yes	No	Update Topic

<swiftlet name="xt$replicator">
  <sources>
    <source name="..." filecache-routername="..." filecache-name="..." controlfile="..." directory="..." check-interval="..." update-topic="..."/>
  </sources>
</swiftlet>

`<sinks>` in `<swiftlet name="xt$replicator">`

Replication Sink Definitions

Each <sink> entry is identified by its name attribute (the Replication Sink).

Parameter	Type	Default	Mandatory	Reboot Required	Description
`filecache-routername`	String	—	Yes	No	File Cache Router Name
`filecache-name`	String	—	Yes	No	File Cache Name
`ivm-username`	String	—	No	No	IntraVM Connection User Name
`ivm-password`	String	—	No	No	IntraVM Connection Password
`enabled`	Boolean	`false`	No	No	Enabled
`controlfile`	String	—	Yes	No	Control File
`directory`	String	—	Yes	No	Directory
`check-interval`	Long	`60000`	Yes	No	Check Interval (ms) (min: 1000)
`update-topic`	String	—	Yes	No	Update Topic

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="..." filecache-routername="..." filecache-name="..." controlfile="..." directory="..." check-interval="..." update-topic="..."/>
  </sinks>
</swiftlet>

`<event-sinks>` in `<sinks>`

Event Sinks

Each <event-sink> entry is identified by its name attribute (the Event Sink).

Parameter	Type	Default	Mandatory	Reboot Required	Description
`destination-name`	String	—	Yes	No	Name of the Queue/Topic
`destination-type`	String	`queue`	Yes	No	Destination Type (choices: `queue`, `topic`)
`delivery-mode`	String	`non_persistent`	Yes	No	Delivery Mode (choices: `non_persistent`, `persistent`)
`event-file-created`	Boolean	`true`	No	No	Event File Created
`event-file-deleted`	Boolean	`true`	No	No	Event File Deleted

<swiftlet name="xt$replicator">
  <sinks>
    <sink name="...">
      <event-sinks>
        <event-sink name="..." destination-name="..." destination-type="..." delivery-mode="..."/>
      </event-sinks>
    </sink>
  </sinks>
</swiftlet>

Replicator Extension Swiftlet

Overview

Features

Replication Sources

Directory Scanning and Change Detection

File Cache Synchronization

Update Topic Publishing

Replication Sinks

Update Topic Consumption

Local Directory Synchronization

State Tracking with Control File

Event Sinks and Notifications

Destination Type and Name

Event Selection and Delivery Mode

Event Message Format

Job Integration for Automation

Sink Job Factory

Source Job Factory

Configuration Guide

Replicating a Directory from One Router to Another

Notifying Applications of File Events

Scheduler Jobs

Sink

Source

Configuration Reference

<sources> in <swiftlet name="xt$replicator">

<sinks> in <swiftlet name="xt$replicator">

<event-sinks> in <sinks>

`<sources>` in `<swiftlet name="xt$replicator">`

`<sinks>` in `<swiftlet name="xt$replicator">`

`<event-sinks>` in `<sinks>`