Store Swiftlet

Overview

The Store Swiftlet provides the persistent and non-persistent message storage for all queues in the SwiftMQ router. It manages the underlying page database (page.db), transaction log, swap files for non-persistent queues, and durable subscriber state. The Store Swiftlet ensures message durability, transactional integrity, and supports backup and maintenance operations.

Features

Persistent Store for Queues

The Store Swiftlet maintains a page-based database (page.db) to persistently store messages for all queues configured with persistent delivery. Each queue is mapped to a dedicated index within the page database, ensuring fast access and isolation. The persistent store supports transactional operations, including commit and rollback, to guarantee message durability and atomicity. The underlying storage is optimized for high throughput and crash recovery, with a root index mapping queue names to their respective storage areas. During startup, the Store Swiftlet performs recovery by replaying the transaction log and running a consistency check on the page database.

Page Database Management

The page database is organized into fixed-size pages, with configurable page size and initial number of pages. The Store Swiftlet can compact or resize the database on startup, based on configuration and scan results. The consistency of the database is checked during startup, and any inconsistencies are corrected if possible.

Transactional Message Operations

All message operations (insert, remove, update delivery count) are performed within transactions. The Store Swiftlet provides read and write transactions, supporting both local and XA (2PC) transactions. Transactions are logged to the transaction log before being committed to the page database, ensuring durability and atomicity.

Configuration Example:

<swiftlet name="sys$store">
  <database path="data/page.db" initial-db-size-pages="51200"/>
</swiftlet>

Transaction Log and Recovery

The Store Swiftlet maintains a transaction log (transaction.log) to record all changes to the persistent store. This log enables crash recovery by replaying committed transactions and rolling back incomplete ones. The log is periodically checkpointed, flushing all changes to disk and truncating the log to reduce disk usage. The checkpoint interval is determined by the configured checkpoint size. The transaction log can be configured to force a disk sync on each write for maximum durability, at the cost of performance.

XA Transaction Support

The Store Swiftlet supports distributed (XA/2PC) transactions by maintaining a prepared log (xa.log or in-queue) that records the state of prepared transactions. This ensures that prepared transactions can be recovered and completed after a crash.

Consistency Checking

During recovery, the Store Swiftlet performs a consistency check on the persistent store. If inconsistencies are found, the store attempts to correct them, possibly resulting in data loss. Administrators are advised to enable force-sync for the transaction log to minimize the risk of inconsistency.

Configuration Example:

<swiftlet name="sys$store">
  <transaction-log path="data/transaction.log" checkpoint-size="209715200" force-sync="true"/>
</swiftlet>

Cache Management

To optimize performance, the Store Swiftlet uses an in-memory cache for database pages. The cache size is configurable with minimum and maximum page counts. The cache manager employs an LRU eviction policy for unpinned pages and flushes dirty pages to disk during checkpoints or when the cache is full. The cache ensures that frequently accessed data is kept in memory for fast operations, while maintaining consistency with the persistent store.

Dynamic Cache Sizing

The minimum and maximum cache sizes can be configured independently. The Store Swiftlet enforces that the minimum size is less than or equal to the maximum, and will reject invalid configurations.

Configuration Example:

<swiftlet name="sys$store">
  <cache min-size="2048" max-size="4096"/>
</swiftlet>

Non-Persistent Store (Swap Files)

For queues configured with non-persistent delivery, the Store Swiftlet uses swap files to temporarily store messages that do not fit in memory. Swap files are created in a configurable directory, with a configurable roll-over size to limit file growth. When a swap file reaches its roll-over size, a new file is created. Swap files are automatically deleted when the queue is closed or the router shuts down.

Swap File Management

Swap files are managed per queue and are automatically cleaned up. The roll-over size determines the maximum size of each swap file before a new one is created.

Configuration Example:

<swiftlet name="sys$store">
  <swap path="data/swap" roll-over-size="20971520"/>
</swiftlet>

Durable Subscriber Store

The Store Swiftlet manages the state of durable topic subscribers by storing their subscription information (client ID, durable name, topic, selector, noLocal flag) in files within a configurable directory. Each durable subscription is mapped to a file named using the pattern $.durable. This ensures that durable subscriptions survive router restarts and can be recovered reliably.

Durable Subscription File Naming

Durable subscription state is stored in files named $.durable in the configured directory. These files are managed automatically by the Store Swiftlet.

Configuration Example:

<swiftlet name="sys$store">
  <durable-subscriber path="data/durables"/>
</swiftlet>

Backup and Restore

The Store Swiftlet provides an integrated backup mechanism to create consistent snapshots of the persistent store, including the page database, transaction log, and durable subscriber files. Backups are stored as save sets in a configurable directory, with automatic retention of a configurable number of backup generations. Backups are performed in a coordinated manner, ensuring a checkpoint is taken before files are copied for consistency. The backup process can be triggered manually via the CLI or scheduled as a job.

Backup Path and Generations

The backup path specifies where save sets are stored. The number of generations determines how many backup sets are retained; older backups are automatically deleted.

Backup Job Integration

Backup operations can be triggered manually or scheduled via the sys$scheduler Swiftlet. The Store Swiftlet registers a 'Backup' job for this purpose.

Configuration Example:

<swiftlet name="sys$store">
  <backup path="backups" keep-generations="5"/>
</swiftlet>

Maintenance Operations: Shrink and Scan

The Store Swiftlet supports maintenance operations to optimize storage and recommend configuration changes. The shrink operation compacts the page database by reclaiming unused pages, reducing disk usage. The scan operation analyzes message sizes in the database and recommends an optimal page size for performance. Both operations are coordinated with checkpoints to ensure consistency and can be triggered manually or scheduled as jobs.

Shrink Job

The 'Shrink' job compacts the page database, removing unused pages and reducing file size. This can be triggered manually or scheduled.

Scan Job

The 'Scan' job analyzes the distribution of message sizes in the database and updates the recommended page size. This helps administrators tune the store for optimal performance.

Configuration Example:

<swiftlet name="sys$store"/>

Automatic Store Conversion and Compaction

On startup, the Store Swiftlet can automatically resize or compact the page database if the recommended page size differs from the current size or if compaction is enabled. This process involves spooling messages out, resizing or compacting the database, and spooling messages back in, ensuring minimal downtime and optimal storage layout.

Page Size Recommendation

The recommended page size is updated based on scan results and can be used to trigger automatic resizing on startup if configured.

Configuration Example:

<swiftlet name="sys$store">
  <database perform-resize-on-startup="true"/>
</swiftlet>

Internal Queue Naming

<clientId>$<durableName> — Queue name pattern for durable topic subscriptions; state stored as $.durable files.
sys$prepared — Internal queue used for in-memory prepared transaction log (PreparedLogQueue) when XA transactions are enabled.

Configuration Guide

Configuring Persistent Store with Custom Page Size and Cache

Use this scenario to optimize the persistent store for high-throughput workloads by increasing the page size and cache size. This reduces disk I/O and improves performance for large messages.

Set the desired page size and initial database size in the database entity.
Increase the cache min-size and max-size for more in-memory pages.
Restart the router for changes to take effect.

<swiftlet name="sys$store">
  <database page-size-current="4096" initial-db-size-pages="65536"/>
  <cache min-size="4096" max-size="8192"/>
</swiftlet>

Enabling Force Sync on Transaction Log for Maximum Durability

Enable force-sync to ensure that every transaction log write is flushed to disk, minimizing the risk of data loss in case of a crash. This is recommended for critical production environments where durability is paramount.

Set force-sync to true in the transaction-log entity.
Restart the router to apply the setting.

<swiftlet name="sys$store">
  <transaction-log force-sync="true"/>
</swiftlet>

Configuring and Scheduling Automatic Backups

Set up the Store Swiftlet to keep regular backups of the persistent store, retaining a specific number of backup generations. This ensures recoverability in case of hardware failure or data corruption.

Set the backup path and keep-generations in the backup entity.
Schedule the 'Backup' job via the sys$scheduler Swiftlet if desired.

<swiftlet name="sys$store">
  <backup path="/mnt/backups/swiftmq" keep-generations="7"/>
</swiftlet>

Optimizing Swap File Usage for Non-Persistent Queues

Increase the swap file roll-over size to accommodate larger volumes of non-persistent messages without frequent file creation. Useful for high-throughput, non-persistent workloads.

Set the roll-over-size property in the swap entity to the desired value.
Restart the router to apply the change.

<swiftlet name="sys$store">
  <swap roll-over-size="52428800"/>
</swiftlet>

CLI Commands

backup

Description: Perform Backup Now

Usage: backup

shrink

Description: Perform Shrink Now

Usage: shrink

scan

Description: Scan page.db

Usage: scan

Scheduler Jobs

Backup

Description: Performs a Backup of the persistent Store

Scan

Description: Scans page.db and recommends a new page size

Shrink

Description: Performs a Shrink of the page.db