FileCache Swiftlet

Overview

The FileCache Swiftlet provides managed, persistent file caches within the SwiftMQ router, enabling efficient storage, retrieval, and transfer of files via message-based protocols. It supports chunked file uploads and downloads, cache size and file limits, expiration, and advanced management through CLI commands and scheduled jobs. Each cache is independently configurable and can be monitored for usage and activity.

Features

Managed File Caches

The FileCache Swiftlet allows administrators to define one or more named file caches, each backed by a local directory. Each cache can be enabled or disabled individually and has configurable limits for maximum cache size (in MB), maximum file size, maximum number of files, and default file expiration. Files are stored persistently and tracked via registry queues. The cache ensures consistency between the registry and the actual files on disk, cleaning up orphaned files or registry entries as needed.

Chunked File Transfer

File uploads and downloads are performed in configurable chunk sizes (in KB), supporting efficient and resumable transfers. The protocol ensures data integrity by using digests and supports concurrent uploads/downloads with per-session tracking.

File Expiration and Cleanup

Files can have a default expiration time, after which they are automatically purged from the cache. Cleanup can be performed periodically per cache or via scheduled jobs. Expired files are deleted unless they are currently being downloaded, in which case deletion is deferred until downloads complete.

Access Control and Security

Files can be marked as private and/or protected with a password digest. Download and deletion requests for such files require the correct password. Additionally, files can be configured to be deleted after a specified number of downloads.

Configuration Example:

<swiftlet name="sys$filecache">
  <caches>
    <cache name="images" enabled="true" directory="/var/swiftmq/filecache/images" max-cache-size-mb="500" max-file-size-mb="50" max-number-files="1000" default-expiration="604800000" chunk-size-kb="64" inactivity-timeout="120000" cleanup-interval="300000"/>
  </caches>
</swiftlet>

Usage Monitoring and Statistics

The Swiftlet maintains detailed usage statistics for each active cache, including the number of files, total cache size, active uploads, and active downloads. For each upload or download in progress, detailed metrics such as file name, file size, transferred size, transfer speed (MB/sec), and percent complete are tracked. These statistics are updated at a configurable interval and are available via the management interface.

Configuration Example:

<swiftlet name="sys$filecache" collect-interval="2000"/>

CLI Commands for File Management

The Swiftlet registers CLI commands for direct file cache management. The listfile command lists files in a cache with support for selectors and paginated output, while the delfile command deletes individual files or all files from a cache. These commands provide administrators with granular control over cache contents without needing to access the filesystem directly.

Selectors and Filtering

The listfile command supports message selectors, allowing administrators to filter files based on properties such as file name, size, or expiration. This enables targeted management and reporting.

Scheduled Jobs for Automation

The FileCache Swiftlet integrates with the scheduler to provide job factories for automated cache maintenance. The available jobs are: - File Purger: Purges files from a cache based on a message selector. - File Copier: Copies files from a source cache to a target cache, optionally deleting after transfer. - File Cache Cleanup: Cleans up expired files in caches matching a SQL-LIKE predicate. Each job exposes parameters for fine-grained control and can be scheduled or run on demand.

Job Parameter Verification

Job parameters such as cache names and selectors are verified for validity before execution, preventing misconfiguration and runtime errors.

Protocol-based File Transfer Sessions

File uploads and downloads are handled via protocol sessions that use temporary queues for communication. Each session is tracked and managed independently, with inactivity timeouts and usage collection. The protocol supports version negotiation and ensures compatibility with multiple client versions.

Session Inactivity Handling

Sessions are automatically closed if no activity is detected within the configured inactivity timeout, freeing up resources and ensuring cache integrity.

Internal Queue Naming

sys$fcreg-<cacheName> — Registry queue for storing file metadata for each cache.
sys$filecache.request — Request queue for file cache controller operations.
sys$filecache.session — Session queue for managing individual file transfer sessions.

Configuration Guide

Defining a File Cache with Expiration and Size Limits

Use this scenario to create a file cache named images with a 500 MB size limit, 50 MB per file, a maximum of 1000 files, and a default expiration of 7 days.

Add a cache entity under the sys$filecache swiftlet.
Set the enabled attribute to true.
Specify the directory where files will be stored.
Configure max-cache-size-mb, max-file-size-mb, max-number-files, and default-expiration as needed.

<swiftlet name="sys$filecache">
  <caches>
    <cache name="images" enabled="true" directory="/var/swiftmq/filecache/images" max-cache-size-mb="500" max-file-size-mb="50" max-number-files="1000" default-expiration="604800000"/>
  </caches>
</swiftlet>

Adjusting Usage Collection Interval

Increase the interval at which usage statistics are collected and updated, reducing monitoring overhead.

Set the collect-interval attribute on the sys$filecache swiftlet to the desired value in milliseconds.

<swiftlet name="sys$filecache" collect-interval="5000"/>

Enabling Periodic Cleanup of Expired Files

Configure a cache to automatically clean up expired files every 10 minutes.

Set the cleanup-interval attribute on the cache entity to 600000 (milliseconds).

<swiftlet name="sys$filecache">
  <caches>
    <cache name="archive" enabled="true" directory="/var/swiftmq/filecache/archive" cleanup-interval="600000"/>
  </caches>
</swiftlet>

CLI Commands

delfile

Description: Delete a single or all Files from a Cache.

Usage: delfile <cache> <file-key>|*

listfile

Description: List Cache Files

Usage: listfile <cache> <startidx> (<stopidx>|*) [<selector>]

Scheduler Jobs

File Cache Cleanup

Description: Cleanup expired Files

File Copier

Description: Copies Files from a Source Cache to a Target Cache

File Purger

Description: Purges files from a File Cache

Configuration Reference

The top-level entity in routerconfig.xml is <swiftlet name="sys$filecache">.

`<swiftlet name="sys$filecache">` Properties

These properties are attributes of the <swiftlet name="sys$filecache"> entity.

Parameter	Type	Default	Mandatory	Reboot Required	Description
`collect-interval`	Long	`1000`	No	No	Collect Interval (ms) for the Usage Section

<swiftlet name="sys$filecache" collect-interval="1000"/>

`<caches>` in `<swiftlet name="sys$filecache">`

Caches

Each <cache> entry is identified by its name attribute (the Cache).

Parameter	Type	Default	Mandatory	Reboot Required	Description
`enabled`	Boolean	`false`	No	No	Enables/Disables this Cache
`directory`	String	—	Yes	No	Local Directory to store Files
`max-cache-size-mb`	Integer	`-1`	No	No	Maximum Size of all Files in this Cache
`chunk-size-kb`	Integer	`10`	No	No	Size of the Chunks for the File Transfer (min: 1)
`max-file-size-mb`	Integer	`-1`	No	No	Maximum Size of a single File in this Cache
`max-number-files`	Integer	`-1`	No	No	Maximum Number Files in this Cache
`default-expiration`	Long	`-1`	No	No	Default Exiration of Files in this Cache
`inactivity-timeout`	Long	`60000`	No	No	Inactivity Timeout after which a File Transfer is stopped
`cleanup-interval`	Long	`120000`	No	No	Interval to Cleanup expired Files of this Cache

<swiftlet name="sys$filecache">
  <caches>
    <cache name="..." directory="..."/>
  </caches>
</swiftlet>

Changelog

13.2.0 (2025-11-03)

SessionImpl: removed visit()