FileCache Swiftlet

Overview

The FileCache Swiftlet provides managed, persistent file caching and transfer capabilities within the SwiftMQ router. It allows administrators and applications to define, monitor, and control multiple local file caches, supporting secure, chunked file uploads and downloads, as well as automated cleanup and purging of expired or unwanted files.

Features

Managed File Caches

The FileCache Swiftlet allows you to define multiple named file caches, each corresponding to a local directory on the router. Each cache can be enabled or disabled, and its storage location, size limits, and operational parameters are individually configurable. Files are stored persistently on disk, and each cache maintains its own registry of files, including metadata such as file key, name, size, digest, expiration, and custom properties. The cache ensures consistency between the file registry and the actual files on disk, automatically cleaning up orphaned or expired files as needed.

Cache Properties

Each cache entity supports properties such as enabled, directory, max-cache-size-mb, chunk-size-kb, max-file-size-mb, max-number-files, default-expiration, inactivity-timeout, and cleanup-interval. These control whether the cache is active, where files are stored, the maximum allowed file and cache sizes, chunk size for transfers, automatic expiration, and cleanup behavior.

File Registry

Each cache maintains a registry of files using a dedicated queue (sys$fcreg-<cacheName>). File metadata is stored as message properties, including file key, name, user, size, digest, expiration, download count, and custom properties. The registry is kept consistent with the file system, and supports efficient querying and management.

Configuration Example:

<swiftlet name="sys$filecache">
  <caches>
    <cache name="mycache" enabled="true" directory="/var/swiftmq/filecache/mycache" max-cache-size-mb="1000" chunk-size-kb="64"/>
  </caches>
</swiftlet>

Chunked and Secure File Transfer

File uploads and downloads are performed in configurable-sized chunks, supporting efficient and reliable transfer of large files. The protocol supports integrity verification via digests, optional password protection, download count limits, and private file flags. Each transfer is tracked as a session, with inactivity timeouts and detailed usage statistics. Uploads and downloads are managed via dedicated temporary queues, and sessions are automatically closed on inactivity or completion.

Session Management

Each file transfer (upload or download) is handled as a session, with its own temporary queue and event loop. Sessions are monitored for inactivity using the inactivity-timeout property, and are automatically closed if idle for too long. Active sessions are tracked and usage statistics are updated in real time.

File Integrity and Access Control

Files can be protected with digests (for integrity verification) and optional passwords (for access control). Files may also be marked as private, and can be configured to be deleted after a certain number of downloads. All access and deletion operations validate these constraints.

Configuration Example:

<swiftlet name="sys$filecache">
  <caches>
    <cache name="securecache" enabled="true" directory="/data/filecache/secure" chunk-size-kb="32" inactivity-timeout="120000"/>
  </caches>
</swiftlet>

Automated Cleanup and Expiration

The FileCache Swiftlet supports automated cleanup of expired files within each cache. The cleanup-interval property controls how often expired files are checked and removed. Files can be assigned a default expiration time, after which they are eligible for deletion. Cleanup can also be triggered manually via jobs or management commands.

Default Expiration

The default-expiration property allows administrators to set a default lifetime (in milliseconds) for files stored in a cache. Files without an explicit expiration will be assigned this value.

Cleanup Jobs

The Swiftlet registers a 'File Cache Cleanup' job with the scheduler, allowing administrators to schedule or trigger cleanup of expired files across caches, using SQL-LIKE predicates to select caches.

Configuration Example:

<swiftlet name="sys$filecache">
  <caches>
    <cache name="tempfiles" enabled="true" directory="/tmp/filecache" default-expiration="86400000" cleanup-interval="60000"/>
  </caches>
</swiftlet>

Administrative Commands and Jobs

The Swiftlet provides management commands and scheduler jobs for advanced file cache operations. The listfile command lists files in a cache with optional selectors and pagination. The delfile command deletes specific files or all files from a cache. Scheduler jobs include 'File Purger' (purge files by selector), 'File Copier' (copy files between caches/routers), and 'File Cache Cleanup' (remove expired files). Jobs support parameter validation and can be scheduled or run on demand.

listfile Command

Lists files in a specified cache, supporting pagination and optional message selectors for filtering. Output includes file metadata and custom properties.

delfile Command

Deletes a single file (by file key) or all files (using *) from a specified cache. Handles password-protected files and active downloads appropriately.

Scheduler Jobs

Jobs such as 'File Purger', 'File Copier', and 'File Cache Cleanup' are registered under the 'File Cache' job group, supporting flexible, automated management of file caches. Parameters include cache names, selectors, target routers/caches, and delete-after-transfer options.

Usage Monitoring and Statistics

The FileCache Swiftlet maintains detailed usage statistics for each cache and active transfer session. The usage entity list tracks the number of files, total cache size, active uploads/downloads, and per-session metrics such as file name, size, transferred amount, transfer speed, and percent complete. These statistics are updated at the interval specified by collect-interval and are visible in the management UI.

Configuration Example:

<swiftlet name="sys$filecache" collect-interval="2000"/>

Internal Queue Naming

• sys$fcreg-<cacheName> — Queue used for the file registry of each cache, storing file metadata as persistent messages.
• sys$filecache.request — Queue used for processing file cache controller requests.
• sys$filecache.session — Temporary queue used for each active file transfer session (upload/download).

Configuration Guide

Defining a File Cache with Size and Expiration Limits

Use this scenario to create a file cache that automatically limits the total cache size, the size of individual files, and sets a default expiration for all files. This is useful for managing disk usage and ensuring old files are purged automatically.

1. Define a new cache entity under the caches list.
2. Set enabled to true.
3. Specify the directory where files will be stored.
4. Set max-cache-size-mb and max-file-size-mb to enforce size limits.
5. Set default-expiration to specify how long files are retained by default.
6. Optionally, adjust cleanup-interval to control how often expired files are removed.

<swiftlet name="sys$filecache">
  <caches>
    <cache name="archive" enabled="true" directory="/var/swiftmq/filecache/archive" max-cache-size-mb="5000" max-file-size-mb="100" default-expiration="604800000" cleanup-interval="300000"/>
  </caches>
</swiftlet>

Enabling Real-Time Usage Monitoring

Increase the frequency of usage statistics collection for more granular monitoring of cache activity. This is useful for environments with high file transfer rates or where up-to-date monitoring is required.

1. Set the collect-interval attribute on the FileCache Swiftlet to a lower value (in milliseconds).
2. Monitor the usage section in the management UI for real-time updates on cache and transfer activity.

<swiftlet name="sys$filecache" collect-interval="500"/>

CLI Commands

delfile

Description: Delete a single or all Files from a Cache.

Usage: delfile <cache> <file-key>|*

listfile

Description: List Cache Files

Usage: listfile <cache> <startidx> (<stopidx>|*) [<selector>]

Scheduler Jobs

File Cache Cleanup

Description: Cleanup expired Files

File Copier

Description: Copies Files from a Source Cache to a Target Cache

File Purger

Description: Purges files from a File Cache

Configuration Reference

The top-level entity in routerconfig.xml is <swiftlet name="sys$filecache">.

<swiftlet name="sys$filecache"> Properties

These properties are attributes of the <swiftlet name="sys$filecache"> entity.

Parameter	Type	Default	Mandatory	Reboot Required	Description
collect-interval	Long	1000	No	No	Collect Interval (ms) for the Usage Section

<swiftlet name="sys$filecache" collect-interval="1000"/>

<caches> in <swiftlet name="sys$filecache">

Caches

Each <cache> entry is identified by its name attribute (the Cache).

Parameter	Type	Default	Mandatory	Reboot Required	Description
enabled	Boolean	false	No	No	Enables/Disables this Cache
directory	String	—	Yes	No	Local Directory to store Files
max-cache-size-mb	Integer	-1	No	No	Maximum Size of all Files in this Cache
chunk-size-kb	Integer	10	No	No	Size of the Chunks for the File Transfer (min: 1)
max-file-size-mb	Integer	-1	No	No	Maximum Size of a single File in this Cache
max-number-files	Integer	-1	No	No	Maximum Number Files in this Cache
default-expiration	Long	-1	No	No	Default Exiration of Files in this Cache
inactivity-timeout	Long	60000	No	No	Inactivity Timeout after which a File Transfer is stopped
cleanup-interval	Long	120000	No	No	Interval to Cleanup expired Files of this Cache

<swiftlet name="sys$filecache">
  <caches>
    <cache name="..." directory="..."/>
  </caches>
</swiftlet>

Changelog

13.2.0 (2025-11-03)

• SessionImpl: removed visit()