Skip to main content

Cleanup Operations

The ImageMetadataCleanup function handles the lifecycle of expired image metadata through a two-phase process: enqueue and dequeue.

Phase 1: Timer Enqueue​

The timer-triggered function:

  1. Queries for all device IDs that have metadata older than the retention period
  2. Enqueues each device ID to the appropriate cleanup queue
  3. Separates by image type so each queue processor handles its own data

Phase 2: Queue Dequeue​

For each device ID dequeued, the cleanup process:

  1. Query the index repository for records matching the device ID and exceeding the retention period
  2. Batch delete the matching records from the metadata table
  3. Batch delete the corresponding index entries
  4. Repeat until all expired records for that device are purged

Partition-Based Deletion​

Metadata is partitioned by device ID in Table Storage. This means deletion for one camera doesn't affect other cameras' data. Each queue message processes one device at a time, keeping operations isolated and recoverable.

The cleanup job relies on a dual-table design — a device-partitioned metadata table for fast reads and a time-partitioned index table for efficient expiry lookups. See Data Structure for a full explanation of why this design is necessary and how both tables are used.

Error Handling​

If a queue message fails processing:

  • Azure Functions automatically retries the message (per queue retry policy)
  • After max retries, the message moves to the poison queue
  • This prevents one camera's failed cleanup from blocking others