Retention Cleanup

Every camera writes image metadata every minute, forever. Without a retention job, the Table Storage metadata would grow without bound. Alert.CA.MetadataCleanup is the small Functions app that retires it — and the interesting part is how it deletes at fleet scale without scanning every camera.

The one idea to hold on to

Cleanup is a two-stage, index-driven fan-out: a timer finds what's expired via the index table, then shards the deletes per device onto a queue. It runs as three enqueue/dequeue pairs — one each for single, pano (360), and pre-stitch metadata:

The enqueue pass computes a cutoff (now − RetentionDays), queries the index table for rows older than it — a cheap partition-key range scan, because the index is partitioned by a minute-precision time bucket, not by camera — then groups the expired rows by device and drops one cleanup message per device. The dequeue pass deletes that device's metadata rows in chunked batches, then the index rows.

This is the payoff of the dual-table index the storage services maintain: the primary table is partitioned by camera (great for reads), so "everything older than N days across all cameras" would otherwise mean walking every camera's partition. The time-bucketed index turns it into one range scan.

Gotcha: the index partition format is minute-precision — trust the code

The index PartitionKey is yyyyMMddHHmm (minute), even though some comments say yyyyMMddHH (hour). The cutoff query compares against the minute format; a hand-written cleanup query that assumes hour buckets would select the wrong rows. This is the same drift flagged in the Infrastructure Table Storage docs — the code is authoritative.

Wrapping up

Retention cleanup is an index-driven, device-sharded fan-out: a per-minute timer finds expired rows with a single range scan over the time-bucketed index, then queues one delete per device for parallel, batched, retryable removal of metadata and index rows alike. It's the reason image metadata stays bounded, and it only works because every image write maintains that index.

That's the acquisition data path end to end. Next, the control plane that drives all of it: The Control Plane.