Scaling & Parallelism
The Metadata Cleanup service uses three independent queues to maximize throughput.
Why Three Queues?
Each image type has its own Table Storage tables and index tables. By using separate queues:
- Independent scaling — If single images generate more metadata than panoramics, the single queue processes more without waiting for pano cleanup to finish
- Isolated failures — A failure in panoramic cleanup doesn't block single image cleanup
- Parallelism — All three queue processors run concurrently on the same Functions host
Queue Configuration
| Queue | Image Type | Tables Affected |
|---|---|---|
single-metadata-delete | Single images | SingleImageMetadata, SingleImageMetadataIndex |
pano-metadata-delete | Panoramic images | PanoImageMetadata, PanoImageMetadataIndex |
prestitch-metadata-delete | Pre-stitched archives | PreStitchImageMetadata, PreStitchImageMetadataIndex |
Throughput
Azure Functions processes queue messages with configurable batch sizes and polling intervals. The cleanup function can handle hundreds of cameras' worth of metadata deletion in each scheduled run.