Camera Snapshot Stream

QUERY /Cameras/operational and QUERY /CameraFilters/* compute a fleet-wide answer per request: a cross-partition Cosmos query plus one Table Storage read per camera, for every client every 10 seconds. The Camera Snapshot stream computes that answer once per process per tick and serves bytes from memory, and pushes changes instead of being polled for them.

Pieces

Piece	Type	Role
CameraSnapshotHostedService	BackgroundService	Every RebuildIntervalSeconds (5 s) rebuilds the snapshot through ICameraSnapshotBuilder inside a DI scope and publishes it to the store. Re-reads the feature flag every 30 s; clears the store when it is off. Exponential backoff on failure, never exits.
CameraSnapshotBuilder	singleton	Operational documents from the Cosmos change feed (pull model, continuation token kept between ticks: one full pass at startup, then only what changed) → public id list (static docs, every StaticIdsRefreshIntervalSeconds) → offline list (every OfflineRefreshIntervalSeconds) → per-device AI scores (one SQL call, every AiScoresRefreshIntervalSeconds) → CameraSnapshotDocument serialised once, ETag = hash of the rows (not of asOfTime, so unchanged rows revalidate as 304). Images come from the operational document's denormalised latestImage* fields; a document without them (written before ImageAcquisition shipped) falls back to one throttled Table read. Dam cameras are kept aside for the private overlay. A rejected continuation (410/400) restarts from the beginning.
CosmosOperationalChangeFeedReader	singleton	The pull-model change feed over the operational logical partition of the camera container.
CameraSnapshotStore	singleton	One volatile reference swap per build. IsEnabled = option AND flag.
CameraSnapshotController	GET /CameraSnapshot (anonymous), GET /CameraSnapshot/private (authorised)	Public: pre-built bytes, ETag, Cache-Control: public, max-age=0, s-maxage=5, must-revalidate, 304 on If-None-Match. Private: AI flags, the latest-event confidence and the latest detection date / start date per camera for the AI role, dam rows for allow-listed companies, private, no-store. Both: 503 + no-store when disabled, not built, or older than StaleAfterSeconds.
CameraHub AI fast path	hub methods	SubscribeAiDetectionEvents() joins ai-detection-events only when the connection's negotiated identity has the AI role (the browser negotiates through the BFF, which attaches the user's Entra bearer, so Context.User is the same principal the controllers authorise); BroadcastAiDetection(id, CameraAiDetectionMessage) (publisher key) fans the camera's new AI state with values to that group only. Published by Notification the moment it patches the flags (new image detection, video clip, stale-flag cleanup), so an operator sees a detection about a second after Condor reported it; the private overlay stays the reconciliation.
CameraHub additions	hub methods	SubscribeCameraUpdates(ids) / UnsubscribeCameraUpdates(ids) → camera-updates:<id> groups (subscribe also returns the cameras' current public rows from the in-memory snapshot, delta-shaped, so a tile is current the moment it scrolls into view instead of after the camera's next capture; dam rows excluded); SubscribeCameraStatusEvents() → camera-status-events; BroadcastCameraUpdate(id, CameraUpdateMessage) requires publisherKey = FrontendApiOptions:ApiKey (constant-time compare, fail closed, stripped before fan-out), then fans out ReceiveCameraUpdate and, when statusKinds is non-empty, ReceiveCameraStatusEvent (public-safe, no AI values).

The legacy per-camera groups (JoinGroup, ReceiveImageBroadcast, …) are untouched; the new groups are name-prefixed so the two cannot collide.

The public document reuses CameraOperationalMetadata with includeAIHits: false — byte-for-byte the row shape an anonymous caller gets from the legacy endpoint today, so the frontend store needs no new mapping.

Gating and rollback

Two switches, both must be on:

• CameraSnapshotOptions:Enabled (App Configuration, default true) — deploy-time master; read at startup.
• SQL FeatureFlag row enable-camera-snapshot-stream — runtime switch, polled every 30 s, shared with the frontend and with ImageAcquisition's publisher. Missing row = off.

Rollback = set the row to 0. Within 30 s the builder idles and the endpoints answer 503 with Cache-Control: no-store; Front Door never caches that, and the frontend treats it as "fall back to legacy polling". No deploy, no restart.

Options (CameraSnapshotOptions)

Key	Default	Meaning
RebuildIntervalSeconds	5	Snapshot rebuild cadence (one change-feed pull per tick)
FeatureFlagPollIntervalSeconds	30	Flag re-read cadence
StaticIdsRefreshIntervalSeconds	300	Public camera id list re-read
OfflineRefreshIntervalSeconds	15	Durable-entity offline list re-read
AiScoresRefreshIntervalSeconds	5	Per-device AI score re-read (CondorCameraEventGetLatestScoresByDeviceIds)
ImageLookupConcurrency	16	Parallel Table Storage reads for the transitional image fallback. The fallback runs once per document version, only for cameras the snapshot publishes (public or dam); a miss is remembered until the document changes. (Before that rule uat queried ~1,450 frame-less documents every 30 s: 4 M table transactions a day.)
EdgeCacheSeconds	5	s-maxage advertised to Front Door
StaleAfterSeconds	90	Serve 503 instead of a snapshot older than this

Load profile

Per instance, independent of user count: one change-feed pull per 5 s that returns only the documents written since the last tick (tens, not the fleet), one durable-entity scan per 15 s, one SQL call per 5 s, one static read per 5 minutes. The camera container keeps every operational document in one logical partition, so the fleet-sized read the legacy endpoint performs per request is exactly what the change feed avoids. Per request: zero storage calls. The private overlay is a projection of the in-memory rows.

Prerequisites, in order (all in this mono repo, one PR): the latestImage* entity fields and CondorCameraEventGetLatestScoresByDeviceIds live in Alert.CA.Infrastructure / Alert.CA.Infrastructure.Data, which Frontend.API and ImageAcquisition reference as projects — there is no package publish step any more. The Infrastructure pipeline deploys the DACPAC; it must run before the feature flag is turned on (a missing procedure is logged and the previous scores are kept, so the order is about correctness of the AI filter, not availability). ImageAcquisition writes the image fields at capture from the same deploy.

Cosmos RU budget (measured in prd, 2026-09-28)

The leases container is not the constraint: its 7-day peak is 4 % of a 6,000 RU/s autoscale ceiling and it serves about 2 requests/s. The new change-feed trigger checkpoints one small lease document per physical partition per batch and renews every 17 s, well under 20 RU/s in total.

The camera container is the one to watch. Every operational document lives in one logical partition (dataType = operational), so one physical partition (of 5) carries every capture patch: 70 patches/s at ~11 RU = ~800 RU/s, 88 % of the container's RU, and that partition peaks at ~68 % of its 10,000 RU/s share (one 100 % minute in the last 7 days).

Change-feed reads are cheap. Cosmos meters them as their own operation type, and the two consumers already in prd measure as: ~1 RU per poll per physical partition when the feed is empty (acquisitionDevice, 32,850 polls for 32,495 RU in a day) and ~0.02 RU per delivered document on a busy feed (condorEvent, 21,019 upserts delivered for ~400 RU beyond the polling). That is one to two orders of magnitude below a query over the same bytes.

Load on the operational partition	RU/s	Behind the flag?
Capture patch grows from 3 to 7 operations, 4 new indexed paths	est. +100–150	no — the Terraform Cosmos module exposes no excluded_path; excluding /latestImage* from the index is a module follow-up
ImageAcquisition change-feed trigger (1 partition, poll every 5 s + ~70 docs/s)	≈ 2	yes
Frontend.API builders, one pull per instance per 5 s (30 instances)	≈ 50	yes
Legacy operational IN-queries removed (0.7/s today; ~100/s at 1,000 tabs)	−20 today, −3,000+ at 1k tabs	—

Each tick logs Camera snapshot change feed: N documents, X RU this tick, so the per-instance cost is a direct reading in App Insights (AppTraces, sum of X per minute divided by 60, times the instance count). Expect a few thousand RU once for the first full pass, then single-digit RU per tick.

Gate for uat/prd: with the flag on, the hot partition's NormalizedRUConsumption stays below 80 % at its worst minute (Azure Monitor, camera, PartitionKeyRangeId). The only lever expected to matter is the index exclusion above.

Operating it

• Build failures: CameraSnapshotHostedService logs Camera snapshot build failed with backoff; the previous snapshot keeps serving until StaleAfterSeconds.
• Flag flips: Camera snapshot stream feature flag is now ON|OFF at Information level.
• 503 volume on /CameraSnapshot is the "feature off or unhealthy" signal; the frontend records CameraSnapshot_Degraded on its side.