When the AI Sees Smoke

Somewhere on a ridge, a camera's frames get flagged by the CONDOR AI system: possible smoke. On the operator's screen, that camera's arrow turns red and pulses, its tile grows an AI badge — and clicking that badge opens the AI Alert Modal: a chronological reel of the detection frames, with playback to watch the smoke develop, a magnifier to inspect it, and a one-click "Move Camera Here" to aim the real camera at the real spot.

The human is the judge; the modal is the evidence table. This lesson is how it's built.

Opening the case file

Everything starts with one store call. AIIndicatorBtn — the badge on tiles and the detail panel — calls openAiAlert(cameraId) on aiDetectionStore. The provider (context/AIAlertModalContext.tsx) watches aiAlertCameraId and springs into action:

1. Fetches the detection frames via useAiDetection(cameraId)
2. Joins the camera's SignalR group, so new frames stream in live while the modal is open — an unfolding detection updates before the operator's eyes
3. Builds arrayOfImages — the frame URLs, oldest → newest, filtered by the image-type selector (all / 60° / 360°)

One context provider owns all of this; the component tree below it (AIAlertModalContent → header, slider, button bars) is deliberately dumb, reading everything from context. If you're changing modal behavior, you're almost certainly editing the context file, not a component.

The evidence reel

AiAlertSlider renders the frames in a Swiper carousel with three choices worth knowing:

• It opens on the newest frame (arrayOfImages.length - 1) — the operator wants now first, history second — and shows a spinner until that newest image has actually preloaded.
• Slides are virtual. Only a few DOM nodes exist at a time, so a long detection sequence (hundreds of frames) doesn't eat memory.
• The magnifier is CSS, not canvas. When enabled, mouse position drives a 150 px lens whose background-position shifts across a 3×-scaled copy of the image. Cheap, sharp, no pixel copying.

Playback (AiAlertPlaybackControls) drives the same Swiper programmatically — a requestAnimationFrame loop advancing ~300 ms per frame across the last 25 frames, with play/pause, loop, a scrub slider, and a frame counter. Because arrayOfImages is oldest-first, playback naturally runs forward in time, ending on the newest frame.

The video clip

Some detections are video detections. They arrive in the same /api/ai-detection feed as the frames, marked isVideoDetection: true, with the clip in outputImageUrl (an mp4) and the still frame in savedImageUrl. The operational record's hasRecentVideoDetectionHit says a clip exists for the camera's latest detection.

The UX rule is one affordance, one click. Wherever the AI chip appears — tile, camera title bar, details view, the mobile camera sheet, the map's camera callout — the same AIIndicatorBtn grows a small play glyph when a clip exists (▶ AI - 60, tooltip "Video clip available"). Nothing new to learn: the chip still opens the case file, and the case file now opens on the clip.

Inside the modal the reel is a timeline of media, not just frames: buildAiMediaTimeline (util/cameras/aiVideo.ts) orders frames and video detections by timestamp, oldest first, and a video detection is never handed to an <img>. The type selector gains a Video option when the feed holds a video detection; frame time-lapse playback steps through frames only, so clips leave the timeline while it runs and the playback indexes stay honest.

A clip slide (AiVideoSlide) costs nothing until asked for: it shows the still frame, dimmed a touch, with a large red video button (the same glyph as the Play video action) and a red frame — unmistakable next to a frame, with no text drawn over the image. Pressing play on the active slide creates the <video> (muted, inline, preload="auto") and the poster overlays disappear; leaving the slide tears it down, so nothing streams off-screen. The controls are ours, not the browser's: native controls paint a "buffered" segment on the scrubber that operators read as "the clip only loaded 75%" even when the whole file is in memory (measured: Chrome reports the full range buffered and still paints it). Our bar draws only what has played, and it is built for reviewing evidence rather than watching a film. The bar is a band that is never drawn over the picture: on desktop it docks into the modal's bottom row through a portal (the slot #ai-video-controls-dock rendered by AiModalCollapsedBottomBtns, the same place the time-lapse transport takes over), so the picture keeps an image slide's full 16:9 width with no bars beside it and nothing on top of it; on phones, and while the slide is fullscreen (the dock would be outside the fullscreen element), the band sits under the picture instead. Because nothing is behind it, the band does not fade. Once the clip can play, a filmstrip of its own frames (util/cameras/clipThumbnails.ts: an offscreen <video> seeks through the clip and draws evenly spaced samples to a canvas, cached per url) replaces the bar on its own row above the transport as one continuous band (no gaps, no per-picture highlight: the pictures are samples along the timeline, not the clip's frames) with an orange playhead; a tap goes to that point in time and a drag scrubs; hovering shows the time and frame number under the pointer; the frame counter always shows position and total (f47 / 150), and faint per-frame ticks run along the band while they are still legible, so a 150-frame clip never reads as twenty pictures. A host without CORS taints the canvas, so the strip is skipped and the plain played-only scrubber stays. Which frames carry a box comes from CONDOR's sidecar, not from guessing: every clip has a boxes.json beside it (util/cameras/clipBoxes.ts) with the frame rate, the size, and for every inferenced frame the boxes it found. The player reads it as soon as the clip is wanted and uses it three ways: the frame length is exact (no measuring), thumbs whose frame carries a box get a detection-coloured top edge and a detection lane along the scrubber shows every run of boxed frames, and the band names the confidence at the frame on screen ("smoke 90%"). Clips without a sidecar fall back to the measured frame rate and plain strip. Frame stepping (, / ., or the skip buttons) pauses and moves one frame, with the frame length measured from the engine's presented frames (requestVideoFrameCallback, median of eight gaps) and a frame number shown once it is known; speed offers 0.25× and 0.5× slow-motion for watching a plume drift and 1.5× / 2× to skim (< / >); loop is on by default (a ten-second clip is watched several times over) and L or the button turns it off; the clip is always muted and has no sound control, since the streams carry no useful audio; the clock shows tenths of a second. The magnifier works on a clip too: turning it on pauses the clip and the slide hands the modal the frame it is paused on (captured to a data URL; the poster before play), so the operator can step to a frame and inspect the plume up close; while the clip plays there is nothing steady to sample, and a click on the picture no longer toggles play while the magnifier is on. Move Camera Here, Play Time Lapse and the type selector stay available. The download menu saves the clip (MP4) or the frame on screen (JPG, captured from the element), both named <camera>_<yyyymmdd-hhmmss>_detection-<id> so evidence files sort by camera then time; it also copies the clip link and opens it in a new tab. A host that refuses a scripted fetch falls back to a new tab, and a host without CORS headers is retried without them (playback fine, frame capture disabled and labelled as such). Both menus render inside the slide rather than through a portal, so they still work in fullscreen. Keyboard: Space/K, ←/→ (5 s), ,/., </>, L, F, D (download clip), S (save frame). Fullscreen is the clip's own container (our controls stay), toggled by the same button, F, or Escape — never the whole page.

Two things keep the reel legible when "All" is selected, and neither draws text over the image: the bottom row names the current slide's kind (60° image, 360° image, Video clip) next to the confidence, and a dedicated Play video action jumps to the newest clip and starts it. That action and the clip's centre button share one video glyph (a screen with a play mark, red), so "video" reads the same everywhere and is never mistaken for the time-lapse play circle. The modal's own type selector (All / 60° / 360° / Video) is unchanged: it lists only enabled kinds and collapses to a label when one kind is left. The AI settings popover in the navbar (and the mobile AI settings sheet) carries a Detection types selector (AiDetectionTypeSelector): All / 60° / 360° / Video. It is tuned for the fewest clicks per intent — from All, one press isolates a kind; pressing another adds it; pressing the last one, or All, shows everything again — and it can never land on "nothing selected". The preference is not modal-only: isEffectiveAiHit(record, prefs) (util/cameras/aiVideo.ts) is the single rule for "does this camera count as an AI hit", and every surface uses it — the AI chip on tiles, title bar, details view and callout, the map arrow state and tile pulse, the hover popup, and the detection sound. Turn Video off and a camera whose only detection is a clip stops lighting up anywhere. The same rule narrows the AI detected filter itself: shouldCameraBeFiltered drops a camera from the tile grid and the map when its record's kind flags are all switched off, exactly the way the duration filter drops long-running hits, client-side from the record already polled (the latest-detections id set from the API is untouched). A record that reports a hit but no kind flags (older backend) is treated as an image hit. This is a per-device preference in localStorage: the backend AI settings schema has no field for it yet. Map arrows re-evaluate on the next operational poll after a change.

A clip that fails to load, or a format the browser cannot play, degrades to an Open clip link; the bottom bar shows the detection's confidence with the same link. The map's site hover popup shows the same play glyph next to its AI anomaly line. The whole feature sits behind the enable-ai-video flag (on by default; a kill switch, not a rollout gate) and the same AI.Access role as the rest of the AI UI — no extra endpoint, no extra request.

"Move Camera Here"

The payoff button. moveCameraToAlertLocation() writes the detection's pan/tilt/zoom into cameraMovementStore as aiAlertMovementTarget and navigates to /?camera=<id>. From there the standard machinery takes over — the XYMouseTrackerOverlay watches that store value and executes the move, lease checks and all. The modal never talks to the PTZ API itself; it just posts a request on the bus like every other movement source.

Two layouts, one fullscreen trick

Collapsed mode shows the header up top and action buttons below the reel. Fullscreen swaps the bottom bar for AiModalFullscreenBtns, an overlay inside the image area wrapped in FadeAfterInactivityContainer — controls melt away while the operator studies frames and reappear on mouse movement. Fullscreen's overlay bar is hidden entirely on mobile and landscape, where swipe pagination replaces the arrow buttons (useMobilePagination = isMobile || isLandscape).

Who sees any of this

The entire AI surface is role-gated. Users without the AI role get no badges, no red arrows, no modal — see the arrow-state demo for the same gate on the map side.

Wrapping up

A detection flows: CONDOR → red badge → openAiAlert → context provider (fetch + SignalR join) → virtual-slide reel opening on the newest frame → optional playback and magnification → "Move Camera Here" posting to the movement bus. Evidence in, judgment out.

Next feature: the activity log — where operator actions become a shared record.