Skip to main content

Aligning PeakFinder to a Photograph

The PeakFinder overlay draws a DEM-derived terrain skyline with peak labels over a real photograph and asks the two to agree pixel-for-pixel. Every disagreement is one of a small, enumerable set of error sources — and almost every one of them, at some point, has been misdiagnosed as "this camera needs manual calibration." None of them do. This page is the alignment model: where each degree of error comes from, which ones are fixed, and which one remains.

ConcernFile
Camera → PeakFinder viewpoint mappingcomponents/organisms/PeakFinderPanel/cameraToViewpoint.ts
SDK lifecycle (journey gating, aim sequencing)components/organisms/PeakFinderPanel/PeakFinderPanel.tsx
Canvas pinning, blend, pano wiringcomponents/organisms/PeakFinderOverlay/PeakFinderOverlay.tsx
FOV from model + zoomutil/cameras/calculateZoom.ts
Pano scroll → headingutil/cameras/transformToPositiveDegrees.ts

The contract​

PeakFinder is aimed with four numbers plus a location: azimut (compass deg), altitude (pitch deg, SDK range −25…25), fieldofview (horizontal deg), and elevationOffset (metres above its own DEM ground height). If those four describe the photo on screen and the canvas covers the exact displayed-photo rectangle, the skyline lands on the skyline. Every one of those italicised phrases has been a bug.

Error sources, ranked by what they cost​

1. Aiming from the wrong pose (was the largest steady-state error — fixed)​

cameraSelected carries two poses:

  • p / t / z — the static-list pose. Updated when the camera list refreshes; lags every move made by an operator, a lease session, or patrol mode.
  • pan / tilt / zoom (string fields) — stamped on every image response and merged into cameraSelected on each fetch/SignalR push (updateSelectedCamDataOnFetch in IImage/SignalRIImage). This is the pose the camera held when the displayed photo was captured.

The overlay originally aimed from p/t/z. The photo is, by definition, described by pan/tilt/zoom — so after any move the overlay pointed where the camera now is while the photo showed where it was, and for cameras whose static record lagged permanently, the overlay was permanently rotated. cameraToPeakFinderViewpoint now prefers the image pose and falls back to the static pose only before the first image lands. This also makes the overlay self-consistent during moves: it re-aims when the new photo arrives, not when the command is sent — and it re-aims with zero animation, because the photo swaps discretely; an animated aim is by definition misaligned for its whole duration, which in lease mode (a frame per second) meant the overlay was mid-glide essentially the entire time the operator was driving.

2. Altitude sign inversion (was up to 2× tilt vertical error — fixed 2026-08-18)​

PeakFinder's altitude runs opposite to the camera's tilt: tilt −7.8° (pitched down) requires altitude(+7.8). Passing tilt through unchanged mis-aimed every camera by twice its own tilt — a different constant per camera, which is precisely the symptom that reads as "needs per-camera calibration." Verified live by stepping the SDK and measuring the rendered horizon. The mapping is altitudeDeg = clamp(−tilt, −25, 25).

3. loadViewpoint clobbering the aim (was nondeterministic vertical error — fixed 2026-08-17)​

panel.loadViewpoint() keeps mutating altitude asynchronously after it returns; any altitude() set before its 'viewpointjourney finished' event is silently discarded (azimut/fieldofview are unaffected — hence "wrong on the vertical axis only"). PeakFinderPanel therefore splits locationKey (drives the slow journey) from aimKey (cheap re-aim) and performs the first aim inside the journey-finished handler. Do not merge those keys back together: one combined key re-ran the journey — and re-clobbered altitude — on every image.

4. Wrong optics from the model table (systematic per-model scale error — fixed)​

getFieldOfView(cMo, zoom) is the only source of "how many degrees wide is this frame," and an unknown cMo silently fell back to the Q6055-E spec. The backend fleet includes Q6355-LE and Q6315-LE (see GetDefaultPanoZoom in ImageAcquisition's UcsdCameraConfigRefresher), neither of which was in CAMERA_MODEL_DETAILS — every such camera got Q6055-E optics: wrong wide-end FOV (62.8° vs the real 60.6°) and a wildly wrong zoom curve (4.44–142.6 mm interpolation vs the real 6.91–214.64 mm). The table now carries both models, the lookup normalises the string (trim, case, AXIS prefix) before falling back, and the fallback logs once per unknown model. The fixed-focal Mobotix entry no longer produces NaN.

Two related subtleties, still true:

  • VAPIX zoom 10000–19999 is the digital range — a further crop on top of full optical telephoto. getFieldOfView clamps at 9999; getEffectiveFieldOfView applies the digital crop with the exact tangent form. The overlay uses the latter.
  • The overlay clamps FOV to the SDK's 8° floor, so at extreme zoom (FOV < 8°) alignment degrades by construction.

5. Canvas not covering the photo's pixels (fixed)​

The photo is object-fit: contain inside its panel; the overlay measures the actual displayed content rectangle each frame and pins PeakFinder's canvas to it. Both layers then share one pixel space, which is what makes a degree in the SDK equal a degree in the photo. The 60 and 360 views are mounted simultaneously, so the lookup walks outward from the overlay's own host — a document-wide query can pin the 360 overlay to the 60 view's image.

6. Eye height (fixed, with a known edge)​

elevationOffset(cameraEle − panel.viewpoint.elevation()) seats the eye at the camera's real elevation instead of PeakFinder's DEM ground height. The cost of an elevation error is range-dependent: ~5 px per 200 m against terrain 70 km out, but hundreds of px against terrain a few hundred metres out. If a camera looks at very close terrain and only its vertical is off, check ele before anything else.

7. What remains: mount pitch/roll — solved by the automatic boresight​

Camera mounts are not perfectly levelled, and the pan/tilt readout is relative to the mount, not to gravity. Nothing in any metadata records this. It was measured live on boucher-cam07 (2026-08-25, DEM-telescope vs photo skyline): ~0.3–1.0°, structured, stable — and it is the LAST error source once everything above is fixed.

It is corrected by the auto-boresight (util/peakfinder/autoAlign.ts, util/peakfinder/skyline.ts, util/peakfinder/boresight.ts), which runs silently the first time the 60 overlay opens on a camera:

  1. The displayed frame is fetched through pages/api/image-proxy (the blob host taints a canvas; same-origin removes that) and its skyline detected per column — biased hard toward returning nothing over returning a wrong edge.
  2. PeakFinder itself is used as the DEM oracle: telescope.centerAltitude() reports the world altitude of the terrain under the view centre (verified to include Earth curvature and refraction), so a binary search per azimuth yields the DEM skyline in degrees, no pixel access — which matters because the SDK canvas has no preserveDrawingBuffer. The sweep visibly swings the panel, so the overlay hides itself (isAligning) meanwhile.
  3. solveBoresight fits a CONSTANT pitch offset with hard gates (≥15 inliers, ≥80% inlier fraction, wide azimuth span, MAD ≤ 0.2°, |correction| ≤ 3°). Roll is deliberately NOT fitted or applied: on a single frame, roll is indistinguishable from the detector latching a nearer ridge on half the columns (observed in dawn haze), so slope-like evidence makes the solve REJECT instead. The single-frame slope is reported as rollDiagDeg only; rollDeg (the rotation the overlay actually renders) is always 0 from a runtime solve. Roll and the pan-encoder offset need the multi-pose solve; a runtime solve inherits both from the solution it refreshes (seeded or fleet) and sweeps at the inherited azimuth, so it only ever updates pitch.

Every photo pixel ⇄ angle conversion in that chain — solver, zoom-optics measurement, on-photo narration, click-to-locate, and the 60-view PTZ click — goes through the one exact pinhole rotation in util/peakfinder/pixelToDirection.ts (pixelToDirection and its inverse directionToPixel). Do not reintroduce the centre-column shortcut elevation = tilt + atan(v / hypot(f, u)): it is exact only at the centre column, and on the true horizon row of a 65° frame it reads the edges 0.5° low at tilt 3.6° and 1° low at tilt −7.4° — fed an exact synthetic camera, the solver returned 0.58° and 0.38° for a true 0.50° offset and rejected outright at tilt −15°, and the FOV fit measured 66.1° / 58.9° for a true 65.1°. Roll in that function is UNDONE (the photo is the level frame turned clockwise by rollDeg, matching the overlay's CSS rotation); the sign was once inverted, which re-applied the roll and read a horizon point in a 2°-rolled photo at −2°.

A rejected solve logs "conditions unfit" and retries next session; a solution is cached 30 days per camera and applied as an additive altitudeDeg correction in PeakFinderOverlay. There is no manual calibration step, and none should ever be added — that road was walked before the sign bug was found, and it was a dead end.

The definitive solve is offline and multi-pose. A single frame cannot separate pitch from roll from encoder pan offset; several frames at different pans/tilts over-determine all three. The pipeline (proven on boucher-cam07, 2026-08-25): pull archived frames via /api/timelapse (each carries its camera-certified EXIF pose pan=… tilt=… zoom=…), compute DEM skylines independently from AWS Terrain Tiles (ray-march with Earth curvature and k=0.13 refraction — cross-checked against hand calculations to ~0.05°), detect each photo's skyline, and run a robust Gauss–Newton fit of (Δpan, pitch₀, roll) jointly across poses — using ONLY azimuths where the DEM's far (≥15 km) skyline stands ≥0.3° clear of all nearer terrain, because along a camera's own ridge the near/far skylines are altitude-tangent and flip ambiguously (and near ridges carry unmodeled tree canopy). Boucher solved to Δpan −0.33°, pitch +0.68° above encoder, roll +0.30°, with per-frame residual medians ≤0.15° (≈4 px). Fleet solutions live in the Cosmos boresight store served by Frontend.API (GET /PeakFinder/boresight); util/peakfinder/seededBoresights.ts is only a code-shipped fallback for cameras with no store entry. Precedence in loadBoresight: fleet offline (approved solver output) > fleet manual > runtime (in-browser auto-align) > seed. A tilted base shows up as a per-pan table (attitudeByPan), resolved at the image's pan by resolveBoresightAtPan. The overlay applies all three terms: dAzDeg and dAltDeg into the aim, rollDeg as a CSS rotation of the render canvas (the SDK has no roll input).

One operational caution: PeakFinder's services rate-limit heavy SDK probing per IP (observed as panel init/journeys degrading from instant to minutes during an intensive debugging day). The runtime auto-align's rare sweeps are fine; do not build anything that hammers the telescope API in bulk — the offline pipeline exists precisely so calibration never needs PeakFinder at all.

The 360 pano is a different geometry problem​

Facts verified in ImageAcquisition (VAPIXClient.GetPanoImagesAsync, PanoImageStorageService.StitchImagesSideBySideAndSavePano):

  • The sweep captures PanoImageCount (default 6) frames at pan increments of 360/N, at tilt = 0, at a per-model PanoZoom chosen so each frame's FOV ≈ the increment.
  • The strip is a naive side-by-side concatenation of those rectilinear frames. No cylindrical warp, no crop, no blend.

Consequences:

  • altitudeDeg: 0 is correct, not a placeholder. Tilt-0 frames put the horizon on the strip's vertical centre line by construction; the strip fills the viewport height and the canvas fills the viewport, so the centre lines coincide. Aiming the pano overlay at the camera's live tilt would be a bug — the pano was not captured at that tilt.
  • The overlay's FOV must be the viewport's true angular span — viewportWidth / stripWidth × 360, routinely ~117°, which is why the pano path raises the SDK clamp to 179 (the SDK accepts it; the documented 8–90 range is for URL parameters).
  • The heading must be unrounded — convertScrollOffsetToDegrees(..., precise = true). At ~11 px/degree, whole-degree rounding is an ~11 px snap.
  • The irreducible frontend error is the projection mismatch. PeakFinder renders a true cylinder (linear in azimuth; vertical ∝ tan(altitude)). The strip is piecewise-rectilinear: within each frame, azimuth is tangent-distributed and treated as linear by every consumer. With a frame FOV of ~57–61° against a 60° slot, the horizontal error is a sawtooth — zero near frame centres and seams, up to ~1.5° between — and the vertical scale away from the horizon disagrees by ~10%. No constant offset, and no per-camera calibration, can remove a sawtooth.

The architectural fix is upstream, and it exists: ImageAcquisition's CylindricalPanoProjector (Services/Imaging/, wired into PanoImageStorageService.StitchImagesSideBySideAndSavePano) reprojects each frame to cylindrical coordinates at stitch time — the focal length is known from the model + PanoZoom (CameraOptics, kept in lockstep with this repo's calculateZoom.ts), tilt is 0, so the warp is a closed-form per-column resample with no feature matching. The strip keeps its exact legacy dimensions and frame centres stay at slot centres, so nothing downstream changes shape; azimuth simply becomes genuinely linear in x, which fixes the PeakFinder overlay, the compass readout, and the click-to-aim pan math at once. Unknown models, implausible FOVs, any warp failure, or the EnableCylindricalPanoStitch config flag being unset (it is opt-in, off by default — set it to true per environment, dev first) fall back to the naive concatenation. Strips produced before that change shipped still carry the sawtooth.

External (UCSD) panos arrive in their own rotational frame with unknown per-frame geometry; they get the bearing-offset calibration for heading and the PeakFinder overlay stays hidden for them (isExternalPano === true).

The render contract, verified against the live SDK (2026-08-26)​

Measured with a standalone harness (bare canvas + the real SDK, PeakFinder's degree grid turned on, known azimuth/altitude steps, pixel measurement of the displacement):

  • fieldofview() is the HORIZONTAL field of view across the canvas width, and the render is an ideal pinhole: feature displacement matches f = (W/2)/tan(fov/2) on both axes (square pixels — the vertical scale uses the same focal). A gridline 28.7° off-axis sat within ~3 px of the pinhole prediction. Our mapping layer is therefore exactly right; there is no hidden aspect or diagonal-FOV convention to compensate for.
  • altitude() is look-down-positive (view pitch = −altitude), re-confirmed visually.
  • The WebGL buffer is sized from the canvas once at init() (CSS size × devicePixelRatio) and re-synced only on real window resizes — synthetic resize events and canvas-only CSS changes are ignored. Same-aspect CSS changes stretch uniformly (angularly harmless); an aspect CHANGE would corrupt the vertical scale, which is why PeakFinderOverlay keys the panel on the pinned rect's aspect so init always sees the final aspect.
  • viewpointjourney finished is not a readiness signal. A panel was observed fully seated (terrain loaded, viewpoint.elevation() reporting ground) and correctly aimable while viewpointJourneyFinished() stayed 0 for ~40 s; under PeakFinder's per-IP throttling the event can simply never fire. Gating the aim on the event therefore leaves the overlay at SDK defaults (az 90 / fov 22.5) — the single biggest source of "gross misalignment" ever found in this feature, because the aim was never applied at all.

The aim is therefore closed-loop, not event-driven (PeakFinderPanel's aim keeper): write the aim, read every parameter back (azimuth, altitude, FOV, elevation offset), and re-assert on any mismatch — fast (500 ms) while unsettled, heartbeat (2 s) once holding. A journey still mutating the camera shows up as readback drift and gets overwritten again; a settled panel goes quiet. This converges under mid-journey clobbers, late events, absent events and throttled servers alike, with no need to detect which one happened. The journey watchdog now re-issues loadViewpoint only when the viewpoint never seated at all (viewpoint.elevation() still 0) — re-issuing after a seat threw away downloaded terrain and restarted the stall. elevationOffset is only applied once a real ground elevation is readable (it reads 0 pre-seat; offsetting against that phantom ground would hoist the eye by the camera's full elevation).

Zero-offset roadmap (what remains, with budget)​

With the aim guaranteed (keeper), the mapping proven (pinhole/HFOV), and the mount solved (boresight), the full 60° error budget on the far skyline is:

SourceSizeStatus
Aim never applied (journey stall)unboundedeliminated by the aim keeper
Mount attitude (pan/pitch/roll)0.3–1°solved per camera; boucher-cam07 residual ≤0.15°
Lens radial distortion (Q6075 wide)~0 centre, up to ~0.3–0.5° at extreme frame edgesunmodelled; measurable from solve residual structure; PF cannot render it — would need photo undistortion or a residual warp
DEM bare-earth vs tree canopy0.2–0.4° on ridges <5 km; <0.05° beyond 15 kmphysics, not error — judge alignment on FAR skyline/labels
PF DEM + our aim quantisation≲0.05°negligible
Ray-march ground track (map marks)was 0.3° at 100 km on E/W bearingseliminated — terrainRaycast steps along the great circle, not a flat lat/lon line
Runtime DEM probe spacing (4.5°)interpolation noise on jagged ridges, ≲0.1–0.2°remains; the far-terrain gate now takes the NEARER bracketing probe, so near-ridge columns cannot pass as far evidence

Production flow (built 2026-09-04): scripts/boresight/solve.ts runs nightly (devops/boresight-nightly.yml) on archived EXIF-posed frames × DEM ray-marched skylines, solving each visited pan on its own (pitch, plus roll when the far skyline spans both halves) and dAzDeg across pans. It does not write the fleet store. Each solved camera is SUBMITTED as a review candidate to Alert.CA.Admin.API (PUT /PeakFinderBoresight/candidates) with every pan's numbers, the frames used and one QA render per pan (photo skyline red, terrain ridge at the encoder pose grey, ridge with the proposed correction green — qa.ts). An administrator approves or rejects it on the Admin app's PeakFinder Calibration page; approval re-checks the shared gates (PeakFinderBoresightGates in AlertCAInfrastructure) and stores the entry as source: offline, which every overlay picks up on its next load. Auth needs no Entra objects: the pipeline sends the shared x-api-key configured as PeakFinderCalibrationOptions:ApiKey (same pattern as the user-ingestion webhook); a person can submit with their own Admin-app token (BORESIGHT_ADMIN_TOKEN). The continuous cheap health check (per-frame skyline residual vs DEM) remains the drift alarm that should trigger a re-solve.

Debugging checklist​

  • Ridge lines diverge from the photo's skyline and label pins sit on the wrong hills, worse in some directions than others — the visibility range is culling terrain the photo shows. Measured live (Boucher, 2026-08-26): at a 20 km range the rendered silhouette sat up to 3.5° from the photo's real 56–160 km skyline. The photo contains every distance, so the overlay's terrain range must never drop below the true skyline distance — hence MIN_VISIBILITY_RANGE_KM (150) enforced in both the slider and applyDisplaySettings. Label density is minimalElevationM's job, never the range.
  • Near forested ridges (<5 km) draw slightly off while far peaks align — bare-earth vs canopy physics (~0.3° at 4 km), not aim error; judge alignment on the far skyline.
  • Overlay rotated by a constant after a camera move — something is aiming from p/t/z instead of the image's pan/tilt/zoom.
  • Vertical-only error, different per camera — altitude sign (or a regression of the journey-clobber gating). Read back panel.altitude() after 'viewpointjourney finished'; reading earlier returns SDK defaults (az 90, alt 0, fov 22.5).
  • Scale error that grows toward the frame edges — FOV wrong: unknown cMo (check the console for the once-per-model warning), or a camera in the digital zoom range.
  • Pano drift that oscillates as you scroll — the piecewise-rectilinear strip (expected, see above) if it's ~1–2°; the FOV clamp or heading rounding if it's large.
  • Whole image dims when the overlay opens — blend regression: opacity must live on the inner wrapper, never on the element carrying mix-blend-mode.
  • Neither layer can be pixel-sampled in-page — photo pixels are CORS-tainted and the SDK canvas has no preserveDrawingBuffer. Verify with an injected reference line + screenshots.