Skip to main content

Data Tiers — Raw to Gold

Data enters this platform from many directions — a camera on a ridge, an AI model flagging smoke, an external inventory system, a dispatcher drawing an alert area — and none of it is in the shape the frontend needs. Between "raw signal" and "what an operator sees" sits a consistent transformation, and this page is the map of it. It's a reference overview: read it to understand how any one entity flows end to end, or to place a subsystem in the bigger picture.

The three tiers​

The platform follows a medallion-style progression — the same idea whatever the entity:

  • Raw — data exactly as it arrives: a UCSD device record, JPEG bytes off a camera, a Condor detection payload, an ESRI polygon. Unvalidated, unjoined, in the source's shape.
  • Enriched — cleaned, validated, joined, and made durable in the platform's own stores (Cosmos, SQL, Blob, Table) via the shared AlertCAInfrastructure layer. This is where encryption, geometry validation, AI localization, subscriber matching, and change detection happen.
  • Gold — the frontend-shaped result: RBAC-scoped CQRS DTOs from the Frontend API, real-time SignalR pushes, and ESRI map layers. This is the only tier the frontend and its users touch.

Follow one entity through the tiers​

The overview above collapses every entity into three stores. The reality per entity is more specific — pick one and trace its raw source, its enrichment, and the gold form the frontend serves:

Rawas it hits the system
  • UCSD Metadata API
  • device configs · credentials · ACAPs (external source of truth)
Enrichedcleaned · joined · durable
  • CameraOperation refresher (5 min) → Cosmos: AcquisitionDevice, CameraStatic, CameraOperational + SQL Camera (RBAC)
  • credentials AES-encrypted; change-detected upserts
Goldwhat the frontend serves
  • Frontend.API camera queries → RBAC-scoped DTOs
  • runtime status via monitoring Table + SignalR

📷 Cameras: raw source on the left becomes the frontend-shaped gold on the right — the middle tier is where the platform's stores and transforms do the work.

Why the middle tier carries the weight​

The Raw and Gold tiers are thin by design; almost all the work is in Enriched, and it's the same handful of transforms recurring across entities:

  • Encryption — camera credentials and addresses are AES-encrypted at rest, decrypted only at the moment of a camera call.
  • Geometry — GeoJSON/ESRI/WKT shapes are validated and converted; a point + radius becomes a latitude-aware circle, so proximity questions are answerable in SQL.
  • AI localization — a raw detection is enriched with fire lat/lon, bearing, and distance before it's stored or mapped.
  • Subscriber matching — an alert is intersected against subscription areas, then filtered by urgency bitmask and schedule, to decide who is notified.
  • Change detection — the camera refresh compares decrypted values and writes only real differences, with an audit trail.

Wrapping up​

Every entity on the platform — cameras, acquisition, images, AI detections, alerts, notifications, users, locations — follows the same medallion path: raw signal in the source's shape, enriched into validated, joined, durable records in the platform's stores, then served as gold, frontend-shaped data through the Frontend API, SignalR, and the map. The middle tier is where the platform earns its keep; the gold tier is the only one users ever see.