Analytics Lag and Ingestion Checks

Synopsis

Intended end result: Restore fresh event flow into analytics and confirm lag is shrinking.

How this page gets you there: Use database lag checks, ingestion log scans, and temporary window tuning to isolate and fix pipeline delays.

Check whether Postgres is receiving fresh rows:

sudo docker exec -it calfire_db sh -lc '
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "
select now() at time zone '\''utc'\'' as now_utc,
       max(ts_utc) as latest_event_utc,
       extract(epoch from (now() - max(ts_utc)))::int as lag_seconds
from inference_events;"'

Check recent client-side ingestion failures:

sudo docker compose --project-name condor -f /mnt/datadisk/condor/docker-compose.yaml logs --since=30m client | egrep -i "error|exception|traceback|401|403|429|timeout|failed"

If the analytics window is too narrow, temporarily widen it and recreate only the server container:

grep '^REDIS_TS_UTC_WINDOW_SECONDS=' /mnt/datadisk/condor/server/.env
sudo sed -i 's/^REDIS_TS_UTC_WINDOW_SECONDS=.*/REDIS_TS_UTC_WINDOW_SECONDS=86400/' /mnt/datadisk/condor/server/.env

cd /mnt/datadisk/condor
sudo docker compose --project-name condor -f docker-compose.yaml up -d --force-recreate server
sudo docker compose --project-name condor -f docker-compose.yaml logs --since=15m client server_collect | egrep -i "Collected|No new events|error|exception"

Synopsis​

Synopsis