Skip to main content

VM Runtime Health Checks

Synopsis​

Intended end result: Get a fast, high-signal health snapshot of the VM and container runtime.

How this page gets you there: Use host, GPU, memory, process, and container checks first, then run CPU trend capture and service-specific verification steps.

Host-level VM inspection (quick checks)​

Use these commands on the VM for a fast health snapshot:

# one-shot CPU and load snapshot with top processes.
top -bn1 | head -20

# GPU utilization, memory usage, and driver/runtime health.
nvidia-smi

# RAM and swap availability in human-readable units.
free -h

# short CPU, run queue, memory, and IO trend sample.
vmstat 1 5

# biggest memory consumers across host processes.
ps aux --sort=-%mem | head -15

# one-shot container CPU/memory usage view.
sudo docker stats --no-stream

CPU over time with sar​

Run long sar collection detached so your shell stays usable while data is gathered.

Start detached collection (1 hour):

nohup sar -u 1 3600 > /tmp/sar-u-1h.out 2>&1 &
echo $! > /tmp/sar-u-1h.pid

Check whether collection is still running:

ps -fp "$(cat /tmp/sar-u-1h.pid)"

Tail live progress/results:

tail -n 30 /tmp/sar-u-1h.out

Review final summary when complete:

grep -E '^[0-9]{2}:[0-9]{2}:[0-9]{2}|Average' /tmp/sar-u-1h.out | tail -n 40

If you want faster output:

sar -u 1 60

If sysstat is enabled and you want historical CPU data from the last 24 hours, query the daily archive instead of waiting for live collection:

sar -u -f /var/log/sysstat/sa$(date +%d)

For a concise hourly trend from today:

sar -u -f /var/log/sysstat/sa$(date +%d) | awk 'NR==1 || /Average/ || /[0-9]{2}:[0-9]{2}:[0-9]{2}/'

Check the current state of the deployed stack:

sudo docker compose --project-name condor -f /mnt/datadisk/condor/docker-compose.yaml ps -a
curl -I http://localhost:8050/health
sudo docker compose --project-name condor -f /mnt/datadisk/condor/docker-compose.yaml logs --tail=100 server

Inspect the data path when the dashboard shows no results:

sudo docker compose --project-name condor -f /mnt/datadisk/condor/docker-compose.yaml logs --since=30m client server_collect server_backfill server_purge server

sudo docker exec -it calfire_db sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "select count(*) as total_events, max(ts_utc) as latest_event from inference_events;"'

sudo docker exec -it condor_redis sh -lc 'redis-cli -a "$REDIS_PASSWORD" KEYS "analytics:date:*" | head'

curl -s http://localhost:8050/api/analytics/day?date=<yyyy-mm-dd> | jq '.'

Rebuild and restart the stack when recovery is faster than surgical debugging:

cd /mnt/datadisk/condor
sudo docker compose --project-name condor -f docker-compose.yaml down
sudo docker compose --project-name condor -f docker-compose.yaml up -d --build
sudo docker compose --project-name condor -f docker-compose.yaml logs -f server_backfill