🧠 Deploying a New Model Version

Automated Deployment (Recommended)

deploy-model.ps1 runs the full pipeline — get the ONNX onto the VM, convert it with TensorRT, sync files to storage, restart containers, and verify — in one command. It's interactive: it asks you to confirm before making changes, and connects to the VM over SSH (you'll enter your VM password once).

Prerequisites

OpenSSH client (built-in on macOS and Windows 10+); PowerShell 7+ recommended.
Azure CLI (az) installed and logged in (az login) — used to download/upload model files from Azure Storage.
Your IP allow-listed for SSH to the VM (see Manual Deployment Steps below for how to add a rule).

Usage

# Full deployment: download ONNX from storage, convert, restart, verify
.\azure\infrastructure\scripts\deploy-model.ps1 -ModelVersion 16

# Full deployment using a local ONNX file instead of pulling from Azure Storage
.\azure\infrastructure\scripts\deploy-model.ps1 -ModelVersion 16 -OnnxFilePath "C:\models\new_model.onnx"

# Preview what would happen without making any changes
.\azure\infrastructure\scripts\deploy-model.ps1 -ModelVersion 16 -DryRun

tip

Always run with -DryRun first when you're unsure — it prints every step it would take without touching the VM, storage, or containers.

What it does

For a given -ModelVersion, the script:

Gets the ONNX onto the VM — downloads it from Azure Storage (modelrepository/ai4gsmokefirev1/{version}/model.onnx), or uploads a local file if you pass -OnnxFilePath. Skipped if model.onnx already exists on the VM (you'll be asked before re-uploading).
Runs the TensorRT conversion on the VM via Docker (trtexec) — this is the slow step, typically 10–30 minutes.
Downloads the resulting model.plan back into your local model_repository/ai4gsmokefirev1/{version}/ folder.
Syncs model.onnx + model.plan to Azure Storage as a backup (skip with -SkipStorageSync).
Updates MODEL_VER in the VM's client/.env and restarts the triton container, waits for it to report ready, then restarts client.
Verifies by printing the Triton and client logs so you can confirm the new version loaded correctly.

Only step 1 needs anything more than -ModelVersion — the rest reuse whatever version you already have on the VM/locally.

Running only part of the pipeline

Use -Action to run a subset of the steps above, useful when re-running after a partial failure:

`-Action` value	Runs	Use when
`Full` (default)	All 6 steps	Normal end-to-end deployment
`ConvertOnly`	TensorRT conversion only	`model.onnx` is already on the VM and you just need `model.plan`
`RestartOnly`	Update `MODEL_VER` + restart containers	Conversion already succeeded, you just need to switch the running version
`DownloadPlan`	Download `model.plan` from VM only	You converted on the VM already and just need the file locally

Other options

Parameter	Default	When to change it
`-VmHost` / `-VmUser` / `-VmCondorPath`	`20.125.45.30` / `condoradmin` / `/mnt/datadisk/condor`	Only if the target VM changes
`-StorageAccountName` / `-StorageContainerName`	`azw2hubcondor` / `modelrepository`	Only if the storage backend changes
`-ModelName`	`ai4gsmokefirev1`	Only if deploying a different model
`-SkipStorageSync`	off	Skip the Azure Storage backup step
`-DryRun`	off	Preview only, no changes made

After it finishes

The script prints next steps at the end. In short:

Confirm the right version is actually running — see Confirming the Right Model Version Is Running below.
Remove the old model version folder from model_repository/ai4gsmokefirev1/ in the repo.
Commit the new version folder and merge to main.
If infrastructure also needs updating (e.g. CLIENT_MODEL_VER in parameters.bicepparam), run the infra scripts separately — see Manual Deployment Steps steps 14+ below.

Confirming the Right Model Version Is Running

Do this after any restart (script or manual) before calling the deployment done — a container restart succeeding doesn't by itself prove the new version loaded. All commands run on the VM, from /mnt/datadisk/condor. Replace <VERSION> with the version you just deployed.

Client startup log — on startup the client logs exactly which model/version it's using:
```
grep "Triton (gRPC)" calfire_inference.log | tail -5
```
Expect a line like Triton (gRPC): <url> | model=ai4gsmokefirev1 v=<VERSION> | concurrency=... — confirm v= matches.
Client container environment — confirms the env var actually driving that log line, not just the log file:
```
sudo docker compose --project-name condor exec client env | grep -E 'MODEL_NAME|MODEL_VER|TRITON'
```
MODEL_VER should equal <VERSION>. If it still shows the old version, client/.env wasn't updated or the container wasn't recreated after the edit — re-check with grep MODEL_VER client/.env before restarting again.
Triton logs — confirms Triton itself loaded that version (not just that the container started):
```
sudo docker compose --project-name condor logs --tail=100 triton | grep -i "ai4gsmokefirev1\|successfully loaded\|ready"
```
Seeing an older version still listed is fine — Triton can serve multiple versions side by side — as long as <VERSION> shows as loaded/ready.
Triton readiness API — the definitive check (this is what deploy-model.ps1 itself polls after a restart):
```
curl -sf http://localhost:8000/v2/models/ai4gsmokefirev1/versions/<VERSION>/ready
```
Exit code 0 (no output needed) = ready. Non-zero = not ready yet; wait and retry, or check the Triton logs above for why.
Live inference activity — once cameras resume polling, watch for errors on the new version:
```
tail -n 300 -f calfire_inference.log
```

Manual Deployment Steps (Fallback / Reference)

Only needed if the automated script can't reach the VM (e.g. SSH blocked, no OpenSSH client available) or you want to run a step by hand to debug it.

Microsoft Team sends link to download the new model ONNX file via Microsoft Teams.
In the Alert.CA.CONDOR repo, under model_repository/ai4gsmokefirev1, create a folder for the new version (e.g. 12, 13) and move the ONNX file there.
Rename the file to model.onnx.
Start a VS Code Tunnel (or another method for moving files to/from the VM).
Open /mnt/datadisk/condor/ on the VM.
Create the same new version folder under model_repository/ai4gsmokefirev1 there, and copy model.onnx into it.
Connect via Bastion to the Azure VM ai4gl-condor-uat-vm:
1. Located in the AlertCA-DEV subscription, resource group rg-wfca-ca-inf-alertca-dev-001, vnet/subnet azw2-inf-vnet-dev-001/azw2-inf-snet-vm-001.
2. Username: Condor:Deployment:VmUsername in app config azw2-appcs-wfca-alertca-hub-001 (AlertCA-HUB subscription, resource group azw2-inf-rg-hub).
3. Password: part of the AICondorDeploymentParam secret in key vault kv-wfcacaalertca-hub-001 (AlertCA-HUB subscription, resource group azw2-inf-rg-hub).
4. If SSH is blocked, add your IP to the NSG rule: Condor VM → Networking → Networking Settings → Rules → AllowSshFromConfiguredRange → add your IP to the source address prefixes (TCP/22, Allow, priority 110).
On the VM, cd /mnt/datadisk/condor.

Run the TensorRT conversion (replace <VERSION> with the actual folder number):

sudo docker run --gpus all -it --rm \
  -v "$(pwd)/model_repository:/models" \
  nvcr.io/nvidia/tensorrt:24.05-py3 bash -lc \
  'trtexec \
    --onnx=/models/ai4gsmokefirev1/<VERSION>/model.onnx \
    --saveEngine=/models/ai4gsmokefirev1/<VERSION>/model.plan \
    --fp16 \
    --inputIOFormats=fp16:chw \
    --minShapes=images:1x3x1024x1024 \
    --optShapes=images:16x3x1024x1024 \
    --maxShapes=images:128x3x1024x1024 \
    --useCudaGraph'

Download the generated model.plan file back to your local machine.
Locally, run sync-models-to-storage.ps1 to back up the ONNX and plan files to the azw2hubcondor storage account (AlertCA-HUB subscription, resource group azw2-rg-wfca-condor-hub-001).

On the VM, recreate the triton and client containers with the new model version (replace <VERSION>):

docker compose ps
sudo docker compose --project-name condor -f /mnt/datadisk/condor/docker-compose.yaml exec client env | grep -E 'MODEL_NAME|MODEL_VER|TRITON'
sudo sed -i 's/^MODEL_VER=.*/MODEL_VER=<VERSION>/' /mnt/datadisk/condor/client/.env
sudo docker compose up -d --force-recreate triton client

Check the logs to confirm the new version is loaded:
```
sudo docker compose --project-name condor -f docker-compose.yaml logs --tail=200 triton
tail -f calfire_inference.log
```
If it isn't, redo the deployment following the steps above rather than proceeding.
Locally, run restore-AICondorDeploymentParamToBicepParams.ps1. If a parameters.bicepparam already exists, it's renamed to parameters.YYYYMMDD-HHMMSS.bicepparam; a new one is created from the AICondorDeploymentParam secret (kv-wfcacaalertca-hub-001) and the Condor:* app config entries (azw2-appcs-wfca-alertca-hub-001).

In the new parameters.bicepparam, double check:

Parameter	Expected Value
`CLIENT_MODEL_VER`	The new model version number, e.g. `12`, `13`
`CLIENT_FIRECAMS_ENABLE`	`'false'`
`CLIENT_ALERTCA_ENABLE`	`'true'`
`CLIENT_ALERTCA_PANORAMA_ENABLE`	`'true'`
`CLIENT_FIRECAMS_PANORAMA_ENABLE`	`'false'`
`CLIENT_POLL_SECONDS`	Derived from log analysis via `analyze_processing_wall.sh`

Locally, run publish-AICondorDeploymentConfigToAppConfig.ps1.
Locally, run deploy-AzureInfra.ps1:
1. Checks passwords are valid.
2. Builds the Bicep template and parameters file.
3. Runs a What-If plan (whatif-output-YYYYMMDD-HHMMSS.txt).
4. Runs Analyze-WhatIfOutput.ps1 to summarize the What-If plan.
5. Proceeds with the deployment.
Once deployment is complete, remove the old model version folder and merge the new model folder to the main branch.

Automated Deployment (Recommended)​

Prerequisites​

Usage​

What it does​

Running only part of the pipeline​

Other options​

After it finishes​

Confirming the Right Model Version Is Running​

Manual Deployment Steps (Fallback / Reference)​