Skip to main content

Triton and TensorRT Troubleshooting

Synopsis​

Intended end result: Restore model serving when Triton or TensorRT engine loading fails.

How this page gets you there: Collect targeted logs, validate existing engines, rebuild when needed, and restart only the inference chain.

Try VS Code tunnel first for log review and model file checks. It is usually faster and lowers the chance of command mistakes.

When tunneled into the VM, start with:

  • /mnt/datadisk/condor/model_repository/ai4gsmokefirev1/
  • Triton container logs from your VS Code terminal
  • /mnt/datadisk/condor/docker-compose.yaml

Fallback: VM command-line checks​

If you cannot use a VS Code tunnel, run the VM commands below.

Check whether Triton is healthy and why it may be failing:

cd /mnt/datadisk/condor
sudo docker compose --project-name condor -f docker-compose.yaml ps triton
sudo docker compose --project-name condor -f docker-compose.yaml logs --tail=200 triton
nvidia-smi

Capture detailed model load failures:

sudo docker logs triton_server --tail=2000 2>&1 | egrep -i "error|failed|ai4gsmokefirev1|tensorrt|plan"

Validate an existing TensorRT engine file:

Use a model version variable so you do not accidentally validate an older version.

MODEL_VER=11

sudo docker run --gpus all --rm \
  -v /mnt/datadisk/condor/model_repository:/models \
  nvcr.io/nvidia/tensorrt:24.05-py3 \
  bash -lc "trtexec --loadEngine=/models/ai4gsmokefirev1/${MODEL_VER}/model.plan --verbose"

Rebuild the engine in place if validation fails:

MODEL_VER=11

sudo docker run --gpus all --rm \
  -v /mnt/datadisk/condor/model_repository:/models \
  nvcr.io/nvidia/tensorrt:24.05-py3 \
  bash -lc "trtexec \
    --onnx=/models/ai4gsmokefirev1/${MODEL_VER}/model.onnx \
    --saveEngine=/models/ai4gsmokefirev1/${MODEL_VER}/model.plan \
    --fp16 \
    --inputIOFormats=fp16:chw \
    --minShapes=images:1x3x1024x1024 \
    --optShapes=images:16x3x1024x1024 \
    --maxShapes=images:128x3x1024x1024 \
    --useCudaGraph"

Restart the inference chain after rebuilding:

cd /mnt/datadisk/condor
sudo docker compose --project-name condor -f docker-compose.yaml restart triton client server_collect