Skip to main content

Processing Wall Analysis

Synopsis​

Intended end result: Quantify inference processing behavior and select safe operational polling thresholds.

How this page gets you there: Run analysis with configurable sigma trimming, interpret variance/outliers, and convert results into interval decisions.

The analyze_processing_wall.sh script extracts and analyzes inference processing times from the client logs. Use this to make data-driven decisions about the polling interval and detect performance regressions.

Run the analysis​

From the local repo (PowerShell):

./scripts/analyze_processing_wall.sh

Specify a custom log file:

./scripts/analyze_processing_wall.sh /path/to/logfile

Output as JSON for scripting:

./scripts/analyze_processing_wall.sh --json

Adjust the sigma threshold​

By default, the script uses a 2-sigma threshold (excludes ~5% of outliers). Use --sigma 1 to --sigma 6 to vary the sensitivity:

2-sigma (default, reasonable outlier rejection):

./scripts/analyze_processing_wall.sh --sigma 2

3-sigma (more conservative, include more outliers):

./scripts/analyze_processing_wall.sh --sigma 3

Get both results for comparison:

./scripts/analyze_processing_wall.sh --sigma 2 --json > analysis-2sigma.json
./scripts/analyze_processing_wall.sh --sigma 3 --json > analysis-3sigma.json

Interpret the results​

Example output:

processing_wall Analysis Summary
--------------------------------
Total Entries:         2431
Sigma Threshold:       2
Oldest (in 2-sigma):   2026-04-29 08:33:16.65
Latest (in 2-sigma):   2026-04-30 18:19:24.55

Overall Average:       22.6676s
Fastest (Min):         17.9600s
Slowest (Max):         53.9200s
Std Deviation:         3.5095s

2-sigma Range:         [15.6487s - 29.6865s]
Entries in Range:      2358
Entries Excluded:      73
Trimmed Average:       22.3505s

Key metrics:

  • Trimmed Average: The mean processing time within the selected sigma range. Use this as your baseline for polling interval decisions.
  • Std Deviation: Variability in processing time. Larger values suggest inconsistent load or performance.
  • Entries Excluded: Outliers beyond the sigma bounds. A high count suggests transient slowdowns or bottlenecks.
  • Time Range: Oldest and latest entries within the sigma range. Helps validate the analysis covers your operational window.

Decision guidance​

Set CLIENT_POLL_SECONDS based on trimmed average plus a safety buffer:

  • Trimmed average <= 22s → CLIENT_POLL_SECONDS = 30 (8s buffer)
  • Trimmed average <= 25s → CLIENT_POLL_SECONDS = 35 (10s buffer)
  • Trimmed average <= 30s → CLIENT_POLL_SECONDS = 45 (15s buffer)

Avoid setting the polling interval too close to the processing time to prevent request overlap and cascading slowdowns.