Skip to main content

Starting, Stopping & Redeploying

You've seen how one camera's orchestration runs, cycle by cycle. Now the fleet questions: with ~1,400 eternal orchestrations, how does one get started or stopped, and — the subtle one — how does an operator changing a camera's config in the Admin API actually reach an orchestration that's already running and, as replay taught us, never re-reads its input? The answers are all lifecycle operations, and they're more mechanical than they look.

The one idea to hold on to​

There is exactly one orchestration instance per camera, and it's managed entirely through Durable Functions' instance lifecycle — start, terminate, purge — keyed on the device id. The engine never mutates a running orchestration. It replaces it.

That "one instance per camera" is literal: the instance id is the device id.

var options = new StartOrchestrationOptions() { InstanceId = deviceId };

Because the id is deterministic, every lifecycle operation can address a camera's orchestration by name — start it, check its status, terminate it — without looking anything up.

The lifecycle primitives​

Three Durable Functions client calls do all the work, and the engine wraps them in VapixOrchestrator:

  • Start — ScheduleNewOrchestrationInstanceAsync(taskName, acqDeviceDoc, options) launches the orchestration with the current device doc as its input. The engine then polls GetInstanceAsync until the runtime status is Running rather than long-polling, to avoid a gRPC timeout.
  • Stop — there's no "pause." Stopping is TerminateInstanceAsync (halt the running instance) followed, once it reports Terminated, by PurgeInstanceAsync (delete its history). Terminate ends execution; purge reclaims the history so nothing lingers in the task hub.
  • Inspect — GetInstanceAsync returns the instance's runtime status (Pending → Running → Completed | Terminated | Failed), which is how both the poll loops and the management endpoints know where a camera stands.

Start or stop is a status field​

How does the engine know whether a change-feed event means "(re)start this camera" or "turn it off"? It reads a status on the device doc's orchestration streams. When UpdateStreamOrchestrations runs, it branches:

  • STARTING → stop the old instance (terminate + purge), re-seed the entity, and start a fresh instance — a redeploy.
  • STOPPED → re-seed the entity to a stopped state, mark the monitoring row stopped, and stop the instance — the camera goes dark, deliberately.

So an operator turning a camera on or off isn't calling a "start/stop orchestration" API directly — they're setting this status in Cosmos, and the change feed turns it into the right lifecycle operation. Which is the real question of this lesson.

How an Admin API change reaches a running camera​

This is the mechanism the earlier lessons kept deferring, and it's worth seeing end to end, because it's counterintuitive: a running orchestration cannot be updated in place. From replay you know it carries its input forward via ContinueAsNew and never re-reads Cosmos; its stream configs live in the entity, seeded once at initialization. There is simply no code path by which a live orchestration notices a Cosmos edit. So a config change lands the only way it can — by replacing the instance. Step through it:

📡 Config → running camerastep 1/5
IntendedCosmos device doc10s
Orchestrationinstance = deviceIdA · 10s
RuntimeCameraState entity10s

Steady state. Camera orchestration “A” is running with a 10s single rate, matching the Cosmos device doc and the entity.

The full path, from click to effect:

The one moment runtime state is realigned to intended state is that InitializeEntity signal — between redeploys, the loop runs on exactly what it was given. This is why, throughout these docs, "config changes need a redeploy": the redeploy is the propagation.

Gotcha: control actions are eventually-applied, and only for active, in-env cameras

Two things follow. First, an Admin API "kick" or config change returns success once the Cosmos write lands — the redeploy happens moments later, asynchronously, as the change feed reacts. It is not instantaneous. Second, both the change-feed and manual paths gate on IsAlertCaActive && Environment == <this env> — a doc that's inactive or tagged for another environment is silently skipped. A camera that "won't take a change" is almost always failing one of these two, not the pipeline.

Doing it at fleet scale: the fan-out​

A single edit redeploys one camera inline. But the change feed delivers a batch, and a bulk update touching hundreds of docs, redeployed inline, would run hundreds of terminate/purge/re-init/start cycles one after another inside the change-feed handler, blocking it the whole time. So the handler branches on batch size: one change deploys inline; anything larger fans out — one message per doc onto the orchestration-redeploy queue, drained by a separate queue-triggered function whose instances run in parallel. Move the batch size and watch the path switch:

docs changed in this change-feed batch:
Cosmos change feed1 changed doc
Count == 1 → inline
Change-feed handlerredeploys it directly
each redeploy:terminate→purge→re-init→start

A single changed doc is redeployed right in the change-feed handler — fast, nothing to coordinate.

Recovering stuck orchestrations​

Eternal orchestrations occasionally need a human. The OrchestrationManagement functions are the recovery toolkit, built on the same lifecycle primitives:

  • Inspect — query current instances and durable-entity state to see what's running or wedged.
  • Reset failed / reset all — restart orchestrations that have faulted, without hand-editing Cosmos docs one by one.
  • Purge single / purge all — clear terminated or orphaned instance history from the task hub (the same purge a redeploy does per camera, run in bulk).

Reach for reset when orchestrations have failed but the config is fine; reach for purge when stale instance history is cluttering the task hub or an entity was left behind.

From the Microsoft docs​

Wrapping up​

Every camera is one orchestration instance keyed on its device id, managed through start (ScheduleNew), stop (Terminate + Purge), and status queries. A STARTING/STOPPED field on the Cosmos doc decides which. And because a running orchestration never re-reads its input, an Admin API change reaches it the only way it can — the change feed terminates and purges the old instance, re-seeds the entity from the fresh doc, and starts a new one. Redeploy is propagation; at fleet scale it fans out through a queue; and the management functions recover the ones that wedge.

That's the engine end to end. Next, how the loop actually talks to a camera: Talking to Cameras.