The life of a device job
What states can a job be in, and how does it get unstuck?
Eight states in three lanes: the path that works, the two ways of waiting, and the two ways of stopping for good.
Drag to pan, scroll to zoom, click an element to trace it.
Open full screenWhat to look for
- The loop from retry-scheduled back to queued — that is automatic recovery, with no human in it.
- The "fencing token" tag on leased.
- Which two states are marked immutable, and why those two.
Read the full explanation
A job moves through queued, leased, running and then to one of four endings. The lanes separate three different meanings so they are not confused: the happy progression, states that are waiting on time or a person, and states that are finished.
The lease is the mechanism that makes concurrent agents safe. When an agent takes a job it gets a lease token, and only the current holder of that token may report on it. If an agent stalls and its lease expires, a later report is refused — the job has moved on without it. This is what stops two agents publishing the same post because one of them was slow.
Waiting is not failing, and the diagram keeps them apart. A retryable failure schedules a retry with a backoff of one, two, then four minutes and re-enters the queue by itself. A security challenge or an expired login does not retry, because retrying cannot help — it parks the device and asks for a person, saying what they need to do.
Nothing is left stuck. Expired leases are reclaimed and their device released, jobs nobody claims expire after twenty-four hours, and succeeded and cancelled are permanently immutable — not even an operator retry can reopen them.
Operators reading the executions screen, and engineers changing retry behaviour.