Ops

Day-2 operations — what happens when a runner crashes, how to stop something gone wrong, and where secrets actually live. None of this needs SSH access to a box; it's all plane behavior or an Admin/API action.

What happens when a runner crashes mid-job

Every dispatched job has a live heartbeat — the runner pings the plane roughly every 2 seconds while it works. If those heartbeats stop (the process crashed, the pod got OOM-killed, the node disappeared), the plane notices within about 6 seconds and reclaims the job automatically: it goes back on the queue for a different runner to pick up, and a "generation" counter on that job increments.

The generation counter is what makes this safe instead of dangerous. If the original runner wasn't actually dead — just slow to report, or its heartbeat got lost on a flaky network — and it finishes the job late and tries to report a result, the plane sees that its generation number is now stale and rejects the write. This is called fencing, and it's what stops a "zombie" runner from overwriting the result a second, legitimate runner already produced. You don't do anything to enable this — it's always on.

One thing to watch: a job that keeps crash-looping (bad graph code, a dependency that's always down) would otherwise reclaim forever. There's a ceiling — reclaim.max_retries in langgraph.json (default 3, or set RUNKITE_RECLAIM_MAX_RETRIES) — after which the plane stops retrying and marks the run as failed instead of endlessly recycling it. Set it to 0 for unlimited retries if you'd rather it keep trying forever (not recommended for anything but a very short-lived debugging session).

Stopping a run that's gone wrong

For one specific run: cancel it from Admin → Runs or via the Agent Protocol API. The runner sees its assignment marked non-current and should stop producing side effects on its next check — well-behaved agent code checks the cancellation signal between steps, so a very long single LLM call in flight may still finish that one call before noticing.

For something bigger — an entire agent or tenant misbehaving — use Admin → Kill switches instead of cancelling runs one at a time: it stops new runs from starting and can cancel every in-flight run in scope in one action. See Kill & break-glass for the full walkthrough, including the opposite tool — a time-boxed policy bypass for when you need something to work again right now while you fix the actual root cause.

Where secrets actually live

Provider credentials (a Salesforce OAuth secret, a GitHub token) live in your connector config or a secrets manager — never copied into every agent's process environment. When an agent needs to call out, the plane mints a short-lived, run-bound session at the moment it's needed and hands back only that. See Secrets for the three ways to reference a credential (env var, mounted file, or HashiCorp Vault) and Connectors & HITL for the full setup.

Reclaim and hooks that run on a schedule (so you know what to expect in logs)

None of these need manual triggering — they're background loops inside the control plane process. They're listed here so a "why did this take up to 15 seconds to show up" question has an obvious answer instead of feeling like a bug.

Reference: docs/ops-runbook.md · Fencing note · Poison-pill note · Admin UI guide