Engineering note · 5 of 6

Count the spend before it happens.

A daily USD/token cap checked only against finished runs has a hole exactly the width of your concurrency: N runs can all pass the check at once, before any of them reports a token back. Runkite closes that hole with an optimistic hold — a guess reserved at create time, settled for real at the end.

The gap a terminal-only check leaves open

Terminal usage — the actual token counts and USD estimate — is only known once a run finishes. If admission only asks "how much have finished runs cost today," a burst of concurrent creates all see the same pre-burst total and all pass, because none of them has reported anything back yet.

cap = $10/day, spent-so-far = $9.50, avg run ≈ $1

terminal-only check:
  run A create → checks $9.50 < $10 → admit
  run B create → checks $9.50 < $10 → admit   (A hasn't finished yet)
  run C create → checks $9.50 < $10 → admit   (still $9.50 — nothing settled)
  → three runs land, cap breached by up to 3×avg before any of them reports back

with a hold:
  run A create → reserve $1 estimate → checks $9.50+$1 < $10 → admit, hold open
  run B create → checks $9.50+$1(A)+$1(B) = $11.50 ≥ $10 → denied
        

What a hold actually is

At run-create time, if finops.reservation is configured, the plane writes one row to a usage_holds table: an estimated USD/token amount for that run, scoped to the same tenant/agent the real budget check uses. Every subsequent admission check for that scope adds up open holds alongside terminal usage — so the reservation counts toward the cap immediately, not after the fact. When the run finishes, the real terminal usage event is recorded and the hold is released in the same transaction path — the estimate never double-counts against the real number.

• Reservation size is configurable per agent — a coding agent that burns 50k tokens a run and a classifier that burns 200 need very different estimates; reservation.agents overrides the default per tenant_id/agent_id key.
• Holds only guard USD/token caps, not max_runs_per_day — run-count admission already counts in-flight runs directly (CountRunsSince), so adding held runs on top would double-count the same run twice.

The failure mode a hold introduces — and how it's closed

An optimistic reservation is a promise to release later. If the runner that opened it crashes before reporting a terminal status, that promise is never kept — the hold sits open forever, silently eating into tomorrow's budget along with today's, because nothing ever releases it.

run creates → hold opens ($1, tenant=acme)
runner crashes mid-execution → no terminal event ever arrives
                              → hold never releases on its own

hold_ttl sweeper (background loop):
  finds holds older than hold_ttl with no matching terminal usage event
  → expires them → cap returns to its real, un-inflated value
        

This is the same shape as generation fencing's problem, one layer up the stack: something optimistic was reserved on the assumption the reserver would report back, and the system needs an explicit answer for "what happens when it doesn't." Here the answer is a TTL sweep instead of a fencing token — but set hold_ttl shorter than your longest real run and you'll expire (and under-reserve for) a job that's still legitimately running. Size it above your slowest expected agent, not your average one.