Engineering note · 6 of 6

Owning a role isn't being it.

Postgres row-level security only actually filters rows for a role that cannot bypass RLS — which most application DSNs, superuser included, can. Runkite's fix is a dedicated, restricted role the connection switches into per tenant request. The one-line detail that made this fail on a real, least-privilege production role — and didn't show up in a superuser test DSN — is worth writing down.

Why RLS needs a second, weaker role at all

A Postgres superuser (and, by default, the owner of a table) bypasses row-level security entirely — FORCE ROW LEVEL SECURITY exists specifically to close that hole, but it only applies to roles that don't already have BYPASSRLS. Most real deployments connect as a superuser-ish application role for simplicity, which means RLS policies would silently do nothing unless the connection deliberately drops into a role that actually respects them.

Runkite's opt-in RUNKITE_POSTGRES_RLS creates exactly that role (runkite_app, NOBYPASSRLS) once at startup, and SET ROLEs into it on every tenant-scoped connection acquire — system-context work (migrations, admin queries) stays on the original login role and sees everything, same as before.

The gotcha: creating a role and being able to become it are different things

Postgres 16 added a convenience: a non-superuser with CREATEROLE who creates a new role is automatically granted membership in it. That sounds like it should be enough to let the creator immediately SET ROLE into what it just made — but membership comes with three independent sub-options (ADMIN, INHERIT, SET), and the automatic grant only turns on ADMIN. Without the SET option explicitly granted, SET ROLE fails — permission denied to set role — on a role you just created and technically "own."

The tempting "just grant everything" fix — WITH ADMIN TRUE, SET TRUE in one statement — fails too, but with a different, sharper error: ADMIN option cannot be granted back to your own grantor. The auto-grant at creation already recorded the new role as your grantor for that membership; re-granting ADMIN back to yourself on the same membership is a circular grant Postgres refuses outright. SET alone, with no ADMIN, has no such conflict and is the only privilege actually needed here — nothing in the bootstrap re-grants runkite_app onward to a third role, so admin option was never buying anything.

Why a superuser test DSN hides all of this

A superuser can SET ROLE to anything, membership grant or not — the membership/SET-option check is skipped entirely for superusers. A test suite that only ever exercises this against a superuser connection (the common shape for local Postgres and CI service containers) will pass cleanly while being completely unable to detect that the exact same code fails for the realistic, least-privileged production role the feature was built to harden in the first place. The fix here wasn't just the one-line grant — it was adding a second test DSN tier (non-superuser, CREATEROLE-only) specifically to stop "passes in CI" from meaning "works in production."

If a security feature is only ever tested against the most privileged credential available, it is only proven to work for the one deployment shape that needed it least.

Fencing Subscribe-before-enqueue createRunCtx Poison pill FinOps holds Postgres RLS grant