Why deploys pull, never push
The question that decided this design was not "how do we deploy faster" but "what does a compromised build job get to do." The honest answer, before this shipped, was: too much. Any workflow triggered by any branch push could, in principle, hold a credential capable of reaching a production server over SSH. A developer editing a workflow file on a feature branch would have been enough to exfiltrate that credential. That is not a hypothetical worth living with once you have named it.
So the build plane lost the ability entirely, on purpose, rather than being told to be careful with it. CI now does exactly three things: lint and test the code, build a container image tagged with the exact commit it came from, and push that image to a private registry. It cannot SSH anywhere. It cannot know a production host exists. That is the whole build-only rule, and it is enforced by deletion, not by policy.
On the other side, a small daemon lives on the runway — the plane where applications actually run — and it is the only thing on the estate that pulls. It listens for a signed notification that main moved, waits for the corresponding image tag to actually exist in the registry, pulls it, resolves its content digest and pins to that digest locally rather than trusting whatever the registry's own "latest" tag happens to point at right now, brings the container up, and watches its health endpoint for a short window. If health fails, it rolls back to the previous known-good digest automatically and reports the failure back onto the commit that caused it, so the person who pushed it sees the result where they'd look for it.
The daemon also reconciles itself on every boot, independent of any webhook: it walks every product it knows about, compares the latest commit on its main branch to what is actually running, and queues up anything that has drifted. That one behaviour quietly closes two different failure modes at once — a webhook that never arrived, and a deploy that was in flight when the daemon itself restarted — without needing a special case for either.
Onboarding a new product onto this pattern turned out to be almost anticlimactic: the unified build pipeline every product already runs was already producing the right tagged image, so wiring a new product in was one block added to a configuration file and a restart, no application code touched. The whole fleet — more than ten products — was cut over to this pattern in a single session, ending with the literal deletion of the one credential set that could have reached production from the build side. Deploys pull because the alternative asks a build job to be trusted with something it never needed in the first place.
← all logs