Kubernetes, evaluated and declined
Two runner incidents in a single day were enough to put an orchestrator on the table seriously: a stuck cleanup job had clogged the build queue's entire capacity twice before lunch, and a separate process was later found to have been silently dead for weeks without anyone noticing. Reaching for Kubernetes at that moment would have been an understandable reflex. It got a real evaluation instead of either a reflex yes or a reflex no, and the answer was written down rather than left as a shrug: not now, and here is exactly when.
The actual pain, once it was named honestly, was small and boring: the build runner's own concurrency limit was too low for the load it was carrying, a class of job could wedge and hold a slot indefinitely, and the disk behind the registry was uncomfortably full. None of those are scheduling problems. An orchestrator solves the problem of placing workloads across many machines and healing them when a machine dies; the estate runs on one physical host, with its planes already cleanly separated by convention and infrastructure-as-code rather than by a scheduler's own enforcement. Adopting Kubernetes here would not have fixed a full disk. It would have meant rewriting every deploy script, every routing rule and every compose file the estate has, all at once — the exact shape of uncontrolled, fleet-wide cascade the estate's own change-management rule exists specifically to prevent, in exchange for solving a problem that was not the one actually causing the incidents.
There is also a portability argument the estate cares about that an orchestrator would have worked against rather than for: the goal is infrastructure that can be picked up and rebuilt from its own definitions on a different machine if it ever had to be, and that portability today comes from the infrastructure-as-code and compose files being simple enough to read end to end. An orchestrator would not have enhanced that reproducibility; it would have replaced it with its own, different, heavier reproducibility story, for a problem the estate does not have yet.
So the fix that actually shipped was the boring one: raise the runner's concurrency, fix the class of job that could wedge, and relieve the disk pressure. And the decision named its own expiry conditions instead of staying open-ended: reconsider when a second physical host needs workload scheduled across it, when the fleet grows past a handful of hosts needing coordinated placement, or when a specific engine needs real, automatic autoscaling that queue-based workers cannot provide. If any of those become true, the recommendation on file is narrow on purpose — bring in a lightweight, Kubernetes-class scheduler for the compute and engine layer specifically, never as a wholesale replacement for how the rest of the estate already runs.
← all logs