Silence is a bug
For sixty-eight days the intrusion feed was dead and every dashboard said quiet. A retired engine container was cooking the host for five days and every dashboard said quiet. Quiet was the problem. A feed that stops is not "nothing happened"; it is a claim with no evidence.
The intrusion feed died from a bug with no error message at all: the relay reading it tracked its position in a file that gets truncated in place rather than rotated, and once the file was truncated the relay's read position sat past the end of a now-smaller file, quietly reading nothing forever. No exception, no failed healthcheck, nothing to page anyone about. It looked exactly like a network with nothing happening on it.
The second failure was mechanical in a different way. An engine had been formally retired in its own source repository weeks earlier — the code removed, the deployment block deleted, a comment left explaining that the next deploy would remove the running container too. But no deploy ever happened, so the container kept running from its last built image, untouched by a retirement that existed only on paper. It spent its remaining days in a tight loop against a cache that had since been locked down: every call answered with an authentication failure, and the loop had no backoff, so it called again immediately. Across a few hours of uptime that added up to over a billion failed calls, and the extra load pushed the host toward its thermal safety cutoff more than once before anyone noticed the pattern.
So every ingest path now has a row with an expected cadence, and a page that turns silence into a finding: ghost containers, hot idle nodes, error storms, stale feeds, silent nodes, duplicate containers across nodes, images from a retired registry. The console does not absorb the alerting engine; it reads it. What it adds is the one question the alerting engine never asked: is anyone still talking?
The rule that survives both incidents is simple enough to enforce by habit: if you add a path that is supposed to produce events, you add its expected cadence to the same table the detector reads, in the same change. Retiring something is not a comment in a repository; it is a state the fleet's own inventory has to agree with, checked, not assumed. Absence of alarms was never evidence of health. It was only ever evidence that nobody was watching for the right shape of nothing yet.
← all logs