Kubernetes · Incident

Two power outages, one etcd quorum loss

When all three control plane nodes cold-boot at once, etcd doesn't come back on its own.

Storage · Backup

The backup that was silently failing for a week

A daily cron job had been failing quietly since a storage migration — and nothing was watching it closely enough to notice.

Kubernetes · Networking

The DaemonSet that wouldn't stay dead

k3s's built-in service load balancer kept regenerating a broken DNS DaemonSet after every restart — until it got its own watchdog.

Networking · Monitoring

When a firmware upgrade breaks your monitoring

An OPNsense firmware upgrade silently broke firewall state metrics — and every future GUI update was set to break the fix again.

Storage · Planning

The RAIDZ expansion that avoided a bigger cage

Growing a ZFS pool used to mean building a second pool and migrating data across. A newer feature made that unnecessary.