Found the culprits… one was the border router being power cycled unexpectedly, leaving filesystem corruption in `/var` that forced single-user mode manual `fsck` (a big #OpenBSD anti-feature here).
The other, was this little gem. Two M.2 SATA SSDs in two different servers deciding that last night would be their last ever. These are 240GB WD Green SSDs, bought purely as boot drives back around 2020 or so. One installed in the M.2 slot on a Intel NUC, the other into a M.2 USB enclosure. They've run these machines almost 24/7 for about 5 years.
I've held my nose and bought some Silicon Power 128GB drives, as that's all Umart has in M.2 SATA form factor. (Well, I can go bigger, but I barely use 12GB.) Reviews were terrible, lots of reports of early failures, maybe I'll be lucky, or maybe these will have the life expectancy of a tray of Tim Tams at a weight-watchers weigh-in. We'll see.
Well, you picked a fine time to leave me boot disk…
Along with your comrade, blowing my OS away…
My pair of downed servers, 12TB of replicas…
now my Ceph cluster won't heal…
…you picked a fine time to leave me, boot disk!
#HomeServer #Ceph
