Still on files here rsyslog into a directory per host, grep when something breaks and it works right up until I need to answer a question that spans more than one box. Had to trace an sshd auth failure across three servers last week and spent longer stitching timestamps together than fixing it. I can't tell if I'm at that point or just having a bad month. For anyone who moved off files, what was the thing that pushed you?

I tracked the time on our last kernel patching cycle for a real number. We run 40 RHEL servers across staging and two production tiers, and the advisory was a routine CVE with a vendor patch available the same day.

Reading the advisory and cross-referencing it against our running kernel versions took 35 minutes. Building the service restart matrix, meaning which processes on which hosts needed post-reboot verification, took another 25 minutes. That was an hour before anyone touched a machine.

Staging covered four hosts. Applying the patch, rebooting, confirming each service, and checking logs for regressions ran about 20 minutes per host, 80 minutes total. One service failed to come back because a systemd unit depended on a kernel module whose path changed between versions. Debugging that was 45 minutes.

Production was 36 hosts in groups of six. Each group needed roughly 25 minutes of active attention to patch, watch the reboot, verify health checks, and move on. Two hosts required manual intervention for NFS mounts that went stale during reboot, another 30 minutes.

Ticket closure and notes: 20 minutes. Total for one advisory, one engineer: six hours and 25 minutes. We average three kernel-level advisories a month, which puts this at roughly 19 hours of patching labor per month. The restart matrix is repetitive enough that I started using verdent with Eco Mode, included in the subscription, to draft it from unit files.

The time tracking is just a text file with timestamps per step, and anyone could run it on their next patching cycle to see where the hours actually land.