Patch Management in the Linux World

At one radio station I volunteer for, we’re mainly a Linux shop. Rivendell provide playout, Icecast streams the audio and it’s Debian on the desktop. With the small number of systems involved, keeping them patched is simple.
But what happens when you scale that out to hundreds of systems? Well, that’s a question that’s been bugging us at my workplace. We could let loose with automatic updates on all the systems. While we’d be patched quickly, we would run into problems with the playout side of things.
In an application like this, it’s better to have manual control of when systems are taken down for patching. Rebooting the file server because a new kernel is available would make things go a bit quiet on the radio. The solution here is to agree a maintenance window and run on CDs while the patching occurs.
That works for a very specific application. But what happens when we scale up to numerous “one-off” servers dotted around the country? We want to make sure everything’s patched to a sensible level and control the release of major changes.
Turns out that can be surprisingly difficult to do in Linux.
The CentOS/RHEL world has a tool called Spacewalk. It offers far more than patch management, but we’ve got tools like Puppet/Foreman looking after the other elements.
It surprised me in how well it tackled the job. It was possible to set up software channels synced to hold the latest CentOS 7 patches.
On the Debian side of the house, Ubuntu has Landscape. Again, like Spacewalk, it offers far more than patch management.
While Landscape is free for up to ten systems, rolling it out at any scale involved buying a full support package for your systems. Understandable for mission critical systems, but not so much for the sporadic boxes we see being built for testing. Admittedly, those systems should probably just be set to auto update.
When it comes to Debian itself, there’s not much on offer that I could find. It’s a shame really, but running your own repositories could solve that problem. Even if it involves a bit of legwork.
Beyond that, there are several commercial offerings that claim to handle patching for Windows, Linux and OS X. I’ve not really tried any of those out as we still operate each OS very differently.
The conclusion is that there’s suitable systems for assisting with patch management out there. However, you’ll need a very specific set of operating systems or a good source of funding to pay for the support.
With regards to the Debian/Ubuntu systems we have deployed, it’s going to be Spacewalk taking over the patch management. While we’re still experimenting with it at the moment, it’s the most promising option for the price.