On-Call and Out of Hours Work – A Problem We’ll Never Likely Solve

If you’ve never been on-call, it’s an experience. And it’s not just limited to the technology world where your weekends are shackled to a phone and your plans have to be ready to change at the drop of a hat. As the years crank on, you grow to dread the ring of your own phone – what have they broken now?

Often you’re expected to resolve whatever mess has occurred with little support. In one role, I lost a complete bank holiday weekend to an issue created by someone trying to be helpful and voluntarily working public holidays. When you’re told by the organisation your work has to be done out of hours as it’ll stop developers but they can interrupt your time off… it makes you realise just how unimportant you are in their eyes.

If you or the organisation you’re in has this mindset – it’s time for that long stare in the mirror and a re-assessment of your life. Everyone’s supposed to be on the same team. Lobbing a problem over the wall at 1729 as “ops problem” and legging it for the car park isn’t going to deliver the best results.

In another role, I ended up on a week-on, week-off rotation. The job saw regular out of hours calls which meant even the week’s grace between stretches was never long enough. Combine getting a telling off for some work you rushed out the door when you’re half-way up a mountain and £30 pcm to cover broadband expenses, you get to know your place.

It’s kinda funny, organisations rely heavily on on-call rotas to reduce costs and protect themselves from outages, loss of revenue and just general embarrassment. In the technology world, we’re still relying on the same sticky tape and dreams of decades gone by. Perl scripts and individual servers may now be fleets of “serverless” instances (there’s a server – you can’t see it but will get burnt by it occasionally) managed by IAC platforms but it’s all the same.

A bad bug or even underlying technology failure such as power outages or BGP routing blips can still trigger that 2AM call. Someone’s then expected to act as digital janitor and clean up the mess. Your job title might have DevOps or SRE in it but it’s still the same routine as we were doing with “data centres” in broom closets.

Over the years I’ve seen incidents last as long as days, hundreds of miles driven to datacentres and many late nights to fix things. The fact it’s so normalised is insane. Especially in smaller organisations where a handful of people handle everything from initial call to complete fix.

For some bizarre reason, developers often get to shirk responsibility here. Most places have a “you make a mess, you clean it up” approach but in the tech world we allow people to have successful careers with an attitude of “I’m too important for that”. Try that in other lines of work and see what happens!

One of the things often overlooked with on call rotas is the mental health aspect. 10+ years responding to out of hours calls did a real number on me. I even set myself a deadline – I had to be out of “support” roles by X date or it was time to check out of this world. It’s insane when I look back but one of my last out of hours calls lost me a bank holiday weekend. Everyone else got the time off – I had to fix a “critical” issue.

The amount of self-restraint it took not to use my physical access to smash up every bit of hardware I could get my mits on was incredible. All for what – a bit of pocket money? I’ll never get that time back.

What’s more, I could see the same resentment building in colleagues across a number of different organisations. Some of them had burnt out before and were on the way for another round. Even today, some of my joy and energy has gone – possibly forever.

Thankfully I’ve moved to roles where on call rotas are not a thing. Unfortunately, the reality is that the unstable stacks that make up the Internet will continue to rely on ruining sleep patterns, building resentment and unhealthy work/life balances while it remains cheaper than providing proper “round the clock” cover (i.e. shift work). Sometimes it seems only legislation or trade unions could change our approach.