I’m surprised there are sysadmins who miss obvious ways to improve their systems, like keeping a server’s /etc and cron jobs under version control. Cloning server configs from each other. Automated backups. Automated rotating and pruning of logs. Writing scripts to set up any software not covered by the package manager — or, better, building packages. Uptime and performance monitoring and alerts.

(The list goes on. This is my basic competency list. I got a lot of these techniques from watching a great sysadmin in action, reading some sysadmin blogs (curiously, sysadmins do not seem to blog nearly as prolifically as developers), pondering what programming skills and tools I can apply, and hours and hours of hair-tearing experience. I’m regularly surprised by how infrequently these things are done, even the really obvious bits of automation.)

I’ve long realized these techniques are about making system setups known and repeatable, like boiling down a test case in a programming language. What I didn’t recognize until I read the NixOS paper is that this is about the same kind of determinism that drives functional programming languages. Running systems are like imperative programs, and the core goal is minimizing the amount of potential state that can 1. exist 2. be unknown 3. blow things up. Imagine reversing all the effects of a function call in a procedural language with global variables kicking around: it’s the exact same problem as reversing the effects of upgrading a piece of software.

It looks like Nix has some great features I can’t imagine “imperative” package managers supporting well, like non-privileged users installing software. My sysadmin work is lately limited to the usual developer kvetching about one’s dedicated sysadmins, so I haven’t dived too deeply (I plan to play with it in a VM sometime soon). Looking at NixOS has left me with a desire to practice more functional programming. The cross-pollination from one field to another is invaluable.

The development box at work wasn’t letting me check anything into subversion — commits were just sitting there, not even timing out. In fact, so were updates. Something was seriously wrong.

I talked about it with a coworker and went to look at the box. Afew a few minutes of poking around, the problem became clear: someone installed a backup program that was trying to do some kind of fake-filesystem and wedged the box. Any process that tried to read from disk froze and couldn’t even be kill -9‘d.

And thanks to this odd little behavior, I could see three reboot processes frozen, presumably trying to read the shutdown scripts. So the person that wedged the box knew they wedged it but they just left it that way.

I got the coworker in the office to pull the plug on the box and it came up OK, but I edited /etc/init.d/arkeia to spit out the following note instead of try to start the backup program:

Dear whoever the hell decided to install arkeia:

You left the dev box wedged overnight, wasting at least an hour of two coders time to figure out what you did and fix it. And we know that you know you broke it, we could see that you tried to reboot and then LEFT IT FOR SOMEONE ELSE TO DEAL WITH rather than actually fix it.

Don’t be a jerk! Clean up after yourself!

Please talk to Jim and Harkins and explain why you left the box broken before you try playing with arkeia and wedge the box again.