/ BLOG / Change management, documentation and automation

I’ve recently taken on a new role at work, and as part of that I’ve now got a big thing for change management and documentation.

I should cover a bit of background. At work we’re a bit different from normal IT departments. Mostly because we’re not a department, although we are treated as such by many of our customers. Ultimately we look after multiple, distinct, systems in multiple areas of business, in multiple locations - none of which are inter-linked at all. This makes it exceptionally important to document and to pass on information. It’s unacceptable for us to say “X has gone to lunch, he’s the only guy who knows your system… Can it wait?”.

One thing that we’ve always done is to document everything on our online helpdesk software. Even if the customer phones it in, it has to go into the helpdesk. This is great for change management and finding culpability, but it’s terrible for keeping configuration files, and information on the overall architecture of systems. Over the last year we’ve been supplementing this with a wiki (the excellent Dokuwiki to be exact) to help record this sort of information. Combined with regular group briefings (read: informal chats) it’s generally been working reasonably well, especially now we’re coming to the end of a major re-organisation of the data in there.

My main issue with what we currently have is keeping up to date configuration files for routers, switches, daemons, and so on. Particularly in combination with lots of rapid changes. Its all well and good to have procedures stating that “you must ‘check in’ the most recent change”, but if it’s too busy and it gets forgotten then you’re screwed. I really want to automate this process. There are some tools out there for this; Kiwi’s Cattools, Ziptie (abandoned, which I realised far too late after dicking about with it for an hour and wondering why some things didn’t work), RANCID.

These’ll work fine to varying degrees, but here’s my niggling problem - I’d love to be able to stick something else next to whatever system we deploy, in order to push configuration changes back. With RANCID I can do this, but we’ve still got very much of an anti-*ix sentiment, and although it is changing very slowly, in the short-to-medium term it would cause the same problem that I’m trying to get rid of - knowledge partitioning. Hiring someone else with the knowledge we need isn’t an option at the moment (we’ve just taken on an additional member of staff who doesn’t have the knowledge or skills).

It’s got me thinking. What trick am I missing here? I know I should be just worried about configuration files right now, but the part of me that loves hacking something together really wants to find or to put something together to solve both problems in one go. However, realistically this isn’t something I have the time to do right now.

How do you do it?