So, I've been given charge of a suite of servers. Some of them scratch machines, some dev boxes, and one or two absolutely mission critical.
None of them have much of a management scheme in place and I need to get a handle on what changes have been made to them and what they do. More importantly, I need to record this is a way that will be accessible to those that come after me.

We have an internal wiki, and there are many clues to be gained from it, however it would be an exaggeration to say that those docs contain what I need to know.

Here are some strategies I'm compiling based on two decades of accidental sysadminning and good ideas I've learned by example from the huge number of different infrastructures I've touched.

Get system config files into source control

We used to do this when I was managing a number of University servers. At that time, CVS was the only available tool, but it served to give the other sysadmins visibilty and roll-back ability of important files. there was a CVS folder inside /etc/ and we could track changes there.
These days I'm using git, and it turns out that the 'local repository' feature of git is exactly what I need. Previously we had to have a remote repository and organise access to it etc, something that was tricky when the very thing you were wanting to track was getting networking configured. Also, CVS and SVN both leave files everywhere (I used to like that) but I didn't want that all over system directories. A global git tracker running from the root of the filesystem looks like it will help a number of things

sudo su root
git init
# figure out which files deserve watching - I'm going for a really sparse selection of things I know have been touched
git add /etc/exports /etc/group /etc/hosts /etc/network/interfaces /etc/motd.tail /etc/dnsmasq.conf /etc/passwd /etc/sudoers /etc/sudoers.d /etc/apache2/
git add /etc/apt/sources.list /etc/apt/sources.list.d/ /etc/udev/rules.d/70-persistent-net.rules

What to track?

My choices here indicate things that I want to keep an eye on. In this case, adding a new user or group is a note-worthy event, and if I did a status check I'd like to know that it had happened.
If you do not recognize any one of the files on that list, you should happily not include it, as tracking it may not be something you need to care about.

I guess we could add /etc/* .. but there's a lot of untouched stuff there so I don't want the noise. My git repo will therefore be able to tell me directly *these* files are the ones that matter on this machine - this is where changes have been made. I deliberately am ignoring the bulk of files that were auto-installed and left at the factory setting.

Consideration should be given to whether some of those files are 'sensitive' - such as places that passwords may be stored. A changed password that was stored in a script somewhere is in fact one of the most important things a systems administrator needs to have logged ... and seeing as these strings may already exist on the system, I'm not sure how evil it would be to track changes to them.

An additional method to find track-worthy files can be to run a scan for files in the etc directory that have been modified since initial install. This will be less help on older machines with years of history, but if you want to find things that have been touched in the last few months

find /etc/ -type f -mtime -30

may give you a modest starting point of files to consider.
Look out for a few 'dynamic' files like /etc/mtab that we (probably?) won't gain anything by tracking

Moving forward of course, we need to remember to add things to this list as services get added or tweaked. Eg, at the point in time where we start to mess with /etc/mysql/my.cnf - this should get tracked *before* edits begin.

Big brother is logging you

I'm going for a slightly paranoid "Who's been messing with this box recently?" approach due to a slightly fragmented and unstructured team. I need to spot any cowboy stuff. There just is too much undocumented wierdness going on. Some folk need watching, just so I can undo their damage.

To this end, I'm also going to track any actions done as root, who has been logging in as root, and who has been using sudo.

git add /root/.bash_history
git add /root/.ssh/authorized_keys

Using version control to monitor logfiles may be over-the top, but I'll try it and see if it helps. Nobody should be logging in as root regularly, and alll sudo actions are 'significant' to some extent, so the noise may be good.

There will be ways to circumvent this logging, but I just want to catch people making mistakes or doing unexpected things, I don't feel I could claim to thwart a true penetrator.
sudo logging normally goes to syslog, it appears. We can split it off by editing the /etc/sudoers file (with visudo) and adding a line (Here is a handy readable tutorial on using /etc/sudoers)
# Log everything folk do as sudo, because it may be important to the history of this machine
Defaults logfile=/var/log/sudo.log
Now we can start tracing that file
git add /var/log/sudo.log
Which will have the effect of *anything* anyone ever does as sudo will count as a state change on that machine that gets logged.

If this becomes noisy, then folk are using sudo too damn much!
Take care, if you have any automated processes that use a locked-down sudo for their own daemon tasks, you will start to hear about it. See the sudo man page for the syntax to adjust for that.
I added
Defaults>vbox !logfile,syslog
Defaults:aegir !logfile,syslog
as these are two daemons that have sudo exceptions in place. (This first allows any user on the machine to become 'vbox' for the pusposes of managing virtuals, the second is the aegir daemon that occasionally needs to restart apache and has limited rights to do so.

I've got what I want, but git status is still full of noise - it will list all untracked files (pretty much everything else anywhere) by default. This doesn't help me focus on the important stuff. Git has the option to turn this off which is exactly what I expected.
git config status.showuntrackedfiles no
... and I've won!!