Pushing patches to 2000 PCs

Ever since Microsoft started the whole "Patch Tuesday" thing, we've been very cautious about rolling out patches (it seems like every 3 months or so, there's one patch which breaks something). Here's what we do:

- Read the descriptions to see which "critical" patches are really critical (almost none), which are probably worth installing (about half), and which are about as much use as a lottery ticket - for example, protecting you from malicious, individually targetted Excel sheets (the other half).

- Add these patches to the post-installation procedure which runs when we reinstall Windows. We do this about 300 times a year on our 2000 PCs, so after a month we have 25 or so PCs running the patches and there's a reasonable chance that we would know if a new patch was breaking something major.

- We don't push patches to all the PCs unless we think there's a credible threat of a major malware outbreak. Since MS-Blast and Sasser, there hasn't been anything very damaging, but about twice a year we decide to take the "sky is falling" threats a little more seriously. It's a good test of our patch push system, anyway. (Update 2009-01-20: we started pushing MS08-067 on the day it appeared, because there seemed to be a credible threat. In the three months since then, we have managed to patch about 98% of our PCs. The limiting factor is that we have laptops which people don't bring into work very often, and a few PCs which are very rarely switched on, partly due to our organisation's sometimes Kafkaesque staffing rules.)

All that's fine, but we still found ourselves removing three or four trojans a week. Most of these are (of course) undetected by anti-virus scanners, either because they are rootkits or because the scanner isn't up to date (sometimes we identify malware before the excellent VirusTotal has even a single scanner which can detect it).

I started to think about generic ways to fix this - perhaps involving patching browser executable files so that the offending Javascript functions don't work, like I did 8-9 years ago with a tool called ATLAS-T to patch Microsoft Word to eliminate macro viruses - but then I read an article about the number of malware links served up by search engines, which said that most of the techniques used by these trojans exploit IE vulnerabilities for which patches exists.

So we decided to try an experiment. We applied all of the current patches (including the various ones for IE6) to 20% of our PCs (all of which run XP). Over the next couple of weeks we noticed that none of the trojans which appeared, were on patched PCs. We've now patched 95% of our PCs and trojans, etc have practically disappeared.

There was a downside, however. At least one Access application stopped working due to an authentification DLL being replaced "for security reasons", and since only about 5 PCs run this particular application, we didn't spot it until half the PCs were patched. The solution was to pretend that the patch was in place - our home-brew procedure looks for %SystemRoot%\KBnnnnnn.LOG and if it finds it, assumes the patch is installed, so we just create a fake log file on the PCs which run this application. But it will probably need rewriting before we push IE7 out to our desktops.

So, for once, the obvious advice ("keep your PC up to date with patches") actually has some use. Does it justify buying software to do it? Of course not (we do everything from distributed command-line scripts), but it may save you some malware cleanups, and the damage to the stability of your application platform by the patches may be sufficiently limited to actually make the whole thing worthwhile.