Our power problem deepen as the power to our building blinked off again this morning for just over one hour. The best we could do was limp ahead on impulse power to safely shut all the servers down in a safe way. Eventually the power man with the special finger turned up and flicked a switch and the power blinked back into life. The finger-crossing routine of restarting everything began in earnest. Finally, the userbase could get back into their emails and documents and not a moment too soon - that was of course - until  the power went off again - and the man with the magic finger had already left.

The problem this time however was the backup battery power had taken a severe beating and the dreaded blinking lights of the UPS and that annoying beeping noise meant we had to shut everything down again in double-quick time. No time for niceties this time around, if a user had not saved their work it was too late, the servers have to come down and they have to do it now - no limping ahead on impulse power for us this time round. Not even venting the core into deep space would help this time - even if we could figure out how to do that without health and safety jumping up and down.

It seems somewhat ironic that the last two machines that can be shut down are the file server and the domain controller. This is ironic for two reasons. Firstly, they are the two key machines in use handling domain functions and managing the all important data store. Secondly, because the battery backup power completely drained just as we got to them (we had shut down twelve other servers in time) and they simply went off. Sure enough, around an hour later we crossed our fingers and they blinked back into life - we had gotten away with it this time.

The problem we face now is that if the power trips for a third time there is barely enough backup battery power to shut down one server, let alone fourt…

I have just finished restarting the computer system after yet another power outage. I have lost count how many power issues we have experienced with our local on-site sub-station (not to mention the number of hours lost), which apparently when installed was going to bring ‘huge benefits’. I keep telling the powers that be that they are playing with fire as equipment really doesn’t take kindly to regular power failures (and then sudden power startups) but the nagging problems remain and praying the power will only be off for a short time doesn’t seem to work anymore.

I have previously written about these power outages (Reflecting on the power outage) which have been going on for over a year now and have to reflect upon the latest bout in terms of the technology we have - which I will probably do in the car later (so if you pass someone on the roadside banging his head on the steering wheel it is probably me). Souping up the batteries on the UPS helps a little but they are there primarily to offer a brief window of extended use and to give enough time to shut everything down safely. Some of the previous reflections have been:

•  Being more vigilant with back-ups and recovery procedures
• A better view of the systems from my desk (so I don’t waste valuable power time trudging around to various computers and devices)
• Reliable mobile warning messages when the power goes off (and comes back on again) - primarily so I can get on with my life - still not found one!
• Emergency kit to deal with system issues - getting bigger all the time (the kit not me)
• Providing some emergency power to hot desks (so at least one or two users can get essential information for a while) - hooray done - but how many desk is enough?

At least the power is back up now and the systems are online so I can go and get a coffee out of the machine (hopefully that will not trip the power - fzzzt).

Some time ago someone told me they thought our web site should be “more Web 2.0″ and when I asked for confirmation as to what they felt Web 2.0 embodied I was met with a glazed expression then they said “you know … more up to date and modern looking like all the other Web 2.0 web sites”.  I could have gone on and asked for web site addresses for examples but I thought I would save them (and me) some pain and instead simply told them I would look into it - this at least seemed to keep them happy for the time being.

Now, I know what Web 2.0 means to me but this enquiry piqued my interest and I started looking into the Web 2.0 phenomenon for a series on my other blog (42 things about Web 2.0 in 2008, part 1 of 3) and after initial investigation it does seem to be a group of concepts and technologies that are painted with a very broad brush. It is even more interesting to see that even in the IT community the idea of Web 2.0 is very much a gray area although the term is bandied around at leisure.

My series is still running but so far I have found that Web 2.0 means:

• Interoperability
  Between applications or data, between presentation and data layers or between users of information
• Accessibility
  Information that is easy to access and deal with
• Usability
  Information that is useful, convenient and capable of being used (and re-used)

I have more work to do looking into Web 2.0 and I am curious what you think. So, in 2008 what does Web 2.0 mean to you?

Following up from my previous post When spam is not spam. Dare I say that we think we may have started getting spam under control. Fighting the ever increasing influx of spam (Guess What … SPAM is on the increase) is becoming a hefty challenge but we have come up with a combined approach that is starting to work. This combines an internal employee white-list with a trading partner white-list, a managed spam keyword black-list, an internally managed keyword black-list, a virus checker and several other scans all built into a multi-tier incoming email analysis system. It took some time to set up but the number of spam emails that we know we can delete automatically has increased whilst the number of false positives has dramatically reduced. In real terms, instead of having to trawl through almost ten thousand quarantined emails per day it is down to just over one thousand.

Mind you we now have a new problem. In addition to dealing with “spam that is not spam” we now have to find a way to defend against emails that “should not be spam but are spam”. Imagine this scenario, the internal email recipient is one the white-list, the trading partner is on the white-list, the attachment is an accepted format and the email passes the keywords check but it has a series of joke images and text that then get flooded around our business internally? How do we deal with that? This stuff gets stored in multiple places, archived, backed up, printed out, before being forwarded to ‘x’ number of additional recipients and don’t get me started on those “forward this email to ten of your friends” e-mails. What is that answer? Ask our big important customers to stop sending us spam? I can just imagine the conversation:

Customer: “We will forward our order by email shortly”

Us: “Thank you very much, oh and by the way…”

Customer: “Yes?”

Us: “Would you please tell your staff to stop sending us spam”

Customer: “What?”

Us: “Yes, we don’t mind the nice order but please stop barraging us with all the other rubbish”

It is a surprise that there are large well known companies out there that seemingly have very little policies on dealing with spam sent by their employees.

I posted an update recently on our ongoing virtualisation project Virtualising takes time but it is worth it which is moving along quite nicely now - albeit with a few hiccups along the way (and one or two long delays). One of the useful side effects of this project has been forcing us to review, rethink and address services that we were running. The old adage says “if it an’t broke don’t fix it” but setting time aside to look at what you are doing and more importantly why is proving quite refreshing - it’s almost like having a technological spring clean! Over time, programs, that are useful at the time, tend to get installed and used for a while then a new server gets put in and some of these older programs end up taking up valuable space and become their own problem areas. Time is not always available for looking at these small issues so they tend to get left so it handy to finally get to revisit them.

As an example, we used to host our own website and our own FTP server. Migrating the web site to a hosted service and using file transfer services such as sendspace instead of FTP have moved a lot of traffic (especially unwanted hack attempt traffic on the FTP connection!) away from our network. The side benefits of a less stressed firewall and a freeing up of bandwidth have been welcomed and with a number of other servers left to analyse it will be interesting to see what else turns up.

I was reading a post by Dennis Howlett “The state of green, 2008” which discusses a recent report “The State of Green Business, 2008“. The aim of the report is to explore the question “How are U.S. businesses doing in their quest to be greener and more environmentally responsible?” and offers a set of indicators to monitor and track progress. These indicators cover areas such as e-waste, energy efficiency, office space and power use and I found that it really helped to put into perspective all of the areas which can be improved.

As an exercise I spent some time reflecting on the information in the report and examining our IT operation and was surprised to see we were already conducting a number of green activities without necessarily realising it at the time - you can read about these in my post “technology and the environment - Green IT“. Obviously these are simply a starting point and the report highlights a number of areas where we might be able to make a greener impact.

Have you started to specifically address green as an issue and if so what activities are your technology department doing to help your business become greener?

I recently talked about problems with Backing Up (and Restoring) XP Drivers and have been trying to find a reliable method for hardware driver management. With so many different configurations in the field and including those in the server room - managing a rebuild is a much more complex task than it really should be. Unless you are lucky enough to purchase all your machines and software at the same time and perform regular technology updates and refreshes then chances are you, like us, have lots of different driver requirements. Much of the time this isn’t a problem. After all, while the machine is running adequately we don’t need to worry about it right? Only when a server fails do we often realise that we don’t have driver ‘X’ at hand and the machine won’t operate properly without it.

Virtual Server technology is started to provide some relief to this problem by layering the virtual drivers away from the physical drivers but where does the responsibility for driver management lie? It would be great to click a button and backup all the existing drivers and be able to utilise them during a rebuild but for this to work it needs the hardware vendors and the operating systems people to work together to implement an industry wide solution.

In the mean time I will continue my search but I wonder - how do you manage your drivers?

I’ve been running Wordpress 2.5 RC1 for around a week and have posted my Wordpress 2.5 RC1 - First Impressions on this long anticipated upgrade. There are a number of CMS platforms out there but Wordpress is quickly transforming into an extremely useful and adaptable platform. We have also been looking at using it internally within our organisation due to its ease of configuration and use. Sharepoint has always been the direction we have been heading in as I’ve written about several times but Wordpress is definitely growing on me! Two of my favourite things about the platform are the ease in which content can be published and the varied array of plug-ins which extend the platform into a number of useful directions.

Which CMS do you use and what do you like about it?

For some time now I’ve been working on a server consolidation project in our environment (Virtualisation Update … Web Server). This has been using virtualisation technology which seems to be in vogue at the moment. So, instead of a number of tower based servers we now have just a few HP DL380 64-bit servers running a virtual images which on the whole is working great. Rebooting machines is much quicker, allocating resources is easier, we are getting better value out of our machines and backing up virtual images gives us a sense of security that we could only dream of five years ago. It is also becoming easier to get all the machine images onto the same software release, see Upgrading Windows 2003 Server SP2 to R2.

One problem we have discovered however, especially being a 24 hour operation, is that when certain patches and updates have been installed on a host server it still needs rebooting which now takes down half a dozen servers at a time instead of just the one. The added trouble is that the virtual images receive updates too so now they often get rebooted twice, once for their own update and once for the host update. I imagine a good software and patch update methodology is the answer to this dilemma but it’s still surprising that with all this technology at hand - that a machine still needs to be switched off and on again in order to process an update.

We used to employ a system of automatically rejecting blacklisted incoming email senders. To do this we relied on services such as xBL. The problem we found was that from time to time genuine companies would, for one reason or another, find themselves on these blacklist sites and, even worse, they were not always aware of it. We found this out the hard way because we didn’t process (or deliver) several orders that were contained upon these incoming emails (Managing Incoming Email)- a hard lesson by any standard.

Of course in the ‘old days’ customer orders used to come in physically on paper but these days many are coming in solely via e-mail. I used to advise against this delivery mechanism for something as important as orders using the point that e-mail is not a guaranteed (or reliable) delivery mechanism. As time progressed though, customer requirements demanded that we accept these communications and it’s hard to argue against it when customer orders are at stake. One thing I have found more of recently is that genuine customer orders on incoming e-mail are starting to show up as potential spam. Again, we typically used to reject spam senders but it now looks as if we have to manually trawl through these - and as we get hit with thousands of these e-mails per day the task is rapidly becoming quite time consuming (Guess What … SPAM is on the increase).

We did try white listing certain supplier domains in the early days and configure our systems not to stop these communications in the hope that we might get a good stab at dealing with these problems but it didn’t take long for interesting post-cards, executables and screen savers to start getting picked up on other internal systems. Informing customers that we really don’t appreciate these things being sent to our staff is not a great way of maintaining a supplier/customer relationship and so we are regularly looking for other ways of dealing with these problems. It still surprises me that some large corporations seemingly allow their staff to have these sort of things on their computers and even more that they allow their staff to send them to trading partners using e-mail.

It seems we want to have the cake AND eat it. We want protection against spam but we also don’t want to miss genuine e-mails that may get flagged as spam. We don’t want e-mails from blacklisted senders but we don’t want to reject genuine orders either. We get ten times more spam than we did last year and no doubt next year the current figures will look like small fry in comparison.

What is the best way of dealing with incoming spam and blacklisted senders whilst also ensuring that valuable trading partner information doesn’t get rejected at the gateway?

Related Post: Sex, Drugs and the Spam Patrol