The engines are running at maximum Captain - she cannot take anymore

Permalink| Discussion:No Comments

Posted in Management on May 6, 2008 at 4:34 pm

Our power problem deepen as the power to our building blinked off again this morning for just over one hour. The best we could do was limp ahead on impulse power to safely shut all the servers down in a safe way. Eventually the power man with the special finger turned up and flicked a switch and the power blinked back into life. The finger-crossing routine of restarting everything began in earnest. Finally, the userbase could get back into their emails and documents and not a moment too soon - that was of course - until the power went off again - and the man with the magic finger had already left.

The problem this time however was the backup battery power had taken a severe beating and the dreaded blinking lights of the UPS and that annoying beeping noise meant we had to shut everything down again in double-quick time. No time for niceties this time around, if a user had not saved their work it was too late, the servers have to come down and they have to do it now - no limping ahead on impulse power for us this time round. Not even venting the core into deep space would help this time - even if we could figure out how to do that without health and safety jumping up and down.

It seems somewhat ironic that the last two machines that can be shut down are the file server and the domain controller. This is ironic for two reasons. Firstly, they are the two key machines in use handling domain functions and managing the all important data store. Secondly, because the battery backup power completely drained just as we got to them (we had shut down twelve other servers in time) and they simply went off. Sure enough, around an hour later we crossed our fingers and they blinked back into life - we had gotten away with it this time.

The problem we face now is that if the power trips for a third time there is barely enough backup battery power to shut down one server, let alone fourt…