Far be it from me to throw buckets of cold water around willy nilly, but I can’t help feeling a strong urge to put a bit of a dampener on the current excitement surrounding the arrival of SPECpower_ssj2008. This is the new `green’ benchmark for servers that has been developed by the Standard Performance Evaluation Corporation (SPEC) with support and participation from AMD, Intel, Dell, Fujitsu-Siemens, HP, IBM and Sun Microsystems. 

All things `green’ are, of course, extremely sexy just at the moment. The industry has woken up to the fact that, despite most individual chips only consuming a few Watts each, a rack of servers still contain enough of them (and attendant fans, power supplies and associated materiel) to hoover up 10kW or more of electricity supply with remarkable ease. Anything they can do to cut back on this is obviously a good thing. Anything they can do to be seen to be cutting back on energy consumption (which may not necessarily be the same thing) is equally good. 

The arrival of the SPECpower_ssj2008 benchmark sets out to provide a tool that allows server makers to show that they are cutting back on energy consumption or, more relevant in a way, that they are at least providing the best possible performance `bang’ for each performance `buck’ consumed. But, honourable as this idea most certainly is in theory, I have some doubts as to whether it gets too close to achieving it in practice.  

It must be acknowledged that it is a good start to addressing the important issue of providing some form of meaningful measurement of something we know is important but is also surrounded by emotional rhetoric, hype and significant black holes in understanding. But the tests sets out in the benchtest have some weakpoints, not least of which is that they do not really exercise the I/O side of a server system. In addition, the workload is measured by using a specifically developed `standard’ software routine that currently only exercises server-side Java. The real world of datacentres involves a lot more than that.   

I’ve seen the benchmark described as being like MPG measurement in a car – divide the number of miles driven by the number of gallons used to get a useful measure. But, this assumes that a mile is a mile is a mile. In the world of IT, the actual question should be – are we talking Welsh miles, perhaps? Are we talking (worse still) Irish miles? Who hasn’t pondered the apparent flexibility of standard measurements when told “it’s only a mile further down the road.” 

This is a start, but much more is needed to help users determine what the real power/performance ratios are with their current systems and what makes the best alternative. For example, while I can see this benchmark becoming the latest hypemark target for hardware manufacturers, it may do little to demonstrate a far more important and unstated factor – that their current applications and operating environments are stupendously energy inefficient.  

In addition, as many business applications are in practice more dependent on fast and reliable I/O capabilities than shear brute power – the speed of the processor and the disk drives is often less important than the bandwidth of the comms channel between them, particularly when the disk is in a NAS or SAN and not in the same box as the processor. There lies a can of worms that not only impacts energy consumption but could have a huge effect – for good or ill – on performance. 

That alone, ironically, could be a component of a future benchmark which shows that the mainframe is, in fact, still the best option.