Is the benchmark for benchmarks that they are no longer worth attempting? The question arises because far less action or brouhaha has occurred in the world of the new TPC-E benchmark than might have been expected. Both Dell and IBM have recently joined Unisys in publishing first results but, strangely, neither has made much fuss about them.

Back in July Unisys became the first server vendor to announce results against the new tests. These set out to simulate the online transaction processing workload of a brokerage firm. This is aimed at giving a much more relevant impression of how a server and database combination will perform in the real world than the old TPC-C benchmark, which while not discredited, has ended up fairly tarnished by rumours and innuendoes about the ways databases and hardware could `tuned’ to improve results.

Being first at anything like a benchtest, of course, will always leave any company on a hiding to nothing. So for a while Unisys had nothing to compare itself against, so no one - even Unisys - had the remotest idea of the relevance of the results. The company has an enviable reputation in many areas, particularly in building infrastructures and systems geared to the specific needs of vertical market sectors. As `a consequence it has usually eschewed the publicity-generating game of `who can build the real screamer’? So could the silence and Dell and IBM about their results have any relationship to that? The answer, at least in part, is yes.

It transpires that building a screamer is what Unisys appears to have done, at least in these early days of TPC-E. Its ES7000/one server came in with a performance of 661 transactions per second (tpsE), running a Microsoft SQL Server 2005 Enterprise x64 Edition database. Dell currently lies second at 220 tpsE, while IBM is third at 170 tpsE. Both of these were running the SP2 version of the same database. This result is no doubt coloured by the fact that the Unisys Server used 16 dual-core processors, while Dell used four and IBM only two.

However, when it came to price/performance, Dell and Unisys reversed positions. Arguably a more important metric for many enterprise users, this showed Dell at $1,020 per tpsE and Unisys at $1,777 per tpsE. IBM came third at $1,898 per tpsE, which might certainly explain its backwardness in coming forward.

The upshot of all this? Well, I do wonder if the reluctance to crow about these things publicly may indicate that there are now some doubts amongst vendors as to what benchmark results actually prove. For example, it is possible to read the price/performance figures to show that Dell can make a 4 processor server that is faster and cheaper than IBM or that Unisys makes bigger, more expensive ones – not much new there, then.

The performance figures can be read to show that the performance, per processor, of SQL Server drops off the more processors that are added, but that is hardly a startling discovery. And without then factoring in the different operating systems each is using – Windows Server Datacenter Edition for Unisys, Server 2003 Enterprise Edition x64 SP1 for Dell and the SP2 Version of that for IBM, getting anything concrete out of the figures gets to be a complex task. Can we say for definite, for example, that ipso facto the SP2 version is slower than SP1? I’m not sure I’d stake my life on it.

For IT managers, maybe this just demonstrates that the old ways of gauging these important issues are still best – get the sales reps to take you out to dinner and have `Scepticism’ set to `High’.