Firms full of dirty data

As much as 30 per cent of data held by companies could be “dirty” in some way, according to a BCS award winner.

Up to 30 per cent of an organisation's data could be "dirty", according to a top informatics researcher.

Delivering a lecture after winning the British Computer Society's (BCS) Roger Needham Award, Professor Wenfei Fan said that businesses and their customers could be negatively affected by "dirty data" any information which is inconsistent, inaccurate, incomplete or out of date. "Poor quality data can give us trouble the problem is everywhere," he said.

According to Fan, Australia has 500,000 dead people with active Medicare cards, while the US Pentagon's bad data quality led that organisation to attempt to send over 200 dead soldiers back to Iraq.

"In the UK, we're doing no better," he said, saying this country has issued 81 million national insurance numbers to a population of only 60 million.

But it's not just the public sector. Fan said that in a customer database of over half a million records, 120,000 become invalid within a year. And, the error rates for industry range from a fairly accurate one per cent to as high as 30 per cent.

And such dirt can lead to high costs, he said. Among other examples, he cited the case of Lehman Brothers, which inaccurately entered 300 million for a 3 million trade, taking 300 billion off the FTSE 100.

It's not just a problem for high finance, but for retail, too. Fan noted an example from Dell, which sold 15,000 computers in Chile for 79 when they were actually worth 303.

Fan claimed dirty data costs US businesses as much as $611 billion (412 billion) and US customers as much as $2.5 billion (1.68 billion) each year. "Real life data is dirty, and dirty data is costly," Fan said.

While he admitted 100 per cent accuracy was essentially impossible, Fan called for better tools to cross-reference databases to detect incorrect data.

Fan is chair of web data management in the School of Informatics at the University of Edinburgh. The Roger Needham award is given by the BCS and Microsoft to a UK-based researcher within ten years of their PhD for their contribution to computer science.

The entertaining lecture can be viewed [a href="

needham_2008" target="_blank"]here[/a]; click option eight' to go directly to Fan's lecture.

Featured Resources

Modern governance: The how-to guide

Equipping organisations with the right tools for business resilience

Free Download

Cloud operational excellence

Everything you need to know about optimising your cloud operations

Watch now

A buyer’s guide to board management software

How the right software can improve your board’s performance

The real world business value of Oracle autonomous data warehouse

Lead with a 417% five-year ROI

Download now

Most Popular

How to boot Windows 11 in Safe Mode
Microsoft Windows

How to boot Windows 11 in Safe Mode

6 Jan 2022
How to speed up Windows 11
Microsoft Windows

How to speed up Windows 11

7 Jan 2022
Solving cyber security's diversity problem
Careers & training

Solving cyber security's diversity problem

5 Jan 2022