Firms full of dirty data

As much as 30 per cent of data held by companies could be “dirty” in some way, according to a BCS award winner.

Up to 30 per cent of an organisation's data could be "dirty", according to a top informatics researcher.

Delivering a lecture after winning the British Computer Society's (BCS) Roger Needham Award, Professor Wenfei Fan said that businesses and their customers could be negatively affected by "dirty data" any information which is inconsistent, inaccurate, incomplete or out of date. "Poor quality data can give us trouble the problem is everywhere," he said.

According to Fan, Australia has 500,000 dead people with active Medicare cards, while the US Pentagon's bad data quality led that organisation to attempt to send over 200 dead soldiers back to Iraq.

"In the UK, we're doing no better," he said, saying this country has issued 81 million national insurance numbers to a population of only 60 million.

But it's not just the public sector. Fan said that in a customer database of over half a million records, 120,000 become invalid within a year. And, the error rates for industry range from a fairly accurate one per cent to as high as 30 per cent.

And such dirt can lead to high costs, he said. Among other examples, he cited the case of Lehman Brothers, which inaccurately entered 300 million for a 3 million trade, taking 300 billion off the FTSE 100.

It's not just a problem for high finance, but for retail, too. Fan noted an example from Dell, which sold 15,000 computers in Chile for 79 when they were actually worth 303.

Fan claimed dirty data costs US businesses as much as $611 billion (412 billion) and US customers as much as $2.5 billion (1.68 billion) each year. "Real life data is dirty, and dirty data is costly," Fan said.

While he admitted 100 per cent accuracy was essentially impossible, Fan called for better tools to cross-reference databases to detect incorrect data.

Fan is chair of web data management in the School of Informatics at the University of Edinburgh. The Roger Needham award is given by the BCS and Microsoft to a UK-based researcher within ten years of their PhD for their contribution to computer science.

The entertaining lecture can be viewed [a href="http://emea25537091.emea.acrobat.com/bcs_

needham_2008" target="_blank"]here[/a]; click option eight' to go directly to Fan's lecture.

Featured Resources

Managing security risk and compliance in a challenging landscape

How key technology partners grow with your organisation

Download now

Evaluate your order-to-cash process

15 recommended metrics to benchmark your O2C operations

Download now

AI 360: Hold, fold, or double down?

How AI can benefit your business

Download now

Getting started with Azure Red Hat OpenShift

A developer’s guide to improving application building and deployment capabilities

Download now

Recommended

Best ransomware removal tools
ransomware

Best ransomware removal tools

22 Jan 2021
Hackers publish over 4,000 files stolen from SEPA in ransomware attack
Security

Hackers publish over 4,000 files stolen from SEPA in ransomware attack

22 Jan 2021
Weekly threat roundup: SAP, Windows 10, Chrome
vulnerability

Weekly threat roundup: SAP, Windows 10, Chrome

21 Jan 2021
Biden nominees highlight tough cyber security challenges
cyber security

Biden nominees highlight tough cyber security challenges

20 Jan 2021

Most Popular

School laptops sent by government arrive loaded with malware
malware

School laptops sent by government arrive loaded with malware

21 Jan 2021
How to move Windows 10 from your old hard drive to SSD
operating systems

How to move Windows 10 from your old hard drive to SSD

21 Jan 2021
What is the Raspberry Pi Pico?
Hardware

What is the Raspberry Pi Pico?

21 Jan 2021