Amazon’s HR proves artificial intelligence is truly dumb

We don't know how neural networks see our world, and that's hilarious and scary

Amazon built an AI system to sift through CVs, and it turned out not to like women a fact that doesn't so much highlight algorithmic bias as it does reveal the stupidity of much of the AI that's already in use.

In 2014, Amazon started investigating ways to automate its hiring. "They literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those," a source told Reuters, which uncovered the project. Within a year, it was clear the system was discounting women as candidates for technical jobs, such as software development roles. Unable to fix it, the company scrapped the entire programme.

This isn't a surprise. First, neural networks are only as good as the dataset used to train them. Suppose you wanted to train a network to spot signs of cancer in medical images. You would show it a huge database of images, telling it which ones featured cancer and which ones didn't, and it would unpick the difference between the two. It wouldn't know what cancer is or who the patients are it's merely looking for anomalies. Because that's relatively simple, it works well when trained with high-quality, physician-labelled datasets. If the training data is incomplete, limited or incorrect, then the neural network will reflect those flaws.

In other words, it's garbage in, garbage out. That's what reportedly tripped up Amazon. The HR department fed the neural network ten years of CVs of previous hires, and so naturally the system looked for staff who were like the ones already working there; all it did was replicate existing hiring practices, but more quickly.

And Amazon, like other tech companies, has a diversity problem its workforce is actually 40% female, but that falls to fewer than a quarter in managerial positions. Because the system saw that fewer women were being hired, it avoided them.

That's an oversimplification: the system has no idea what gender is, it doesn't know what a woman is, and it doesn't understand systemic workplace discrimination. It isn't sexist, it's too stupid for that. The neural network, trained on flawed data, simply noted that the word "women" such as in the phrase "women's college" or "head of the women's chess club" was found on unsuccessful CVs and not on the resumes of the people that Amazon hired, so it tossed them onto the reject pile.

Such a flaw can't be easily corrected, although Amazon reportedly tried. That's because neural networks don't actually think like we do. We assume that the AI spotted words such as "women", but men and women could have used different language or phrasing, or some other signal unseen to us. Researchers have managed to fool neural networks by adding in background noise that we humans can't see. In one case, a toy turtle looked to the machine like a rifle; in another, a stop sign was seen as a speed sign. We don't know whether it's the colour, or the shape, or something else that the network is using to identify an object. We don't see the same.

Another lesson in what we don't know about how neural networks think comes via Janelle Shane, who offers hilarious examples of the strange way neural networks see the world via her "AI Weirdness" blog (aiweirdness.com). She trained her neural network on Sherwin-Williams paint colours, then asked it to come up with its own names. The results included "Sudden Pine", "Stanky Bean" and "Turdly". Her latest efforts include snake types ("Cancan Rattlesnake"), My Little Pony characters ("Creep Well") and college courses of the future ("Survivery").

And these are the machines we're going to use to organise and optimise our world? Don't believe the hype. Whenever someone sings the praises of neural networks, ask about the training data and how accuracy will be assessed especially if it's being used for sensitive tasks such as hiring, policing and governance. And remember that if Amazon can't get it right, there's no reason to think that a startup's neural network isn't going to spit out Turdly results, too.

Featured Resources

Unlocking collaboration: Making software work better together

How to improve collaboration and agility with the right tech

Download now

Four steps to field service excellence

How to thrive in the experience economy

Download now

Six things a developer should know about Postgres

Why enterprises are choosing PostgreSQL

Download now

The path to CX excellence for B2B services

The four stages to thrive in the experience economy

Download now

Recommended

10 keys to AI success in 2021
Whitepaper

10 keys to AI success in 2021

10 Mar 2021
MLOps 101: The foundation for your AI strategy
Whitepaper

MLOps 101: The foundation for your AI strategy

10 Mar 2021
How to choose an AI vendor
Whitepaper

How to choose an AI vendor

10 Mar 2021
Realising the benefits of automated machine learning
Whitepaper

Realising the benefits of automated machine learning

10 Mar 2021

Most Popular

Microsoft is submerging servers in boiling liquid to prevent Teams outages
data centres

Microsoft is submerging servers in boiling liquid to prevent Teams outages

7 Apr 2021
University of Hertfordshire's entire IT system offline after cyber attack
cyber attacks

University of Hertfordshire's entire IT system offline after cyber attack

15 Apr 2021
NSA uncovers new "critical" flaws in Microsoft Exchange Server
servers

NSA uncovers new "critical" flaws in Microsoft Exchange Server

14 Apr 2021