Microsoft’s new AI tells stories based on your photos

But the AI struggles to describe pictures as anything but “awesome”

Microsoft's latest AI system can automatically caption photos based on the people and objects in them, and the context they were taken in.

In development at Microsoft Research, the aim of the project is to explain what appears in a picture, as well as what seems to be happening and how it might potentially make a person feel, the researchers told Live Science.

Computers are already at work identifying what is in photos, such as Facebook's auto-tagging and Google's image search, which is capable of matching similar images based on colour, objects and even postures.

Facebook's own AI development group, FAIR, last year revealed its own AI can recognise images in photos, while in April this year it started captioning photos for blind people via its own Automatic Alternative Text engine.

Advertisement - Article continues below
Advertisement - Article continues below

In order to build its own system, Microsoft's researchers used deep learning methods, for instance, getting the computer to learn how to identify cats in photos by analysing thousands of examples of cats.

Scientists fed more than 8,100 new images into its storytelling system to analyse what stories it generated.

An image captioning program might take five images and suggest: "This is a picture of a family; this is a picture of a cake; this is a picture of a dog; this is a picture of a beach."

But Microsoft's storytelling program could take those same images and suggest: "The family got together for a cookout; they had a lot of delicious food; the dog was happy to be there; they had a great time on the beach; they even had a swim in the water."

The system is still in it early stage for the time being.

Researchers had to gauge how effective the system was at generating stories. Because of the large number of stories the computer generated, they deicide to use an automated method. They found that this automated method rated the computer storyteller as performing almost as well as human storytellers.

Advertisement - Article continues below

However, the system still has a tough time differentiating words, and in the initial tests, has been prone to describe everything as being "awesome".

Margaret Mitchell, study senior author and a computer scientist at Microsoft Research, said the future for this technology is to "help people share their experiences while reducing nitty-gritty work that some people find quite tedious".

This is the latest AI experiment from Microsoft's research arm, which also let loose the disastrous Tay chatbot on Twitter earlier this year.

Amazon recently built out its own AI division with the purchase of Orbeus, an image recognition firm.

Featured Resources

What you need to know about migrating to SAP S/4HANA

Factors to assess how and when to begin migration

Download now

Your enterprise cloud solutions guide

Infrastructure designed to meet your company's IT needs for next-generation cloud applications

Download now

Testing for compliance just became easier

How you can use technology to ensure compliance in your organisation

Download now

Best practices for implementing security awareness training

How to develop a security awareness programme that will actually change behaviour

Download now



Best smartphone 2019: Apple, Samsung and OnePlus duke it out

24 Dec 2019

Most Popular

data governance

Brexit security talks under threat after UK accused of illegally copying Schengen data

10 Jan 2020
Microsoft Windows

What to do if you're still running Windows 7

14 Jan 2020
Microsoft Windows

Memes and Viking funerals: The internet reacts to the death of Windows 7

14 Jan 2020

Openreach offers free full-fibre installation for thousands of homes

14 Jan 2020