Microsoft’s new AI tells stories based on your photos

But the AI struggles to describe pictures as anything but “awesome”

Microsoft's latest AI system can automatically caption photos based on the people and objects in them, and the context they were taken in.

In development at Microsoft Research, the aim of the project is to explain what appears in a picture, as well as what seems to be happening and how it might potentially make a person feel, the researchers told Live Science.

Advertisement - Article continues below

Computers are already at work identifying what is in photos, such as Facebook's auto-tagging and Google's image search, which is capable of matching similar images based on colour, objects and even postures.

Facebook's own AI development group, FAIR, last year revealed its own AI can recognise images in photos, while in April this year it started captioning photos for blind people via its own Automatic Alternative Text engine.

In order to build its own system, Microsoft's researchers used deep learning methods, for instance, getting the computer to learn how to identify cats in photos by analysing thousands of examples of cats.

Scientists fed more than 8,100 new images into its storytelling system to analyse what stories it generated.

An image captioning program might take five images and suggest: "This is a picture of a family; this is a picture of a cake; this is a picture of a dog; this is a picture of a beach."

Advertisement - Article continues below
Advertisement - Article continues below

But Microsoft's storytelling program could take those same images and suggest: "The family got together for a cookout; they had a lot of delicious food; the dog was happy to be there; they had a great time on the beach; they even had a swim in the water."

The system is still in it early stage for the time being.

Researchers had to gauge how effective the system was at generating stories. Because of the large number of stories the computer generated, they deicide to use an automated method. They found that this automated method rated the computer storyteller as performing almost as well as human storytellers.

However, the system still has a tough time differentiating words, and in the initial tests, has been prone to describe everything as being "awesome".

Margaret Mitchell, study senior author and a computer scientist at Microsoft Research, said the future for this technology is to "help people share their experiences while reducing nitty-gritty work that some people find quite tedious".

This is the latest AI experiment from Microsoft's research arm, which also let loose the disastrous Tay chatbot on Twitter earlier this year.

Amazon recently built out its own AI division with the purchase of Orbeus, an image recognition firm.

Featured Resources

Top 5 challenges of migrating applications to the cloud

Explore how VMware Cloud on AWS helps to address common cloud migration challenges

Download now

3 reasons why now is the time to rethink your network

Changing requirements call for new solutions

Download now

All-flash buyer’s guide

Tips for evaluating Solid-State Arrays

Download now

Enabling enterprise machine and deep learning with intelligent storage

The power of AI can only be realised through efficient and performant delivery of data

Download now



Best smartphone 2019: Apple, Samsung and OnePlus duke it out

24 Dec 2019

Most Popular


Zoom kills Facebook integration after data transfer backlash

30 Mar 2020
data breaches

Marriott data breach exposes personal data of 5.2 million guests

31 Mar 2020
cyber crime

FBI warns of ‘Zoom-bombing’ hackers amid coronavirus usage spike

31 Mar 2020
data management

Oracle cloud courses are free during coronavirus lockdown

31 Mar 2020