Microsoft's AI-powered Sketch2Code can build apps from whiteboard sketches

Sketch2Code tool is available on GitHub and offers an automated service for web and app design

designs on whiteboard

Microsoft has developed an AI-powered web design tool that can turn sketches of websites into functional HTML code, called Sketch2Code.

How and why it has been developed was explained in a YouTube video by Tara Shankar Jana, senior product manager, Microsoft AI, who said that Microsoft's AI tool aims to "empower every developer and every organisation to do more with AI".

Drag-and-drop website building apps are nothing new, as lots of companies offer a service to move custom designs into a digital workspace, but this is the first to use artificial intelligence to complete the design.

Sktech2Code was born out of the "intrinsic" problem, as Shankar Jana puts it, of sending pictures of wireframe web or app designs from whiteboards or paper to a designer to create an HTML prototype.

To break this process Microsoft developed a web-based application which cuts out the extra human element (in this case the designer) and instead, images taken of sketches are sent to AI servers based on Redmond's Azure cloud infrastructure. 

These AI servers then run those images against a pre-built AI model and then creates an HTML code base followed by a resulting app. At the centre of this system is something called a "custom vision object prediction model" which is basically a tool of image recognition, which has been specifically trained with datasets of hand-drawn images.

The model identifies the basic HTML elements such as buttons, labels and text boxes, which allows it to predict when any of those elements are present in any given image. It also can recognise handwritten text within the boxes to create a fully formed app or a webpage.

The tool is available for developers on GitHub and its code is independent of HTML, according to Shankar Jana, and can be extracted with XAML and Universal Windows Platform. 

Featured Resources

2021 Thales access management index: Global edition

The challenges of trusted access in a cloud-first world

Free download

Transforming higher education for the digital era

The future is yours

Free download

Building a cloud-native, hybrid-multi cloud infrastructure

Get ready for hybrid-multi cloud databases, AI, and machine learning workloads

Free download

The next biggest shopping destination is the cloud

Know why retail businesses must move to the cloud

Free Download

Recommended

What is ethical AI?
artificial intelligence (AI)

What is ethical AI?

8 Oct 2021
Why it's the perfect time to rethink recruitment
Business strategy

Why it's the perfect time to rethink recruitment

10 Sep 2021
What are the pros and cons of AI?
machine learning

What are the pros and cons of AI?

8 Sep 2021
The benefits of hot desking
Business strategy

The benefits of hot desking

1 Sep 2021

Most Popular

Best Linux distros 2021
operating systems

Best Linux distros 2021

11 Oct 2021
Apple MacBook Pro 15in vs Dell XPS 15: Clash of the titans
Laptops

Apple MacBook Pro 15in vs Dell XPS 15: Clash of the titans

11 Oct 2021
Veritas Backup Exec 21.3 review: Covers every angle
backup software

Veritas Backup Exec 21.3 review: Covers every angle

14 Oct 2021