Microsoft's AI-powered Sketch2Code can build apps from whiteboard sketches

Sketch2Code tool is available on GitHub and offers an automated service for web and app design

designs on whiteboard

Microsoft has developed an AI-powered web design tool that can turn sketches of websites into functional HTML code, called Sketch2Code.

How and why it has been developed was explained in a YouTube video by Tara Shankar Jana, senior product manager, Microsoft AI, who said that Microsoft's AI tool aims to "empower every developer and every organisation to do more with AI".

Drag-and-drop website building apps are nothing new, as lots of companies offer a service to move custom designs into a digital workspace, but this is the first to use artificial intelligence to complete the design.

Sktech2Code was born out of the "intrinsic" problem, as Shankar Jana puts it, of sending pictures of wireframe web or app designs from whiteboards or paper to a designer to create an HTML prototype.

Advertisement
Advertisement - Article continues below
Advertisement - Article continues below

To break this process Microsoft developed a web-based application which cuts out the extra human element (in this case the designer) and instead, images taken of sketches are sent to AI servers based on Redmond's Azure cloud infrastructure. 

These AI servers then run those images against a pre-built AI model and then creates an HTML code base followed by a resulting app. At the centre of this system is something called a "custom vision object prediction model" which is basically a tool of image recognition, which has been specifically trained with datasets of hand-drawn images.

The model identifies the basic HTML elements such as buttons, labels and text boxes, which allows it to predict when any of those elements are present in any given image. It also can recognise handwritten text within the boxes to create a fully formed app or a webpage.

The tool is available for developers on GitHub and its code is independent of HTML, according to Shankar Jana, and can be extracted with XAML and Universal Windows Platform. 

Featured Resources

Digitally perfecting the supply chain

How new technologies are being leveraged to transform the manufacturing supply chain

Download now

Three keys to maximise application migration and modernisation success

Harness the benefits that modernised applications can offer

Download now

Your enterprise cloud solutions guide

Infrastructure designed to meet your company's IT needs for next-generation cloud applications

Download now

The 3 approaches of Breach and Attack Simulation technologies

A guide to the nuances of BAS, helping you stay one step ahead of cyber criminals

Download now
Advertisement

Recommended

Visit/technology/30736/what-is-ethical-ai
Technology

What is ethical AI?

21 Jan 2020
Visit/careers/28212/a-guide-to-cyber-security-certification-and-training
Careers & training

A guide to cyber security certification and training

13 Jan 2020
Visit/email-clients/19598/hotmail-outlookcom-upgrades-your-questions-answered
Software

Hotmail.co.uk migration to Outlook.com: Qs answered

11 Nov 2019
Visit/careers/28219/it-manager-job-description-what-does-an-it-manager-do
Careers & training

IT manager job description: What does an IT manager do?

28 Oct 2019

Most Popular

Visit/business-strategy/mergers-and-acquisitions/354602/xerox-to-nominate-directors-to-hps-board-reports
mergers and acquisitions

Xerox to nominate directors to HP's board – reports

22 Jan 2020
Visit/operating-systems/25802/17-windows-10-problems-and-how-to-fix-them
operating systems

17 Windows 10 problems - and how to fix them

13 Jan 2020
Visit/business-strategy/public-sector/354608/uk-gov-launches-ps300000-sen-edtech-initiative
public sector

UK gov launches £300,000 SEN EdTech initiative

22 Jan 2020
Visit/web-browser/30394/what-is-http-error-503-and-how-do-you-fix-it
web browser

What is HTTP error 503 and how do you fix it?

7 Jan 2020