'Make It Real' AI Prototype Turns Drawings Into Working Software

An anonymous reader quotes a report from Ars Technica: On Wednesday, a collaborative whiteboard app maker called "tldraw" made waves online by releasing a prototype of a feature called "Make it Real" that lets users draw an image of software and bring it to life using AI. The feature uses OpenAI's GPT-4V API to visually interpret a vector drawing into functioning Tailwind CSS and JavaScript web code that can replicate user interfaces or even create simple implementations of games like Breakout. "I think I need to go lie down," posted designer Kevin Cannon at the start of a viral X thread that featured the creation of functioning sliders that rotate objects on screen, an interface for changing object colors, and a working game of tic-tac-toe. Soon, others followed with demonstrations of drawing a clone of Breakout, creating a working dial clock that ticks, drawing the snake game, making a Pong game, interpreting a visual state chart, and much more. Tldraw, developed by Steve Ruiz in London, is an open source collaborative whiteboard tool. It offers a basic infinite canvas for drawing, text, and media without requiring a login. Launched in 2021, the project received $2.7 million in seed funding and is supported by GitHub sponsors. When The GPT-4V API launched recently, Ruiz integrated a design prototype called "draw-a-ui" created by Sawyer Hood to bring the AI-powered functionality into tldraw. GPT-4V is a version of OpenAI's large language model that can interpret visual images and use them as prompts. As AI expert Simon Willison explains on X, Make it Real works by "generating a base64 encoded PNG of the drawn components, then passing that to GPT-4 Vision" with a system prompt and instructions to turn the image into a file using Tailwind. You can experiment with a live demo of Make It Real online. However, running it requires providing an API key from OpenAI, which is a security risk.

Read more of this story at Slashdot.