AI planners in Minecraft could help machines design better cities

A dozen or so steep-roofed buildings cling to the edges of an open-pit mine. High above them, on top of an enormous rock arch, sits an inaccessible house. Elsewhere, a railway on stilts circles a group of multicolored tower blocks. Ornate pagodas decorate a large paved plaza. And a lone windmill turns on an island, surrounded by square pigs. This is Minecraft city-building, AI style.

Minecraft has long been a canvas for wild invention. Fans have used the hit block-building game to create replicas of everything from downtown Chicago and King's Landing to working CPUs. In the decade since its first release, anything that can be built has been.

Since 2018, Minecraft has also been the setting for a creative challenge that stretches the abilities of machines. The annual Generative Design in Minecraft (GDMC) competition asks participants to build an artificial intelligence that can generate realistic towns or villages in previously unseen locations. The contest is just for fun, for now, but the techniques explored by the various AI competitors are precursors of ones that real-world city planners could use.

The Generative Design in Minecraft competition challenges AIs to design settlements for previously unseen locations

Successful entries typically use a range of techniques to identify when to level terrain or where to place bridges and buildings. These include old-school path-finding algorithms that connect remote parts of a settlement, cellular automata that can produce complex structures using simple rules, and machine learning.

The competition has come a long way in three years. The first time around, settlements often looked machine-made, with buildings arranged in repetitive rows or random clusters. This year's winners, announced on Thursday, featured settlements with believable layouts adapted to each location. Roads hug hillsides, bridges span rivers, and houses even contain furniture.

Open-ended and subjective, GDMC was set up to push the limits of AI. Unlike other AI competitions, such as the DARPA challenges for self-driving cars or robots, it has no clear finish line. What makes a good village? There isn't a numerical value that you can optimize for," says co-organizer Christoph Salge, a computer scientist at the University of Hertfordshire, UK.

The open-endedness of the challenge means that AIs need to master multiple objectives. To win, they must impress eight human judges from a range of backgrounds, including architects, archaeologists, and game designers.

These judges score the AI city planners in four areas: how well they adapt their designs to specific locations; how well the layouts work, according to criteria such as whether there are bridges and roads between different areas; how appealing they are aesthetically; and how much the designs evoke a narrative-are there details that tell a story about how a town came to be, such as a ruin or a pit from which building materials might have been mined? Making a Minecraft village for an unseen map is something a 10-year-old human could do," says Salge. But it is really difficult for an AI."

Levelling the land

For example, one entrant started by identifying the type of environment-desert or forest, say-and then generated buildings that looked as if they had been built out of common local materials. Another was good at leveling the landscape and laying down plazas. This tactic worked well on flat, open terrain, where it produced striking Japanese-style temple complexes. But it was less successful on a small island, which it paved over completely.

Even the winning entries still make silly mistakes. In one settlement, some of the houses are buried up to their eaves in sand. This is clearly because the algorithm wants to build on solid ground, says Salge. It sinks buildings until they hit rock.

Claus Aranha, who studies evolutionary computation at the University of Tsukuba in Japan, advised three entrants to the competition. He thinks it is a good way to explore and test out new AI techniques. One thing that I really like is that there are many different approaches to this challenge," he says.

Realistic-looking game worlds are one thing. But AI is already being used to analyze how cities are built. Techniques and approaches similar to those being deployed in the competition could one day help design real cities that are healthier and safer.

For example, Aranha has found that most entries take a top-down approach, meaning the AI city generator looks at a given area and generates a settlement to fit. That can give good overall results, but the details may be off. Aranha thinks that a multi-agent approach, where several AIs work independently to build structures informed by their immediate surroundings, could lead to more coherent and realistic designs.

He is now going to use this insight to help his own work, in which he uses simulations to explore the impact of different urban planning policies on disaster scenarios such as earthquakes or wildfires. He generates virtual cities by teaching a neural network what cities look like with data from OpenStreetMap. By automatically generating thousands of virtual cities that differ in properties such as street layouts or number and position of open spaces, he can assess whether a policy that required 10% of residential area to be reserved for parks would save lives.

The CityScope Champs-Elysee project from the MIT Media Lab uses agent-based simulation to explore proposed designs

Meanwhile, Arnaud Grignard and his colleagues at the MIT Media Lab are using agent-based simulation to explore possible designs for busy public spaces, including a regenerated Champs-Elysees in Paris. And New York startup Topos is using AI to help understand how the layout of a city affects those living in it. In one project it used a range of AI approaches, including image recognition and natural-language processing, to learn how different areas in New York were used by the people living there. It then redrew the boundaries of New York's five boroughs on the basis of similarities between neighborhoods, such as whether they are residential or commercial, leafy or urban. The resulting map arrays the boroughs as more or less concentric rings around a central Manhattan.

Jasper Wijnands, at the University of Melbourne in Australia, is also convinced that AI has a place in future urban design. He and his colleagues have started exploring the use of generative adversarial networks (GANs) to do style transfer on images from Google Street View.

Style transfer is typically used to reproduce one image in the style of another, such as making a selfie look like as if it were painted by Van Gogh. But instead of a visual style, Wijnands got his AI to learn a style" that reflected the public health data in different city blocks. He then asked it to reproduce Street View images in the style of neighborhoods where public health was good. In other words, his AI can touch up images of bad neighborhoods so that they look like good ones. City planners could then use these tweaks-a green space here, a wider street there-as a guide for urban improvements.

The AI was not taught what sorts of things planners think make cities better, but it hit upon common ideas by itself. It's interesting to see that the GAN output is consistent with our scientific understanding of the impact of green space on health," says Wijnands.

His team now has a $1.2 million grant to develop the approach, and he is introducing it to his urban-planning students.

Design impacts

One of the more immediate uses for AI in city planning is to understand the impact of urban design at a global scale. In January Wijnands and his colleagues published a study in The Lancet Planetary Health in which they looked at 1,692 cities, home to a third of the world's population. They used convolutional neural networks, typically used for image recognition, to classify different urban layouts according to the number of serious road accidents that had happened in them. Cities with more high-transit rail networks and denser street layouts arranged around small blocks were shown to be safer than more sprawling layouts arranged around cul-de-sacs.

Those results may not be too surprising, but the data could not have been analyzed at all without automation.

Visions of utopian living are always based on presuppositions about what kinds of urban spaces make people happier or healthier. But these are hard to test, and ambitious regeneration projects can fail. AI city planners could help in a number of ways, revealing the hidden impacts of certain existing layouts or simulating thousands of potential designs. Salge is now working with planners in the US on how future competitions might incorporate more realistic data about how people use cities, such as how they move about or where they go shopping. That could make the artificial creations even more lifelike-and potentially more useful.

But don't expect AI to take over planning completely. Cities are a lot more than an arrangement of objects on the ground: they are lived in. And that means they are the result of many trade-offs, says Dave Amos, an urban planner who has a popular YouTube channel called City Beautiful. As Amos puts it in a video reviewing the winning entry to the GDMC competition in 2018: Planning is inherently a political process. You need people to butt heads about what the development is going to be like."