Q&A: Why the Metaverse Needs to Be Open

A vast digital mirror world accessible through virtual reality is just around the corner, if you believe the latest pronouncements coming out of Silicon Valley. This so-called metaverse has captured the tech zeitgeist and is attracting billions in investments. Last month, Facebook CEO Mark Zuckerberg even announced plans to turn his trillion-dollar social network into "a metaverse company".

The name comes from author Neal Stephenson's dystopian sci-fi classic Snow Crash, which envisaged a gigantic 3D playground where millions of people escaped the drudgery of the real world. To its proponents, the metaverse is the natural evolution of the internet. They envisage a sprawling network of interconnected virtual worlds for such diverse activities as gaming, watching live entertainment, buying real-estate, and collaborating on design projects.

How to achieve that vision is an open question. Online video games like Fortnite and Roblox are pitching themselves as precursors to the metaverse because of their large 3D environments where millions gather every day. They've been adding social features, character personalization, and the ability create new environments, and have even been holding concerts to tempt people to spend ever more of their time immersed in these virtual worlds.

But the key elements of today's internet are its openness, connectivity, and interoperability, something these early metaverse-like experiences are unable to replicate. While the web has the common language of HTML and Javascript and established protocols to ensure seamless browsing, there are no shared standards for building or connecting the virtual worlds that are supposed to populate the metaverse.

That's why early pioneers came together last week at leading computer graphics conference SIGGRAPH 2021 to chart a path forward. In a session titled "Building the Open Metaverse," representatives from Fortnite developer Epic Games, Roblox, chipmaker NVIDIA, and 3D-development platform Unity outlined their work and the challenges ahead.

They discussed agreeing on the 3D equivalent of the JPEG; finding ways to share not only 3D objects, but also their behavior between virtual worlds; dealing with the vast data requirements of 3D environments; and making it possible to create 3D content without your own design studio. The consensus was that there are some promising developments, but many problems are currently unsolved.

To find out more, we spoke with one of the session organizers, Marc Petit of Epic Games, who oversees the company's Unreal Engine, a 3D creation platform. The conversation has been edited for length and clarity.

IEEE Spectrum: What is the metaverse?

Marc Petit: It's a hard question. For me, it's the evolution of the internet as the fabric for our personal and professional lives. With the internet right now, you keep on scrolling through video. We believe that the metaverse will bring in the era of interactive content, where you have agency over the content and you can control it, you can interact with it, and you can ultimately immerse yourself into the content and not just look at it through a small window. The foundation is real-time 3D media. Whether those elements get integrated into your existing environment through augmented reality, or you immerse yourself with a [virtual reality] headset into the content, all of that is a consequence of the content being 3D and interactive.

Spectrum: Why is openness going to be an important feature for any future metaverse? And what does openness mean, in this context?

Petit: I think the openness is mandated for two reasons. One, for technical reasons, because the internet was based on the ability for things to communicate among themselves. If we have a lot of virtual worlds where you cannot go from one to the next or they are incompatible with each other that's not going to be a good experience. So I think that the very nature of a connected system requires that there is openness.

And then there is the societal aspect of it. I think it's an opportunity to create business models and rules that are more fair for people, [protect] their privacy, and make sure creators get their fair share for the content they create. Because, hopefully, we're going towards an economy of creation, where people who make the money are the people who created the content, not the people who own the platforms. We want everybody to become a consumer and a creator and so we need the platform and the economy that allows participation for everybody.

The metaverse is not here. There are huge technical challenge that we need to solve as an industry.

Spectrum: Lots of companies have developed virtual worlds of their own. How far have they got with linking them together?

Petit: The metaverse is not here. There are huge technical challenge that we need to solve as an industry. Let me be clear about this, we are at the first baby steps. It's about eating an elephant one mouthful at a time. Right now we have to solve the problems of exchanging objects. There's glTF and USD [two leading 3D graphics file formats], but we still can't agree on a material representation today. We're getting there and progress has been made, but it's still not a slam dunk. And some proprietary technology is still important and hasn't been really moving to the open space.

Then it's going to be all about behavior, and we have two kinds of behavior. Procedural behaviors, what we call rigging: so how a character moves, how a vehicle drives itself. And then we have the simulation aspects. We all live in the same world with the same laws of physics. So you would hope that we could harmonize physics to each simulation. But we all have different ways to represent logic and behaviors. That's one of the big things we need to tackle.

A screenshot from 'Fortnite'Epic Games

Spectrum: Are there any obvious route for making things like behavior and attributes interoperable?

Petit: No. When you start thinking about persistent worlds, the technological paradigm is going to be very different. You have to maintain a living and breathing world and do in-flight edits. It's not like a game where you can stop the game, change the game and restart the game. We're talking about persistent world where it's always on. There's a lot of room for innovation in terms of how you handle interactivity and scripting in an environment where you're always live and you have millions of users. Some of the models that we have today are not adequate and need to be evolved.

Spectrum: How do you think the industry will come together to achieve this goal?

Petit: All the companies cooperate and acknowledge the value and the need for open source systems and foundations. I think standards will play a big role. But the standards don't drive innovation, they tend to drive commoditization. So we are managing this duality. We support the standards, but with Unreal Engine 5 we are pushing our engine, our data representation, to become the benchmark and the most advanced solution to create and playback high-fidelity, fully simulated worlds.

It takes a few geniuses, like we have on our staff, to invent those kinds of technologies and prove that they work. Then, when it's proven, it becomes shared and open. I know some people say open source can do innovation, and it can happen. But I think it's following the innovation that's done by some of those private groups.

Spectrum: Assuming we solve these problems and create this open system of interlocking virtual worlds, what will that make possible?

Petit: Seamless integration. Being able to aggregate and consume information from everywhere, and letting data from multiple platforms coexist in a single representation. We already have that in the web with microservice architecture and connectivity, so we can see how this could evolve. Once we have got this shared representation challenge figured out, I think we can define services so that things can interoperate. Being able to use your digital assets and your digital properties is probably the best example. If I buy a Ferrari to play Fortnite I'd love to use it on Roblox.

We're making sure that anybody can create content that other people want to see. Because nobody's watching bad video, and this will be the same for 3D. Nobody wants crappy content.

Spectrum: A common theme during the SIGGRAPH session was that for the metaverse to be truly open, it needs to be easier for users to make their own 3D content. How how far off is that?

Petit: We are we investing proactively in making sure that happens. For example, our Twinmotion product is the easiest way that you can create 3D content right now. We are proving that you can use game mechanics to make things simple. In Twinmotion, if you want snow on your scene you push a button and the snow falls and accumulates.

For content we have Megascans [a library of 3D scans of real world objects], where our mandate is to scan enough of the world so that we can recreate the entirety of the world using proceduralism and machine learning techniques. And then there are all of the techniques around AI-assisted creation and AI-assisted artistry. So all of those efforts and investments by Epic are making sure that anybody can create content that other people want to see because it meets the bar in terms of quality. Because nobody's watching bad video, and this will be the same for 3D. Nobody wants crappy content.

Spectrum: A 3D version of the internet involves a lot more data than today's 2D one. How big of a challenge is it going to be to actually serve this to millions of people?

Petit: Well, it's interesting, because there's a lot of data right now that is represented in video that could be represented more efficiently in 3D. But you're right, the data will be big. I think you'll see software architectures adjust with that reality, with server side systems doing the heavy lifting and [3D] glasses doing something much lighter weight. We're starting to see the evolution of that computing infrastructure, with GPUs moving to the edge and the clients being more capable. Do we have enough? No, never. There's never enough compute!

Spectrum: You've made it clear that there are still fundamental challenges to overcome. Which solutions do you think are close at hand, and which problems are going to be harder to solve?

Petit: I think we are cracking the code of creating content that people want to watch. That's around the corner. And building this critical mass of technology where anybody can actually create their own virtual worlds with as much efficiency as possible. Everybody's at the stage of integrating real time 3D as a medium into their workflow. The more difficult thing is going to be the connection of those workflows and the emergence of platforms that embody the values that we discussed earlier. That's going to be what takes the most time, because platforms are hard to build.