The Visionary Designer Behind Google’s Warehouse-Scale Data Centers

Photo: Google

THE INSTITUTE Growing up in Rio de Janeiro, Luiz Andre Barroso seemed destined to become a physician. After all, his grandfather, father, uncle, and aunt were all health professionals. But when he was 8 years old, he decided he wanted to become an electrical engineer.

I don't advise anybody to make career decisions when they are 8 years old," Barroso says, laughing. But his early determination paid off. Today he is a vice president of engineering at Google in Mountain View, Calif.

The IEEE senior member credits his grandfather, a surgeon in the Brazilian Navy, with inspiring him to become an engineer. His grandfather's hobby was radio, and Barroso would spend hours helping him fix radios and build microphones and antennas at the family's cacao farm.

Barroso, who has worked for Google for nearly 20 years, heads the office of cross-Google engineering, responsible for technical roadmaps that cut across multiple product lines. He has worked on some of the company's most popular products including its Web index search and the Google Maps navigation app.

The Google Fellow is best known as the designer of the company's data centers, which house hundreds of thousands of computer servers and disk drives. The facilities have brought us cloud computing, powerful search engines, and faster Internet services.

For his pioneering work on data centers, Barroso was named this year's recipient of the Eckert-Mauchly Award from the Association for Computing Machinery and the IEEE Computer Society. The award recognizes contributions to computer and digital systems architecture.

In this interview with The Institute, he talks about what led to the need for the huge data centers and what he's working on these days.

JOURNEY FROM BRAZIL

After Barroso received bachelor's and master's degrees in electrical engineering from Pontifical Catholic University of Rio de Janeiro in 1989, he was accepted to the University of Southern California's computing engineering program, in Los Angeles, where he earned a Ph.D. in 1996.

His plan was to return to Brazil to become an engineering professor, but the country was experiencing an economic downturn. Universities weren't hiring. He got an interview in 1995 to work for Digital Equipment Corp.'s Western Research Laboratory (WRL), in Palo Alto, Calif.

Many people may not appreciate it today, but Digital was an amazingly innovative company in the '70s, '80s, and '90s," he says. I couldn't believe that these folks wanted to interview me, and then I couldn't believe they gave me a job. To this day, I don't remember being as thrilled with a professional accomplishment as I was the day they invited me to interview."

The WRL was small and prestigious, Barroso says, and the 20 or so researchers there did applied research, which he liked.

There I found some of the top researchers in computer architecture, which was my field of study at the time," he says. I had the mentorship, and I had the resources to begin investigating how to build hardware to run more modern applications."

He explains that in the 1970s and early '80s, large-scale computers were designed to run high-performance number-crunching applications used for weather forecasting and simulating nuclear reactions. But in the early 1990s, demand grew for high-performance computers capable of running business applications and Web services.

I was really interested in figuring out if the hardware we had been designing for numerical workloads for the last two decades was a good fit for this new field or not. And it turns out that it was not," he says. We were designing in the '90s for the market of the '80s."

Barroso researched various workloads to see how Digital's hardware could support them.

One we were investigating was AltaVista, which was, in some ways, the world's first bona fide search engine," he says. I didn't know at the time that I would eventually join the company that became almost synonymous with search engines."

LONG-TERM GOOGLER

Compaq bought Digital in 1998 and then canceled the microprocessor project that Barroso had been working on for more than two years.

I was bummed and thought, Wow, this is a tough business, this microprocessor design business," he says. You invest years, and at any one point in time the economics of the situation may change. There's a big window of vulnerability for your projects to be canceled, so I began to think about doing something different."

Two of Barroso's former Digital colleagues, Jeffrey Dean and Sanjay Ghemawat, encouraged him to join them at Google.

Barroso says he didn't think he would be a good fit there.

Are you insane?" he says he asked the two. I design chips. You build search engines. Why in the world would you think I'll be useful? But, of course, they knew that I had an interest in high-performance applications. So I decided: What the heck, let's give that a try."

As it turned out, Google was about to become much more of a hardware company than it had been.

BUILDING A SERVER FARM

When Barroso joined Google in 2001, the company-like others-housed its servers at leased space in third-party data centers, which were basically cages in which a few racks of computing equipment were placed. In 2004, as the economy recovered from the dot-com crash, demand began to grow for space at the facilities. At the same time, Google's search business was expanding rapidly, and it recently had launched its email product, Gmail.

Those services, Barroso says, required a building's worth of machines to run. And the hardware and the software together had to deliver the performance needed-which he says could be achieved only by taking a holistic approach" to design and deployment. In other words, he says, The data center itself had to be treated as one massive warehouse-scale computer, built from the ground up.

At the time," he says, we didn't know that we were inventing almost a new kind of computer."

The first such data center, in The Dalles, Ore., was completed in 2006.

Barroso says the first time he set foot in Google's massive, new data center, there were machines, cables, and networking hardware as far as the eye could see.

I saw something that I had only dreamed about during the design phase," he says. It was an amazing moment."

From a sustainability standpoint, Google's facilities were efficient. The centers implemented fault-tolerance software and hardware infrastructure to make the servers more resilient against disruption.

If you offer an Internet service, people expect it to be up all the time," Barroso says.

He coauthored a book on the architecture in 2009, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines.

Luiz joined at just the right moment, when data centers and energy-efficient servers were becoming more and more important to Google," says Urs Holzle, senior vice president of Google's technical infrastructure. In a span of just two to three years, Luiz and his team transformed the design of data centers, reducing the cooling overhead of our data centers by a factor of five versus conventional designs."

Photo: Robbert Frank Hagens/Alamy A Google Data Center in The Netherlands. DATA CENTER INNOVATIONS

Barroso's team explored several areas for improving efficiency. One was to allow the data center floor to run at warmer temperatures-which can actually help cooling systems run more efficiently without sacrificing component reliability, Barroso says.

He is looking for ways to improve computing efficiency by speeding up communication. Today's computers can deal well with events that take milliseconds, such as accessing a disk drive or sending a message over the Internet-or nanoseconds, like loading a piece of data from main memory. But more and more events in a data center are happening at the midrange microsecond scale, such as sending a message from one machine inside the building to another.

For a collection of servers to perform better than one server, the efficiency of the communication between these servers has to be very high," Barroso explains. You can throw 10 servers instead of one at a problem, but you may get only two times the performance improvement if communication performance is poor. Addressing that microsecond scale will address the scalability and, therefore, the efficiency of data center scale workloads."

Microsecond time scale events are a relatively new thing in computing, he says, and the computer industry has yet to react to it.

It hasn't figured out how to make things like that be efficient," he says. By keeping this issue unaddressed we hurt the efficiency and the ease of programmability of big data center scale workloads."

In Attack of the Killer Microseconds," a 2017 article in Communications of the ACM, he and his coauthors described the problem and pointed out ways that the computer industry could solve it.

Other projects Barroso is leading include Google and Apple's free contact-tracing app for COVID-19, the Exposure Notifications System. The app, which runs on iPhones and Android phones, exchanges private keys with other phones via Bluetooth. A person who tests positive for COVID-19 can press a button to send an alert to phones that have been in close proximity with her phone in an anonymous fashion. Public health officials in about 16 countries and 20 U.S. states have released the app to citizens.

It's been really a point of pride that we're able to, in a small way, be part of the arsenal that public health authorities have to fight the pandemic," Barroso says.

SENSE OF COMMUNITY

Barroso joined IEEE when he was a graduate student. All the clever people I knew were members of the IEEE," he says, so I decided that if I wanted to behave like a clever person in college, I should become an IEEE member."

He says he stays with the organization because IEEE gives him a sense of community through its conferences, speaker panels, and publishing opportunities.

It provided me with this community of people to exchange ideas with," he says. Even though I've never been an academic, the forums that IEEE provides for industrial technologists to interact with academics is really valuable. I've taken full advantage of that."

IEEE membership offers a wide range of benefits and opportunities for those who share a common interest in technology. If you are not already a member, consider joining IEEE and becoming part of a worldwide network of more than 400,000 students and professionals.