The Accidental Engineer Who Conjured Up Extended Reality

In the 1980s, people weren't wearing head-mounted cameras, displays, or computers. Except for high school student Steve Mann, who regularly wore his homemade electronic computer vision system (seeing aid).

Back then, Mann attracted stares, questions, suspicion, and sometimes hostility. But it didn't stop him from refining the technology he developed. It now underlies augmented-reality eyeglasses-including those by Google and Magic Leap-that are used in operating rooms and industrial settings such as factories and warehouses.

Steve Mann

Employer:

University of Toronto

Job title:

Professor of electrical and computer engineering, computer science, and forestry

Member grade:

Fellow

Alma maters:

McMaster University in Hamilton, Ontario; MIT

Although head-mounted computers haven't reached smartphone-level ubiquity, when Mann wears XR (eXtended Reality, something he and Charles Wyckoff invented at MIT in 1991) gear these days as a professor of electrical and computer engineering, computer science, and forestry at the University of Toronto, he doesn't turn as many heads as he used to.

In part because of his inventiveness and creativity, the IEEE Fellow was honored for his contributions to wearable computing and the concept of sousveillance-the practice of using personal recording devices to watch the watchers and invert traditional surveillance power structures-with this year's IEEE Masaru Ibuka Consumer Technology Award. Sponsored by Sony, the award was bestowed by the IEEE Consumer Technology Society at the Consumer Electronics Show held in January in Las Vegas.

Empowering people through wearable tech

Mann is regarded as the father of wearable computing." Asked what he thinks about the moniker, he says it's less about the title and more about empowering people to see the world-and themselves-in new ways.

His research and systematic reimagining of how electronic devices can support and extend human abilities, especially vision, have yielded benefits for society. Among them are assisting the visually impaired with the ability to identify objects and enabling experts to remotely view what frontline workers see and then guide them from afar.

His IEEE award came one month after he received the Lifeboat Foundation's Guardian Award, given to a scientist or public figure who has warned of a future fraught with dangers and encouraged measures to prevent them." The foundation is a nonprofit, nongovernmental organization dedicated to encouraging scientific advancements while helping humanity survive existential risks and possible misuse of increasingly powerful technologies including genetic engineering, nanotechnology, and robotics/AI.

A natural-born tinkerer

It stands to reason that Mann would become a leading tinkerer. His earliest memories are of welding with his grandfather and knitting with his grandmother-unusual hobbies for a typical 4-year-old, though not in Mann's family. His father, who worked for a men's clothing company, supplemented his income by buying and renovating houses, long before the concept of flipping houses became widespread.

We were always living in a house under construction," Mann recalls. I used to help my dad fix things when I was 4 or 5-hammer in my hand-normal stuff." His grandfather, a refrigeration engineer, taught him how to weld. By age 6, he was wiring and building homemade radios. By the time he was 8, he had started a neighborhood repair business, fixing televisions and radios.

In a sense, preschool for me was learning engineering and science," Mann says with a laugh. I grew up putting together wood, metal, or fabric. I knew how to make things at a very young age."

Learning to see what others miss

When Mann was 12 years old, his father brought home a broken oscillograph (an early version of the oscilloscope, used to display variations in voltage or current as visual waveforms). It turned out to be a defining moment in his life. Too impatient to accept that the waveform dot on the machine's display moved only up and down instead of both vertically and horizontally, Mann invented a way to push its image through physical space.

He placed the oscillograph-which he now keeps on a shelf in his laboratory-on a board mounted on roller skate wheels. He connected the device to a police radar and rolled it back and forth. When he realized the machine's motion, combined with the dot's vertical movement, created visible waveforms of the radar's signals, as a function of space rather than time, he unknowingly made a revolutionary discovery.

Later he would describe that merging of physical and virtual worlds as extended reality"-a concept that underlies today's AR and XR technologies. It wouldn't be the last time Mann's curiosity turned a problem into an opportunity.

Decades later, on the main floor of his Toronto home, he co-founded InteraXon, the Toronto-based company behind the Muse brain-sensing headband, used to help people manage sleep, stress, and mental health.

Mann shares legendary 1970s Xerox PARC researcher Alan Kay's belief that The best way to predict the future is to invent it." Mann, however, adds: Sometimes you invent it by simply refusing to accept the limitations of the present."

A member of MIT's Media Lab

In high school, Mann won several math competitions designed to challenge students at university level. In 1982 he enrolled in McMaster University, in Hamilton, Ontario, to pursue a degree in engineering physics (an interdisciplinary program that combines physics, mathematics, and engineering principles). As an undergraduate, Mann was already experimenting with early prototypes of wearable computers-head-mounted displays, body-worn cameras, and portable computing systems that predated mainstream mobile tech by decades.

Mann [far right] sits alongside fellow MIT Media Lab graduate students, modeling the wearable computers or smart clothes they were developing as part of their Ph.D. research. Pam Berry/The Boston Globe/Getty Images

He earned a bachelor's degree in 1986. He continued his studies at McMaster to earn a second bachelor's degree in electrical engineering in 1989, then a master's degree in engineering in 1991.

He then enrolled in a doctoral program at MIT, where he joined its renowned Media Lab, a hotbed for unconventional research blending technology, design, and the human experience. He formalized and expanded his ideas around wearable computing, wearable computer vision systems, and wearable AI. He also published some of the earliest academic papers that described the concept of sousveillance.

He completed his Ph.D. in media arts and sciences in 1997.

Mann's doctoral research contributed foundational concepts and hardware that influenced future smart glasses and devices for life logging, the practice of creating a digital record of one's daily life. He also helped blaze a trail for the fields of augmented reality and ubiquitous computing.

Knitting passions into a unique academic career

After completing his Ph.D., Mann returned to Canada and took a position at the University of Toronto as a professor of electrical and computer engineering in 1998. He says he is equally as fascinated by how technology interacts with the natural world as he is by how to remove barriers between the physical world and virtual world.

His interests connect to what he calls vironmentalism," which regards technology as a boundary between our environment and our vironment" (ourselves). This gives rise to his vision of mersive" technologies that link humans not just to each other but also to the environment around them.

Go beyond [what's covered at] school. Define yourself by what you love so much you'd do it [even if no teachers or managers were demanding it]. AI can replace a walking encyclopedia. It can't replace passion."

It's advancing technology for humanity and Earth," he says, riffing on IEEE's mission statement. His guiding principle also explains his cross-appointment in the University of Toronto's forestry department (now part of the Faculty of Architecture, Landscape, and Design)-an unusual entry on an electrical and computer engineering professor's CV.

IEEE and building community

Prior to his groundbreaking doctoral work at MIT, Mann had already joined IEEE in 1988. He credits the organization with connecting him to pioneers like Simon Haykin, the radar visionary he met at McMaster while he was in high school. Haykin pushed him to dream big, he says.

Mann has been active in the IEEE Computer and IEEE Consumer Technology societies. He has served as an organizer, session chair, and program committee member for IEEE conferences related to wearable computing and pervasive sensing.

In 1997 he helped found the International Symposium on Wearable Computers, and numerous other wearable computing symposia, conferences, and events.

He has given keynote talks and presented papers on topics including sousveillance, ubiquitous computing, and other humanistic aspects of technology at the IEEE International Symposium on Technology and Society and the IEEE International Conference on Pervasive Computing and Communications.

His contributions include influential papers in IEEE journals, especially various IEEE Transactions and Computer Society magazines.

Probably his most well-known paper is Wearable Computing." Published in Computer magazine in October 1997, the seminal work outlined the structure and vision for wearable computing as a formal research field. He also contributed articles on sousveillance-exploring the intersection of technology, ethics, and human rights-in IEEE Technology and Society Magazine.

He has collaborated with other IEEE members to develop frameworks for wearable computing standards, particularly around human-computer interfaces and privacy considerations.

Forever the inventor

Mann continues to teach, run his lab, and test new frontiers of wearable devices, smart clothing, and immersive environments. He's still driven, he says, by the same forces that powered his backyard experiments as a child: curiosity and passion.

For students who hope to follow in his footsteps, Mann's advice is simple: Go beyond [what's covered at] school. Don't define yourself by the classes you took or the jobs you had. Define yourself by what you love so much you'd do it even if no teachers or managers were demanding it". He adds that, AI can replace a walking encyclopedia. It can't replace passion."

Mann says he has no plans to retire. If anything, he says, his most productive years are yet to come.

I feel like I'm a late bloomer," he says, chuckling at the irony. I was fixing radios when I was 8, but my best work? That's going to happen between 65 and 85."