Unlocking Voices: AR For Nonverbal Autism
Arthur T Knackerbracket has processed the following story:
The HoloBoard augmented-reality system lets people type independently.
Jeremy is a 31-year-old autistic man who loves music and biking. He's highly sensitive to lights, sounds, and textures, has difficulty initiating movement, and can say only a few words. Throughout his schooling, it was assumed he was incapable of learning to read and write. But for the past 30 minutes, he's been wearing an augmented-reality (AR) headset and spelling single words on the HoloBoard, a virtual keyboard that hovers in the air in front of him. And now, at the end of a study session, a researcher asks Jeremy (not his real name) what he thought of the experience.
Deliberately, poking one virtual letter at a time, he types, "That was good."
It was not obvious that Jeremy would be able to wear an AR headset, let alone use it to communicate. The headset we use, Microsoft's HoloLens 2, weighs 566 grams (more than a pound), and the straps that encircle the head can be uncomfortable. Interacting with virtual objects requires precise hand and finger movements. What's more, some people doubt that people like Jeremy can even understand a question or produce a response. And yet, in study after study, we have found that most nonspeaking autistic teenage and adult participants can wear the HoloLens 2, and most can type short words on the HoloBoard.
The HoloBoard prototype that Jeremy first used in 2023 was three years in the making. It had its origins in an interdisciplinary feasibility study that considered whether individuals like Jeremy could tolerate a commercial AR headset. That study was led by the three of us: a developmental psychologist (Vikram Jaswal at the University of Virginia), an electrical and software engineer (Diwakar Krishnamurthy at the University of Calgary), and a computer scientist (Mea Wang, also at the University of Calgary).
Our journey to this point was not smooth. Some autism researchers told us that nonspeaking autistic people "do not have language" and so couldn't possibly communicate by typing. They also said that nonspeaking autistic people are so sensitive to sensory experiences that they would be overwhelmed by augmented reality. But our data, from more than a half-dozen peer-reviewed studies, have shown both assumptions to be wrong. And those results have informed the tools we're creating, like the HoloBoard, to enable nonspeaking autistic people to communicate more effectively.
Nonspeaking autistic people may also appear inattentive, engage in impulsive behavior, and score poorly on standard intelligence tests (many of which require spoken responses within a set amount of time). Historically, these challenges have led to unfounded assumptions about these individuals' ability to understand language and their capacity for symbolic thought. To put it bluntly, it has sometimes been assumed that someone who can't talk is also incapable of thinking.
Most attempts to provide nonspeaking autistic people with an alternative to speech have been rudimentary. Picture-based communication systems, often implemented on an iPad or tablet, are frequently used in schools and therapy clinics. If a user wants a cookie, they can tap a picture of a cookie. But the vocabulary of these systems is limited to the concepts that can be represented by a simple picture.
There are other options. Some nonspeaking autistic people have learned, over the course of many years and guided by parents and professionals, to communicate by spelling words and sentences on a letterboard that's held by a trained human assistant - a communication and regulation partner, or CRP. Part of the CRP's role is to provide attentional and emotional support, which can help with conditions that commonly accompany severe autism and that interfere with communication, including anxiety, attention-deficit hyperactivity disorder, and obsessive-compulsive disorder. Having access to such assisted methods of communication has allowed nonspeaking autistic people to graduate from college, write poetry, and publish a best-selling memoir.
But the role of the CRP has generated considerable controversy. Critics contend that the assistants can subtly guide users to point to particular letters, which would make the CRP, rather than the user, the author of any words produced. If nonspeaking autistic people who use a letterboard really know how to spell, critics ask, why is the CRP necessary? Some professional organizations, including the American Speech-Language-Hearing Association, have even cautioned against teaching nonspeaking autistic people communication methods that involve assistance from another person.
And yet, research suggests that CRP-aided methods can teach users the skills to communicate without assistance; indeed, some individuals who previously required support now type independently. And a recent study by coauthor Jaswal showed that, contrary to critics' assumptions, most of the nonspeaking autistic individuals in his study (which did not involve a CRP) knew how to spell. For example, in a string of text without any spaces, they knew where one word ended and the next word began. Using eye tracking, Jaswal's team also showed that nonspeaking autistic people who use a letterboard look at and point to letters too quickly and accurately to be responding to subtle cues from a human assistant.
Read more of this story at SoylentNews.