New AR System Alters Sight, Sound and Touch

There's the typical pitter-patter sound that comes from a person drumming their fingers along a tabletop. But what if this normal pitter-patter sound was perceived as a series of hollow echoes? Or rolling thunder? A new augmented reality system called Tactile Echoes provides users with experiences like this, and could be used for a wide range of gaming, entertainment and research purposes.

Notably, the system does not require any equipment between the user's fingertips and the contact surface, meaning users can enjoy the real sensation of their environment along with the visual, haptic and auditory augmented enhancements.

Tactile Echoes is possibly the first programmable system for haptic augmented reality that allows its users to freely touch physical objects or surfaces augmented with multimodal digital feedback using their hands," says Anzu Kawazoe, a PhD candidate at the University of California, Santa Barbara who co-designed Tactile Echoes.

It accomplishes this using a sensor that is placed on the top of the user's fingernail, which detects the vibrations that are naturally produced within the finger as it touches a surface. The vibrational signals are processed and translated into programmed sounds.

Different tactile feedback and sounds can be played with each interaction, because the vibrational patterns in our fingers change depending on what surface we touch, or the intensity of pressure applied. For example, Tactile Echoes may play a light, fun echo when you tap an object lightly, or play a sudden thud if you jab the object with force.

We were motivated by the idea of being able to almost magically augment any ordinary object or surface, such as a simple wooden table, with lively haptic and effects that playfully respond to how or where we touch," explains Kawazoe.

Her team took the system one step further by integrating the wearable device with virtual environments created by a smart projector or a VR or AR headset. In this way, users can touch" virtual objects in their real environment, and experience enhanced graphic, sound and haptic feedback.

The researchers tested Tactile Echoes through a number of user experiments, described in study published May 26 in IEEE Transactions on Haptics. First, study participants were asked to describe different sounds, and what perceptions and associations are evoked with each one. In a second experiment, participants used Tactile Echoes to complete an interactive, augmented reality video game that was projected onto an ordinary desktop surface. Users reported that the Tactile Echoes feedback greatly enhanced the responsiveness, and their level of engagement and agency in playing the game.

While Tactile Echoes is still a prototype, Kawazoe says her team is interested in collaborating with companies to commercialize this tech. Some of the most promising applications we can envisage include augmented reality games that can be played on any table top, musical devices whose interface can be projected wherever needed, and educational systems for enlivening learning by K-12 students," she says.

She also notes that her team's research so far with Tactile Echoes has revealed some interesting perceptual phenomena that occur when haptic feedback is delayed through the system. In particular, they believe that perceptual masking is happening, whereby the perception of one stimulus affects the perceived intensity of a second stimulus.

We are thinking that this tactile masking effect is working on the Tactile Echoes system. Specifically, time-delayed tactile feedback is as perceived stronger," explains Kawazoe. We are preparing new experiments to investigate these effects, and plan to use the results to further improve the Tactile Echoes system."