AI System Warns Pedestrians Wearing Headphones About Passing Cars
How can headphone-wearing pedestrians tune out the chaotic world around them without compromising their own safety? One solution may come from the pedestrian equivalent of a vehicle collision warning system that aims to detect nearby vehicles based purely on sound.
The intelligent headphone system uses machine learning algorithms to interpret sounds and alert pedestrians to the location of vehicles up to 60 meters away. A prototype of the Pedestrian Audio Warning System (PAWS) can only detect the location but not the trajectory of a nearby vehicle-never mind the locations or trajectories of multiple vehicles. Still, it's a first step for a possible pedestrian-centered safety aid at a time when the number of pedestrians killed on U.S. roads reached a three-decade high in 2018.
"Sometimes the newer vehicles have sensors that can tell if there are pedestrians, but pedestrians usually don't have a way to tell if vehicles are on a collision trajectory," says Xiaofan Jiang, an assistant professor of electrical engineering and member of the Data Science Institute at Columbia University.
The idea first came to Jiang when he noticed that a new pair of noise-cancelling headphones was distracting him more than usual from his surroundings during a walk to work. That insight spurred Jiang and his colleagues at Columbia, the University of North Carolina at Chapel Hill, and Barnard College to develop PAWS and publish their work in the October 2019 issue of the IEEE Internet of Things Journal.
Photo: Electrical Engineering and Data Science Institute/Columbia University The Pedestrian Audio Warning System detects nearby cars by using microphones and machine learning algorithms to analyze vehicle sounds. Many cars with collision warning systems rely upon visual cameras, radar, or lidar to detect nearby objects. But Jiang and his colleagues soon realized that a pedestrian-focused system would need a low-power sensor that could operate for more than six hours on standard batteries. "So we decided to go with an array of microphones, which are very inexpensive and low-power sensors," Jiang says.
Read this article for free on IEEE Xplore until 28 January 2020.The array of four microphones is located in different parts of the headphone. But the wearable warning system's main hardware is designed to fit inside the left ear housing of commercial headphones and draws power from a rechargeable lithium-ion battery. A custom integrated circuit saves on power by only extracting the most relevant sound features from the captured audio and transmitting that information to a paired smartphone app.
The smartphone hosts the machine learning algorithms that were trained on audio from 60 different types of vehicles in a variety of environments: a street adjacent to a university campus and residential area, the side of a windy highway during hurricane season, and the busy streets of Manhattan.
However, relying purely on sound to detect vehicles has proven tricky. For one thing, the system tends to focus on localizing the loudest vehicle, which may not be the vehicle closest to the pedestrian.The system also still has trouble locating multiple vehicles or even estimating how many vehicles are present.
Photo: Electrical Engineering and Data Science Institute/Columbia University The hardware for the Pedestrian Audio Warning System can fit inside the ear housing of commercial headphones. As it stands, the PAWS capability to localize a vehicle up to 60 meters away might provide at least several seconds of warning depending on the speed of an oncoming vehicle. But a truly useful warning system would also be able to track the trajectory of a nearby vehicle and only provide a warning if it's on course to potentially hit the pedestrian. That may require the researchers to figure out better ways to track both the pedestrian's location and trajectory along with the same information for vehicles.
"If you imagine one person walking along the street, many cars may pass by but none will hit the person," Jiang explains. "We have to take into account other information to make this collision detection more useful."
More work continues on how the system would use noises or other signals to alert headphone wearers. Joshua New, a behavioral psychologist at Barnard College, plans to conduct experiments to see what warning cue works best to give people a heads up. For now, the team is leaning toward either providing a warning beep on one side of a stereo headphone or possibly simulating 3D warning sounds to provide more spatially-relevant information.
Beyond ordinary pedestrians, police officers performing a traffic stop on a busy road or construction workers wearing ear protection might also benefit from such technology, Jiang says. The PAWS project has already received US $1.2 million from the National Science Foundation, and the team has an eye on eventually handing a more refined version of the technology over to a company to commercialize it.
Of course, one technology will not solve the challenges of pedestrian safety. In its 2019 report, the Governors Highway Safety Association blamed higher numbers of pedestrian deaths on many factors such as a lack of safe road crossings, and generally unsafe driving by speeding, distracted, or drunk drivers. A headphone equipped with PAWS is unlikely to prevent even a majority of pedestrian deaths-but a few seconds' warning might help spare some lives.
< Back to IEEE Journal Watch