You see them all over city streets: pedestrians wearing headphones or earbuds – their faces glued to their phones as they stroll along oblivious to their surroundings.

Known as “twalking,” the behavior is not without its dangers. Headphone-wearing pedestrians often can’t hear the auditory cues – horns, shouts, or the sound of approaching cars – that signal imminent harm. As a result, the number of injuries and deaths caused by twalking in the U.S. has tripled in the last seven years. Last year, pedestrian deaths in the U.S. were at their highest level since 1990.

To counter this growing public safety concern, researchers at the Data Science Institute at Columbia University are designing an intelligent headphone system that warns pedestrians of imminent dangers. The headphones have miniature microphones and intelligent signal processing that detects sounds of approaching vehicles. If a hazard appears near, the system sends an audio alert to the pedestrian’s headphones. The research team is developing prototypes and testing them on streets close to Columbia. Once developed, the intelligent wearable system could help reduce pedestrian injuries and fatalities.

“We are exploring a new area in developing an inexpensive and low-power technology that creates an audio-alert mechanism for pedestrians,” says Fred Jiang, a Data Science Institute member and an assistant professor of electrical engineering at Columbia Engineering.

The smart-headphone project was awarded a $1.2 million grant from the National Science Foundation in 2017, and the team has since published two conference papers as well as a journal paper in IEEE Internet of Things Journal on their research. They have also received several honors including a best demo award from an ACM conference and a best presentation award from an IEEE conference.  The research team includes Peter Kinget, chair of electrical engineering at Columbia and a Data Science Institute member; Shahriar Nirjon, a professor of computer science at the University of North Carolina (UNC) at Chapel Hill; and Joshua New, a psychology professor from Barnard College. Graduate students from both Columbia and UNC also work on the project.

The research and development of the smart headphones is complex: It involves embedding multiple miniature microphones in the headset as well as developing a low-power data pipeline to process all the sounds near to the pedestrian. It must also extract the correct cues that signal impending danger. The pipeline will contain an ultra-low power, custom-integrated circuit that extracts the relevant features from the sounds while using little battery power.

The researchers are also using the most advanced data science techniques to design the smart headset. Machine-learning models on the user’s smartphone will classify hundreds of acoustical cues from city streets and nearby vehicles and warn users when they are in danger. The mechanism will be designed so that people will recognize the alert and respond quickly. The team is now testing its design both in the lab and on the streets of New York – a city known for its congestion and its cacophony of sounds. New, the psychology professor from Barnard, says he will conduct perceptual and behavioral experiments with people to see how the alerts can be effectively provided to pedestrians who walk in cities wearing headphones.

Jiang said his aim is to develop a prototype of the smart headphone system at Columbia and then transfer the technology to a commercial company.

“We hope that once refined,” he says, “the technology will be commercialized and mass produced in a way that will help cities reduce pedestrian fatalities.”

— Robert Florida