DSI Professor Leads Team to Design Smart Headphones

Fred Jiang
Fred Jiang

To counter a growing public safety concern, a professor at the Data Science Institute is leading a research team to design an intelligent headphone system that will warn pedestrians of imminent dangers from vehicles.

Pedestrians who wear headphones with handheld devices in cities face a greater risk of being hit by a vehicle. Thenumber of injuries and deaths to them has tripled in the last seven years in the U.S., according to one study. Walkers wearing headphones cannot hear the auditory cues --- horns, shouts or the sound of an approaching car -- that signal impending danger.

Therefore, the intelligent wearable headphone system will be outfitted with miniature microphones and intelligent signal processing to detect sounds of approaching vehicles. If a hazard appears near, the system will send an audio alert to the headphones of the pedestrian. Once developed, the intelligent wearable system could help reduce pedestrian injuries and fatalities from accidents.

The four-year research project, supported by a $1.2 million grant from the National Science Foundation, is directed by Fred Jiang, a professor of electrical engineering at Columbia Engineering and affiliate of the Data Science Institute, where he co-chairs the Smart Cities Center.  The research team includes Peter Kinget, chair of the Electrical Engineering Department at Columbia and an affiliate of the Data Science Institute; Shahriar Nirjon, aprofessor of Computer Science at the University of North Carolina (UNC) at Chapel Hill; and Joshua New, a psychology professor from Barnard College. Graduate students from both Columbia and UNC will work on the project.

The researchers intend to publish their findingsso other developers can adapt it for other applications. In the workplace, for instance, construction workers often wear headsets to muffle the sounds of heavy machinery. But that means they also sometimes can’t hear sounds of approaching hazards. Their headsets could be retrofitted with the intelligent audio-alert system. It could also be mounted on children’s backpacks to warn them of impending dangers, or could be embedded in people’s clothing to combat potential hazards in the home, office, or the street.

As researchers, we are exploring a new area -- in developing an inexpensive and low-power technology that creates an audio-alert mechanism for pedestrians,” says Jiang. “We’ll develop a few prototypes that we’ll test in the lab and on city streets. We hope the technology will eventually be transferred and commercialized.”

 The technology being developed by the team is complex. It involves embedding multiple miniature microphones in the headset as well as developing a data  pipeline to process all the sounds near to the pedestrian and extract the correct cues that signal impending danger. The pipeline will contain a new, ultra-low power, custom-integrated circuit that extracts the relevant features from the sounds while using very little battery power. This circuit will reduce the battery size and increase the operation time of the headset. Machine-learning classifiers will classify hundreds of acoustical cues to localize nearby vehicles and warn users when they are in danger. The alert mechanism must be designed so that people will recognize it and respond quickly.

 New, the psychology professor, says he’ll conduct perceptual and behavioral experiments with people to address how the alerts can be most effectively provided to users.

 “This may not seem like a difficult problem but it cannot be assumed that people will always notice any kind of alert generated – or be able to understand them quickly enough to respond,” says New. “There is research suggesting, for instance, that many children sleep straight through fire alarms – alarms that simple intuition would suggest are easily detectable. So we’ll be testing whether the alerts should be simple tones, verbal alerts, or synthesized sound.”

 Kinget, an electrical engineer who specializes in designing nanoscale integrated analog circuits, has worked in industrial research and still interfaces with companies on research projects. In his view, universities should undertake high-risk research projects that companies, driven by short-term profits and beholden to investors, can’t do. And the intelligent wearable system is a perfect example of this kind of research, he says.

 “Our team has a chance to build something new,” adds Kinget. “We are taking a big risk and accepting a major challenge in trying to build such a complicated system. But on the other hand, if it works we’ll see a great reward — knowing that a technology we created will help many people to live safer lives.”

-- By Robert Florida

550 W. 120th St., Northwest Corner 1401, New York, NY 10027    212-854-5660
©2017 Columbia University