Lending your car an ear

Imagine all the sounds on the road that we experience on a daily basis. On our way to work; running errands; or meeting friends for a coffee in the afternoon: An electric vehicle hums friendly while passing pedestrians at an intersection. On the other side of the road a cyclist is ringing her bell while a dog barks at her from the sidewalk. In the distance you can hear an ambulance approaching. A tram is crossing the intersection with its metal wheels screeching in the tracks.

For human drivers, the ear is already one of the essential sensors to perceive the world around them – working in synch with the eyes. Cars have been relying on cameras for many years now to make sense of the world around them. And interior microphones already allow drivers and passengers to interact with their digital assistant in the car. 

But exterior microphones are just on the verge of becoming integrated into their exterior sensor set. CARIAD has been testing microphone-hardware and algorithms together with the Fraunhofer Institute for Digital Media Technology IDMT to make the acoustic perception technology ready for series production. The overall name of the project: The Hearing Car.

Many parts of The Hearing Car development are part of a larger research project called KI4Boardnet. CARIAD and Fraunhofer IDMT are members here as well and drive forward industry discussions around the potential of this novel sensor concept. It underlines the significance of exterior microphones in modern vehicle’s electrical/electronic architectures. 

To test the microphones and measurement systems CARIAD and the Fraunhofer IDMT went on a road trip to the proving ground in Sweden earlier this year. 

“Our tests show that exterior microphones can change the way we interact with our cars and improve safety for automated driving functions,” says Hagen Jaeger, Product Owner Exterior Acoustic Perception, at CARIAD. First analyses of data from road tests under various conditions from snow and ice to heat and wetness look promising.

Understanding the audio

Exterior microphones will sense an overwhelming array of different sounds from a car’s vicinity. According to the Wiki.Audio scientific platform, the noise level a pedestrian on the curbside experiences five meters away from the center of a busy road is about 80 decibels (dB). 

“But among all this noise, algorithms from The Hearing Car are able to identify specific sounds patterns. These could typically be emergency signals from an approaching ambulance”, says Moritz Brandes, Project Manager at Fraunhofer IDMT in Oldenburg. 

In the domain of fully automated driving and autonomous driving the siren detection will become crucial. As cars will then have to yield automatically to the emergency vehicles as soon as they hear their signal. 

Opening up a new UX dimension

What’s more, exterior microphones could be able to identify specific voice commands by the users. And even analyze specific sounds of the car’s body parts that might act as an early indicator for damage or wear.

Being calibrated with the voice of a driver, vehicles will unlock a whole new dimension of the digital experience. The AI-based digital assistants will engage with their drivers even on the outside of the car – allowing them to open up the trunk on command, for example.

With regards to predictive maintenance, exterior microphones could be able to detect anomalies in the sounds of body parts. They will enhance information provided by other sensors analyzing the state of health of these parts and issue early maintenance warnings before any greater damage can arise. In principle they can even detect road conditions for example, depending on the sound created by the tires on dry or wet or icy street surfaces. 

Another use-case being researched is amplifying sounds from the outside to the inside of the car – alerting drivers to situations around them they could be missing without the help of the new microphone. Imagine a cyclist ringing a bell on the right side of your car. This could be easily overheard while the driver is listening to the traffic news on the radio. With an exterior microphone the sound of the bicycle bell could be amplified and played from the car’s speakers on the right side of the sound system. Such an acoustically transparent car could enhance safety for drivers and other road users alike.