Read the interview with PI Prof. Tapio Lokki and his winning proposal: Audio Rendering for Augmented Reality Telepresence (AURA)
PI Prof. Tapio Lokki and Aalto University
Tapio Lokki is a professor at the Aalto Acoustics Lab which is a world-know research unit on 3D sound, room acoustics, XR audio and audio signal processing at Aalto University. Prof. Lokki did his PhD on auralization over 20 years ago and has been studying sound rending in VR and AR applications over decades. In addition, Dr. Nils Meyer-Kahlen will do the research in SPIRIT/AURA project, and he recently defended his PhD work on transfer-plausible acoustics for AR. He is a world leading expert in AR sound evaluation and he masters 3D sound rendering methods.
Can you give a brief overview of your winning proposal?
What are its key objectives and innovative aspects?
The AURA project aims to address the challenge of binaural audio rendering in real-time holographic telepresence application. The challenge is tackled by implementing a complete dynamic, room-adaptive spatial audio rendering system that complements visual holographic calling technology. Moreover, we examine the room acoustic estimation and rendering accuracy requirements to ensure perceived audiovisual congruence between the audio rendering and the visual hologram. Finally, we demonstrate the overall impact on the quality of experience of binaural rendering in general and room adaptive acoustic rendering in particular. We will conduct our experiments with the following topics:
I. Implementation of blind room acoustic identification and spatial audio rendering software
II. Spatial congruence of acoustic and visual rendering depending on room estimation accuracy
III. Quality of experience evaluation based on a real-time, two-way holographic calling system
What motivated you to apply for the SPIRIT Open Call?
The SPIRIT platform implements a visual real-time holographic telepresence application. However, realistic sound environment is missing, and we love to contribute and bring our knowledge on AR sound and room acoustics to SPIRIT. For example, many open questions persist regarding the precise requirements for room acoustic identification and rendering in the context of sound for AR telepresence. In the AURA project we will identify spatial room impulse response (SRIR) from the user’s voice or other sound sources in the user’s environment. Some features of identified SRIRs could then be used to tune sound rendering algorithms to have a realistic sounding sound environment for users. Another less demanding goal is also of interest—audiovisual congruence, i.e., the perception that the sound and the visual holographic rendering originate from the same position in 3D space. Finally, we plan to conduct an overall quality of experience study to show the benefits of the developed audio rendering system.
How do you envision this project making an impact?
We plan to enhance the SPIRIT platform by making the implementation of our acoustic identification and spatial audio rendering technologies open source. Consequently, they can later be used by other SPIRIT project members. Having a full pipeline tailored to AR use cases will help our lab, other labs, and companies in future research and development projects regarding AR audio. Finally, by conducting the described quality of experience evaluation, we aim to demonstrate that room-adaptive spatial audio rendering benefits high-quality holographic calls in general. This would help decision-makers inside and outside of SPIRIT gauge the importance of such audio technologies.