BEARS (VR)
Both-Ears spatial audio training in virtual reality for young Cochlear Implant users.
Background & Motivation
BEARS is a Virtual Reality (VR) package that targets young people with bilateral Cochlear Implants to develop their spatial listening skills with both ears and improve their quality of life through gamified rehabilitation.
Cochlear Implants are electronic devices that stimulates the auditory nerve through electrodes placed in the cochlea of the inner ear, allowing some severely deaf people to perceive sounds. This is usually used in the form of medical intervention to help people to regain their hearing. The pioneer research started in UK in 1979 [1], In Year 2020/21, 433 children and 415 adults in UK received cochear implants [2]. In the past, surgeons would usually put implant on one side. As the technology matures, more and more people with hearing loss are going to receive implants for both of their ears, mostly at very young age. However, there isn't any formalised research thus no guideline to systemically support these implant receivers and their families in teaching themselves how to use CI for active listening with both ears. That's why BEARS project came into place to address the gap.
There is lots of research showing that sound localisation, as well as speech understanding in noise and music listening, can get better with training [3]. Human brain is so flexible and adaptable that it continues to develop in adults and children.
Audio Experience Design (AXD) at Imperial studies the physical and perceptual nature of sound and applies novel techniques in wearable interactive audio systems and hearing devices. AXD's work has demonstrated effects on 3D sound spatialisation (localising the source of sound in a 3-dimensional space) with virtual reality (VR) based applications. In BEARS project, I am facilitating the game development team and the clinical team to understand the needs for this applicaiton and design better experience for young cochlear users.
I devoted 6 months in this product as a research intern and followed the development phase for another 3 months to provide my technical support for the clinical trial, including add-on content production, debugging, and VR device configuration.
Gameplay
We design 3 categories of mini-games in order to give players very rich play experiences and each game type targets specific skillsets. In general, the three BEARS mini-games are:
1) Target Spatialisation (sound localisation);
2) Speech perception in noise;
3) Music appreciation.
The game development team worked closely with Speech and Hearing experts from UCL and neuro-scientists and audiologists from Cambridge to create an interesting context that makes sense to participants of the upcoming clinical trials. For example, to create fun and vivid interaction that involves the children's active engagement with their body and senses for speech-perception game, we framed the play in the context of the BEAR dinor where space travellers (customers) addressed in fantasy clothes will come to order food and drinks. To facilitate the onboarding process of both clinicians and young players, we designed tutorials to teach the player step-by-step how to interact with game objects and take actions in VR.
There are many other features implemented and suggested by the game development team to make the experience
Target Spatialisation
A target with a special audio cue will be generated around the player, and the goal for the player is to locate the target as quick as possible.
Speech perception in noise
Customers will make orders among other talking voices, and the goal for the player is to locate the customer asking for the order, and understand what is being ordered in order to serve the right food.
Music Appreciation
A variety of music-related challenges are included in BEARS music to introduce to our players how to interpret and appreciate musical features.
My Contribution
In this project, I explored the frontier of spatial audio and user experience in VR for hearing rehabilitation as a clinical intervention. My main tasks were to produce appropriate game materials, design children-friendly interactions, and outline strategies to keep players motivated using a rational level design approach. I supported both the game development team and clinical trial team in end-user research, digital assets generation, and player experience optimisation.
I embraced the iterative process of game development and participatory nature of this research/development project, which entails creating functional prototypes, collecting feedback from Patient and Public Involvement (PPI) activities, and improving interaction flows to maximise players’ satisfaction.
Using insights from our human-centred design research, I designed rich contents and improved usability of the game based on player interviews, digital playtesting, pilot trials, and extensive reviews with subject domain experts.
I proposed and implemented two new game mechanics and developed the existing ones further with contextualised interactions. I co-created various game interfaces and the narrative between the overall quest. These efforts would allow our players with delayed language perception and difficulty in hearing to understand and register the instructions embedded within our application.
Meanwhile, I communicated our team’s work effectively using design documentation, storyboards, and video demonstrations. My engagement with the general public at conferences and community outreach events extended the social impact of BEARS project.
Our Team & Collaborators
BEARS is funded by the NIHR and would run for 5 years. The BEARS's multi-disciplinary team, consisting of clinical scientists, developmental scientists, speech and hearing scientists, language experts from topic research institutions and Guy's and St Thomas hospital.
I am affiliated with Imperial Audio Experience Design and formed an agile development team with the music producer and the game developer, supporting product management, patient engagement, and game design.
Audio Experience Design (AXD) at Imperial studies the physical and perceptual nature of sound and applies novel techniques in wearable interactive audio systems and hearing devices.
Reactify Music LLP (Reactify) is a unique music production company focusing on next-generation music and sound formats, such as interactive, generative, and reactive techniques, via mediums like games, mobile apps, physical installations, and web-based experiences.
Join the BEARS Community!
If you are intereted in knowing more about this research project, please contact Dr Picinali or the clinical lead for more information.
Development Update
Now I am still working for BEARS projects and continue to develop the design to suit our community's demand. Find our latest updates on this development log.