AI-Based Sound Cue Visualization Prototype Study for Improving Interaction Experience of Hearing-impaired in VR Environment
In response to the accessibility challenges faced by hearing-impaired individuals in virtual reality (VR) environments, this study proposes an AI-based sound cue visualization system to enhance their interaction experience. The advancement of XR technologies has revolutionized immersive experiences, yet hearing-impaired users continue to face significant constraints in both verbal and non-verbal communication. Addressing these limitations, this research develops a system leveraging the YAMNet model, an audio event classification model capable of distinguishing between 521 audio classes, to detect and visualize auditory information in real-time. At the core of the system architecture lie three main modules: sound detection and classification, visualization transformation, and VR interface. Through these modules, we implement three distinct prototypes: spatial experience support recognizing environmental and animal sounds to enhance immersion; emotion recognition activation analyzing vocal characteristics to improve communication abilities; and safety and emergency response for detecting and x-alerting environmental hazards. Built on Unity with Meta Quest 3 hardware, the prototype utilizes Sentis for efficient on-device deep learning inference. To evaluate the system's effectiveness, this study proposes comprehensive user experience assessments targeting hearing-impaired individuals and their support personnel. These evaluations will measure VR environment immersion, information comprehension, and satisfaction using a 5-point Likert scale. While representing a significant step toward digital inclusivity, this research requires further empirical testing with hearing-impaired individuals and stakeholders. Future development paths include conducting real user tests in multiplayer VR environments, advancing sound recognition capabilities, and exploring sign language-to-text conversion systems, ultimately contributing to an equitable digital environment where universal participation becomes reality.