A Multi-Modal Approach to Emotion-Aware Automatic Speech Recognition Using Dy namic Emotion Trajectories and Global Style Tokens.

Abstract

This paper presents a novel approach to emotion-aware Automatic Speech Recognition (ASR) by
integrating emotion detection and dynamic emotion trajectory modelling. Our system combines
acoustic features and physiological signals to achieve more accurate and contextually aware
emotion recognition. The use of Global Style Tokens (GSTs) enhances the system's ability to de
tect nuanced emotional transitions in speech, outperforming existing state-of-the-art models.
We evaluate the system using the IEMOCAP and RAVDESS datasets, achieving emotion classifi
cation accuracies of 88.1% and 85.6%, respectively. The system also achieves an F1-score of 0.87
on IEMOCAP and 0.85 on RAVDESS, with Area Under the Curve (AUC) scores of 0.90 and 0.88,
and Root Mean Squared Error (RMSE) values of 0.13 and 0.14. Additionally, we propose future
enhancements, including expanding multimodal inputs, improving real-time scalability, han
dling mixed emotions, and adapting the model for cross-lingual and low-power environments.
Our findings contribute to the development of emotionally intelligent ASR systems capable of
improving human-computer interactions in various applications.