A Multi-Modal Approach to Emotion-Aware Automatic Speech Recognition Using Dy namic Emotion Trajectories and Global Style Tokens. - A. Aoun , Abeer; B. Dabbabi, Karim (Libyan Journal of Science &Technology, 2026)
This paper presents a novel approach to emotion-aware Automatic Speech Recognition (ASR) by integrating emotion detection and dynamic emotion trajectory modelling. Our system combines acoustic features and physiological signals to achieve more accurate and contextually aware emotion recognition. The use of Global Style Tokens (GSTs) enhances the system's ability to de tect nuanced emotional transitions in speech, outperforming existing state-of-the-art models. We evaluate the system using the IEMOCAP and RAVDESS datasets, achieving emotion classifi cation accuracies of 88.1% and 85.6%, respectively. The system also achieves an F1-score of 0.87 on IEMOCAP and 0.85 on RAVDESS, with Area Under the Curve (AUC) scores of 0.90 and 0.88, and Root Mean Squared Error (RMSE) values of 0.13 and 0.14. Additionally, we propose future enhancements, including expanding multimodal inputs, improving real-time scalability, han dling mixed emotions, and adapting the model for cross-lingual and low-power environments. Our findings contribute to the development of emotionally intelligent ASR systems capable of improving human-computer interactions in various applications.