Singapore Institute of Technology
tong20b_interspeech.pdf (622.14 kB)

Automatic assessment of dysarthric severity level using audio-video cross-modal approach in deep learning

Download (622.14 kB)
conference contribution
posted on 2024-04-03, 05:54 authored by Han Tong, Hamid Sharifzadeh, Ian McLoughlinIan McLoughlin

Dysarthria is a speech disorder that can significantly impact a person’s daily life, and yet may be amenable to therapy. To automatically detect and classify dysarthria, researchers have proposed various computational approaches ranging from traditional speech processing methods focusing on speech rate, intelligibility, intonation, etc. to more advanced machine learning techniques. Recently developed machine learning systems rely on audio features for classification; however, research in other fields has shown that audio-video cross-modal frameworks can improve classification accuracy while simultaneously reducing the amount of training data required compared to uni-modal systems (i.e. audio- or video-only).

In this paper, we propose an audio-video cross-modal deep learning framework that takes both audio and video data as input to classify dysarthria severity levels. Our novel cross-modal framework achieves over 99% test accuracy on the UASPEECH dataset — significantly outperforming current uni-modal systems that utilise audio data alone. More importantly, it is able to accelerate training time while improving accuracy, and to do so with reduced training data requirements.


Journal/Conference/Book title

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, October 25–29, 2020, Shanghai, China.

Publication date



  • Published

Rights statement

Tong, H., Sharifzadeh, H., McLoughlin, I. (2020) Automatic Assessment of Dysarthric Severity Level Using Audio-Video Cross-Modal Approach in Deep Learning. Proc. Interspeech 2020, 4786-4790, doi: 10.21437/Interspeech.2020-1997.

Usage metrics


    Ref. manager