TY - JOUR
T1 - Multimodal Assessment of Parkinson's Disease
T2 - A Deep Learning Approach
AU - Vásquez-Correa, Juan Camilo
AU - Arias-Vergara, Tomas
AU - Orozco-Arroyave, J. R.
AU - Eskofier, Björn
AU - Klucken, Jochen
AU - Nöth, Elmar
N1 - Funding Information:
Manuscript received February 22, 2018; revised July 3, 2018 and August 13, 2018; accepted August 15, 2018. Date of publication September 24, 2018; date of current version July 1, 2019. This work was supported in part by CODI from University of Antioquia under Grants PRG-2015–7683 and PRV16-2-01, and in part by the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant 766287. The work of B. Eskofier was supported by the German Research Foundation (DFG) within the framework of the Heisenberg professorship program under Grant ES 434/8-1. (Corresponding author: Juan Camilo Vásquez-Correa.) J. C. Vásquez-Correa and J. R. Orozco-Arroyave are with the Faculty of Engineering, University of Antioquia UdeA, Medellín 050010, Colombia, and also with the Pattern Recognition Laboratory, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen 91054, Germany (e-mail:,jcamilo.vasquez@udea.edu.co; rafael.orozco@udea. edu.co).
Publisher Copyright:
© 2013 IEEE.
PY - 2019/7
Y1 - 2019/7
N2 - Parkinson's disease is a neurodegenerative disorder characterized by a variety of motor symptoms. Particularly, difficulties to start/stop movements have been observed in patients. From a technical/diagnostic point of view, these movement changes can be assessed by modeling the transitions between voiced and unvoiced segments in speech, the movement when the patient starts or stops a new stroke in handwriting, or the movement when the patient starts or stops the walking process. This study proposes a methodology to model such difficulties to start or to stop movements considering information from speech, handwriting, and gait. We used those transitions to train convolutional neural networks to classify patients and healthy subjects. The neurological state of the patients was also evaluated according to different stages of the disease (initial, intermediate, and advanced). In addition, we evaluated the robustness of the proposed approach when considering speech signals in three different languages: Spanish, German, and Czech. According to the results, the fusion of information from the three modalities is highly accurate to classify patients and healthy subjects, and it shows to be suitable to assess the neurological state of the patients in several stages of the disease. We also aimed to interpret the feature maps obtained from the deep learning architectures with respect to the presence or absence of the disease and the neurological state of the patients. As far as we know, this is one of the first works that considers multimodal information to assess Parkinson's disease following a deep learning approach.
AB - Parkinson's disease is a neurodegenerative disorder characterized by a variety of motor symptoms. Particularly, difficulties to start/stop movements have been observed in patients. From a technical/diagnostic point of view, these movement changes can be assessed by modeling the transitions between voiced and unvoiced segments in speech, the movement when the patient starts or stops a new stroke in handwriting, or the movement when the patient starts or stops the walking process. This study proposes a methodology to model such difficulties to start or to stop movements considering information from speech, handwriting, and gait. We used those transitions to train convolutional neural networks to classify patients and healthy subjects. The neurological state of the patients was also evaluated according to different stages of the disease (initial, intermediate, and advanced). In addition, we evaluated the robustness of the proposed approach when considering speech signals in three different languages: Spanish, German, and Czech. According to the results, the fusion of information from the three modalities is highly accurate to classify patients and healthy subjects, and it shows to be suitable to assess the neurological state of the patients in several stages of the disease. We also aimed to interpret the feature maps obtained from the deep learning architectures with respect to the presence or absence of the disease and the neurological state of the patients. As far as we know, this is one of the first works that considers multimodal information to assess Parkinson's disease following a deep learning approach.
KW - convolutional neural networks
KW - deep learning
KW - gait
KW - handwriting
KW - Parkinson's disease
KW - speech
UR - http://www.scopus.com/inward/record.url?scp=85052672836&partnerID=8YFLogxK
U2 - 10.1109/JBHI.2018.2866873
DO - 10.1109/JBHI.2018.2866873
M3 - Article
C2 - 30137018
AN - SCOPUS:85052672836
SN - 2168-2194
VL - 23
SP - 1618
EP - 1630
JO - IEEE Journal of Biomedical and Health Informatics
JF - IEEE Journal of Biomedical and Health Informatics
IS - 4
M1 - 8444654
ER -