Multimodal deep learning for cardiovascular disease detection using pulse wave and vocal signals: a prediction model development and validation study.

BACKGROUND

Cardiovascular diseases (CVDs), particularly coronary artery disease (CAD) and its severe sequela, heart failure (HF), pose a massive global public health burden, necessitating efficient, non-invasive early screening tools. Traditional Chinese Medicine (TCM) offers a unique holistic diagnostic paradigm through “four examinations,” where “pulse diagnosis” (Mai Zhen, perceiving internal hemodynamics) and “auscultation-olfaction diagnosis” (Wen Zhen, assessing external systemic manifestations via voice) are pivotal. However, these subjective methods lack objective quantification. This study aims to scientifically operationalize the TCM “combined diagnosis” (He-Can) principle for automated CVD discrimination through a prediction model development and validation study.

METHODS

We implemented a rigorous multimodal data acquisition protocol. Radial artery pulse waves (reflecting TCM pulse diagnosis) were collected under standardized conditions, and vocal signals (reflecting TCM auscultation) of sustained /a:/ phonation were synchronously recorded from 553 subjects (Healthy, CAD, HF). To assess the potential added value of multimodal integration, we systematically compared single-modality models (pulse-only and voice-only) with multimodal fusion models using four deep learning architectures (MLP, GAN-Discriminator, ResNet-MLP, and Bi-LSTM) under ten-fold cross-validation.

RESULTS

In an independent external validation cohort (n = 194), the fused model achieved an accuracy of 0.7165 and an AUC of 0.8454, indicating maintained performance on unseen data. Explainable AI analyses (SHAP and LIME) suggested that the model’s predictions drew on both pulse-derived hemodynamic features and vocal acoustic features. Among the higher-contributing features were pulse dynamic variables (e.g., t3/tmax) and vocal acoustic biomarkers (e.g., MFCCs), suggesting that both modalities contributed to model predictions. IMPLICATION: This study provides empirical support for the feasibility of digitally integrating selected TCM-informed diagnostic signals within a prediction model framework. By digitizing and integrating pulse and voice signals, we developed a multimodal prediction model with potential utility for non-invasive cardiovascular screening in primary care, while linking the TCM concept of combined diagnosis with contemporary artificial intelligence and cardiovascular physiology.