Harnessing multimodal emotion features in depression detection across gender: Integrating large language model, acoustic fusion and facial expression recognition.

BACKGROUND

Depression is a complex and multidimensional mood disorder that manifests across text, audio, and visual modalities. Multimodal features capture the emotional diversity and may differ by gender. Therefore, this study aims to evaluate tri-modal emotion fusion for depression detection and gender-specific patterns.

METHODS

This study consists of 189 individuals (male: 103 and female: 86) obtained from the DAIC dataset. The dataset focuses on each participant's semi-structured depression interviews, including verbatim transcripts, audio, and blurred-face videos. Text features were processed by an LLM, audio/video by CNN-BLSTM-Transformer/SVM, and fusion for depression detection by Gibbs and XGBoost. We used SHAP and Procrustes analyses to explore gender-specific tri-modal emotion combinations across semantic, acoustic, and visual features.

RESULTS

Tri-modal emotion combinations significantly outperformed single-modal and dual-modal baselines (all p < 0.001) with best metrics (MSE = 18.06, RMSE = 4.25). Men may be more likely to report symptoms like anger, fear, and surprise, as opposed to the more commonly recognized symptoms like sadness or loss of interest. SHAP analysis showed that the most important tri-modal emotion combination was "Surprise(Text)-Fear(Audio)-Angry(Visual)" in males (accuracy = 75%, precision = 66.7%) and "Sadness(Text)-Sadness(Audio)-Sadness(Visual)" in females (accuracy = 72.7%, precision = 75%). Depressed males and females differed significantly in semantic, acoustic, visual, and tri-modal emotion associations (p < 0.05).

CONCLUSIONS

Combining diverse modal emotional expressions is a more comprehensive and accurate approach for detecting depression. Men often externalize depression through high-arousal emotions, whereas females express low-arousal internalizing states. These gender differences are essential for developing specific multimodal detection systems for the depressed population.