Multimodal behavioral phenotyping for depressive-spectrum classification and severity estimation using eye tracking, facial behavior, and transcript-derived language.

INTRODUCTION

Depression assessment remains largely dependent on symptom reports and clinician judgment, while objective tools for depressive-spectrum stratification and severity estimation remain limited. Existing digital and multimodal depression-detection studies often focus on binary case-control classification, handle missing modalities incompletely, provide limited calibration assessment, and rarely combine depressive-spectrum classification with continuous symptom-severity estimation. We therefore developed a quality-aware multimodal framework integrating eye tracking, facial behavior, and transcript-derived language for classification across normal control (NC), subthreshold depression (SD), and major depressive disorder (MDD), together with prediction of 17-item Hamilton Depression Rating Scale (HAMD-17) severity.

METHODS

A total of 186 participants completed a controlled task battery including interview, emotional reading, free viewing with verbal description, fixation, gaze orienting, smooth pursuit, prosaccade, and antisaccade tasks. Eye-tracking, facial-video, and transcript-derived language data were converted into modality-specific features. Baseline-3 combined modality-specific encoders, quality-aware gated fusion, and joint classification-regression learning under a nested repeated-resampling framework with explicit missing-modality handling. Baseline-3+ further incorporated Transformer-based cross-modal interaction and uncertainty-based dynamic task weighting. Performance was evaluated on held-out outer-loop test sets after temperature scaling. Interpretability analyses included gate profiling, selective prediction, SHAP, Integrated Gradients, and counterfactual analysis.

RESULTS

Baseline-3+ showed the most favorable classification and calibration profile, with accuracy, balanced accuracy, and F1-macro approaching 0.90 across both classification routes and lower expected calibration error than Baseline-3. For severity estimation, the improvement was route-dependent and mainly reduced the regression disadvantage observed under the hierarchical route. Misclassification was concentrated near the SD boundary. Interpretability analyses showed stable quality-aware modality reweighting, with facial features providing the dominant signal, complemented by eye tracking and smaller but meaningful language contributions.

DISCUSSION: This framework addresses key limitations of prior binary and incompletely calibrated depression-detection models by jointly supporting depressive-spectrum classification, severity estimation, missing-modality handling, calibrated prediction, and individual-level interpretation. Its most plausible role is to augment clinical assessment, particularly for boundary states such as SD.