Decoding Hybrid EEG-fNIRS Upper Limb Motor Execution with Capsule Dynamic Graph Convolutional Neural Network.

In this study, we proposed a capsule dynamic graph convolution network (EF-CapsDGCN) for accurate decoding of upper limb motor execution (ME) based on both electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals. In EF-CapsDGCN, EEG/fNIRS features are extracted using the same convolutional architecture but different parameter settings. The extracted features from both modalities are then dynamically routed to capsules. Afterwards, the single-modality capsules are concatenated to form EEG-fNIRS multimodal capsules. Each capsule is treated as a graph node, and hidden feature representations are learned through dynamic graph convolution. Finally, after concatenating the original capsules with the learned hidden features, the combined features are passed through multi-head self-attention and then flattened to feed into a fully connected layer for classification. Compared to current state-of-the-art methods such as ANN, DeepConvNet, DNN, and EF-Net, the proposed method demonstrated superior classification performance on the multimodal EEG-fNIRS dataset HYGRIP. Furthermore, our model achieves at least 8% higher classification accuracy in multimodal EEG-fNIRS compared to single modality EEG/fNIRS. These results demonstrate the potential of capsule dynamic graph convolution for the multimodal fusion of EEG and fNIRS. The proposed model is promising for accurately decoding motor execution-based brain computer interfaces with EEG-fNIRS multiple signals. Overall, this study provides an effective solution for multimodal-BCI decoding.Clinical Relevance- This study demonstrates that integrating EEG and fNIRS signals via a capsule dynamic graph convolution network (EF-CapsDGCN) improves upper limb motor execution decoding accuracy by at least 8% compared to single-modality approaches, offering clinicians a more reliable tool for developing brain-computer interface systems to enhance rehabilitation or assistive device control in patients with motor impairments.