SACM: SEEG-Audio Contrastive Matching for Chinese Speech Decoding.

OBJECTIVE

Speech disorders such as dysarthria and anarthria can severely impair patients' ability to communicate verbally. Speech decoding brain-computer interfaces (BCIs) offer a potential alternative by directly translating speech intentions into spoken words, serving as speech neuroprostheses.

METHODS

This paper reports an experimental protocol for Mandarin Chinese speech decoding BCIs and proposes a contrastive learning-based decoding algorithm termed SEEG and Audio Contrastive Matching (SACM). Stereo-electroencephalography (SEEG) and synchronized audio data were collected from ten patients with drug-resistant epilepsy as they performed a word-level reading task. SACM leverages the cross-modal correlation between neural activity and audio signals to decode matched speech segments.

RESULTS

The proposed framework achieved accuracies significantly exceeding random matching in both isolated-word and continuous speech decoding tasks, and outperformed SEEG-only baselines across seven backbone architectures in the isolated-word setting. Electrode- wise analysis revealed that a single ventral sensorimotor cortex electrode achieved performance comparable to that of the full electrode array. Our code is publicly available.

SIGNIFICANCE: To our knowledge, this is the first work on multimodal decoding for tonal speech BCIs.