An open-set learning framework for EEG-based sleep stage classification.

Objective.Existing sleep stage classification methods generally rely on a closed set assumption. They assume all sleep categories are predefined, which overlooks unknown sleep patterns encountered in real world electroencephalogram (EEG) analysis. This study overcomes this constraint by identifying both known and unknown sleep stages.Approach.We developed a two stage open set learning framework for EEG sleep staging. First, an adaptive Gaussian method models each known category to compute Gaussian scores, utilizing class specific quantile adaptive thresholds to separate known from unknown samples. Second, we integrate a trainable decision boundary OpenMax (TDBOM) module. TDBOM optimizes decision boundaries during training to suppress samples with large boundary deviations, while calibrating normal samples to yield reliable probability estimates.Main results.Evaluated on three public EEG datasets, our framework accurately classifies known sleep stages and reliably detects unknown categories, significantly boosting open-set recognition performance.Significance.This work provides a practical and robust solution for EEG-based sleep stage classification under open-set conditions, facilitating more reliable sleep analysis and supporting potential applications in real-world clinical monitoring and sleep disorder assessment.