Annular channel eigenmodes: a physical-layer approach to suppressing OAM modal crosstalk.

Modal crosstalk is a fundamental limitation for orbital angular momentum (OAM)-based spatial-division multiplexing. Here, we introduce annular channel eigenmodes (ACEs) -rigorously derived as the optimal band-limited solution for maximizing energy concentration within distinct annular channels. This approach reformulates the design as a Hermitian eigenvalue problem, efficiently yielding optimal beams that are physically isolated in space. Numerical simulations demonstrate that under identical conditions, conventional Gaussian-enveloped perfect optical vortices (POVs) exhibit an average modal crosstalk of -16 dB, whereas ACEs suppress crosstalk to nearly -30 dB. Moreover, the crosstalk suppression of ACEs continues to improve exponentially with increasing channel width, while that of POVs saturates at a fundamental limit. We experimentally generated ACEs and confirmed a 36% enhancement in energy confinement relative to POVs. ACEs thus provide a physically robust basis for high-fidelity, high-density OAM communications.