Near-infrared spectral generation and regression modeling with a hybrid CVAE-1D-CNN framework: application to soil organic matter estimation.

BACKGROUND

Near-infrared (NIR) spectroscopy combined with chemometric modeling is a widely used rapid and non-destructive analytical technique, showing promise in estimating soil organic matter (SOM). However, acquiring a sufficient number of experimental spectra and corresponding SOM values is time-consuming, expensive, and often impractical, which can compromise the accuracy of regression models. Therefore, there is an urgent need for a strategy that can overcome data scarcity while maintaining robust estimation.

RESULTS

This study development a hybrid framework that integrates a conditional variational autoencoder (CVAE) with a one-dimensional convolutional neural network (1D-CNN) for spectral data generation and regression modeling. The CVAE generated realistic spectra conditioned on target SOM content, and the generated spectra were combined with measured spectra to form an augmented dataset for regression modeling. The results showed that the CVAE accurately reproduced key spectral features and generated spectra consistent with the specified SOM values. The augmented dataset improved the estimation performance of the regression models. Among these models, the 1D-CNN outperformed partial least squares regression (PLSR) and random forest (RF), highlighting its superior ability to extract informative features from spectral data.

SIGNIFICANCE: A novel spectral analytical methodology that help alleviate data scarcity and enhances regression performance under limited-sample conditions was established. By combining data augmentation with advanced regression models, the approach advances rapid and non-destructive soil analysis and provides a useful reference for other spectroscopic applications facing sampling limitations.