Penalized likelihood estimation for semiparametric mixed models, with application to alcohol treatment research.

In this article, we implement a practical computational method for various semiparametric mixed effects models, estimating nonlinear functions by penalized splines. We approximate the integration of the penalized likelihood with respect to random effects with the use of adaptive Gaussian quadrature, which we can conveniently implement in SAS procedure NLMIXED. We carry out the selection of smoothing parameters through approximated generalized cross-validation scores. Our method has two advantages: (1) the estimation is more accurate than the current available quasi-likelihood method for sparse data, for example, binary data; and (2) it can be used in fitting more sophisticated models. We show the performance of our approach in simulation studies with longitudinal outcomes from three settings: binary, normal data after Box-Cox transformation, and count data with log-Gamma random effects. We also develop an estimation method for a longitudinal two-part nonparametric random effects model and apply it to analyze repeated measures of semicontinuous daily drinking records in a randomized controlled trial of topiramate.