Abstract
Linear mixed-effects model (LMM) has been widely used in hierarchical and longitudinal data analyses. In practice, the fitting algorithm can fail to converge because of boundary issues of the estimated random-effects covariance matrix, i.e., being near-singular, non-positive definite, or both. The traditional grand mean centering technique cannot generally improve the numerical stability and may even increase the correlation between random-effects. Also, current available algorithms are not computationally optimal because the condition number of random-effects covariance matrix is unnecessarily increased when the random-effects correlation estimate is not zero. To improve the convergence of data with such boundary issue, we propose an adaptive fitting (AF) algorithm using an optimal linear transformation of the random-effects design matrix. It is a data-driven adaptive procedure, aiming at reducing subsequent random-effects correlation estimates down to zero in the optimal transformed estimation space. Extension of the AF algorithm to multiple random-effects models is also discussed. The AF algorithm can be easily implemented with standard software and be applied to other mixed-effects models. Simulations show that the AF algorithm significantly improves the convergence rate, and reduces the condition number and non-positive definite rate of the estimated random-effects covariance matrix, especially under small sample size, relative large noise, and high correlation settings. We also propose a new two-step modeling strategy for LMM fitting and random-effects selection. This parsimonious LMM with uncorrelated random-effects in the optimal transformed space is favored by the likelihood ratio test and Akaike Information Criterion. Two real life longitudinal data sets are used to illustrate the application of this AF algorithm implemented with software package R (nlme).
