TY - JOUR

T1 - Scalable and accurate variational Bayes for high-dimensional binary regression models

AU - Fasano, Augusto

AU - Durante, Daniele

AU - Zanella, Giacomo

PY - 2022

Y1 - 2022

N2 - Modern methods for Bayesian regression beyond the Gaussian response setting are often computationally impractical or inaccurate in high dimensions. In fact, as discussed in recent literature, bypassing such a trade-off is still an open problem even in routine binary regression models, and there is limited theory on the quality of variational approximations in high-dimensional settings. To address this gap, we study the approximation accuracy of routinely used mean-field variational Bayes solutions in high-dimensional probit regression with Gaussian priors, obtaining novel and practically relevant results on the pathological behaviour of such strategies in uncertainty quantification, point estimation and prediction. Motivated by these results, we further develop a new partially factorized variational approximation for the posterior distribution of the probit coefficients that leverages a representation with global and local variables but, unlike for classical mean-field assumptions, it avoids a fully factorized approximation, and instead assumes a factorization only for the local variables. We prove that the resulting approximation belongs to a tractable class of unified skew-normal densities that crucially incorporates skewness and, unlike for state-of-the-art mean-field solutions, converges to the exact posterior density as p → ∞. To solve the variational optimization problem, we derive a tractable coordinate ascent variational inference algorithm that easily scales to p in the tens of thousands, and provably requires a number of iterations converging to 1 as p → ∞. Such findings are also illustrated in extensive empirical studies where our novel solution is shown to improve the approximation accuracy of mean-field variational Bayes for any n and p, with the magnitude of these gains being remarkable in those high-dimensional p>n settings where state-of-the-art methods are computationally impractical.

AB - Modern methods for Bayesian regression beyond the Gaussian response setting are often computationally impractical or inaccurate in high dimensions. In fact, as discussed in recent literature, bypassing such a trade-off is still an open problem even in routine binary regression models, and there is limited theory on the quality of variational approximations in high-dimensional settings. To address this gap, we study the approximation accuracy of routinely used mean-field variational Bayes solutions in high-dimensional probit regression with Gaussian priors, obtaining novel and practically relevant results on the pathological behaviour of such strategies in uncertainty quantification, point estimation and prediction. Motivated by these results, we further develop a new partially factorized variational approximation for the posterior distribution of the probit coefficients that leverages a representation with global and local variables but, unlike for classical mean-field assumptions, it avoids a fully factorized approximation, and instead assumes a factorization only for the local variables. We prove that the resulting approximation belongs to a tractable class of unified skew-normal densities that crucially incorporates skewness and, unlike for state-of-the-art mean-field solutions, converges to the exact posterior density as p → ∞. To solve the variational optimization problem, we derive a tractable coordinate ascent variational inference algorithm that easily scales to p in the tens of thousands, and provably requires a number of iterations converging to 1 as p → ∞. Such findings are also illustrated in extensive empirical studies where our novel solution is shown to improve the approximation accuracy of mean-field variational Bayes for any n and p, with the magnitude of these gains being remarkable in those high-dimensional p>n settings where state-of-the-art methods are computationally impractical.

KW - Bayesian computation

KW - Data augmentation

KW - Variational Bayes

KW - Truncated normal distribution

KW - Unified skew-normal distribution

KW - High-dimensional probit regression

KW - Bayesian computation

KW - Data augmentation

KW - Variational Bayes

KW - Truncated normal distribution

KW - Unified skew-normal distribution

KW - High-dimensional probit regression

UR - http://hdl.handle.net/10807/257959

U2 - 10.1093/biomet/asac026

DO - 10.1093/biomet/asac026

M3 - Article

SN - 0006-3444

VL - 109

SP - 901

EP - 919

JO - Biometrika

JF - Biometrika

ER -