QR decomposition "trick" for (generalized) linear models

Hi all - I was curious whether the “QR trick” (see e.g. The QR Decomposition For Regression Models) could be used to speed up/improve fitting of regression models in pyro when the covariates are correlated. I’m doing this using a custom guide which I think is correct. However, I get much worse convergence and final error (I know the true posterior in the linear model case). If anyone is interested my code is here:
Google Colab
Maybe this approach just isn’t helpful for SVI, or I’m doing something wrong in the setup (I don’t think pyro is doing anything wrong).

i think you’re just seeing suboptimal optimization driven by initialization choices.

e.g. if i instead use

Q = Q * math.sqrt(N)
R = R / math.sqrt(N)

i get

Error in E[beta] 0.041163329035043716
Error in diag cov[beta] 0.008744467981159687
Error in cov[beta] 0.0009263675310648978
1 Like