I was looking carefully at the implementation of Projected normal and its reparameteizer and I noticed there is a mismatch between degrees of freedom of the distribution (N) and the reparametrized one (N*2), that I think might result in silently fitting a different model than the one intended by the user.
For example, let consider a 1d projected normal with 2d concentration parameters. The two parameters are expressed as xy coordinates of the center of the gaussian to be projected but they rapresent location and scale of the projected distribution.
This situation should be reparametrized into a 2d Normal with scale fixed to 1 and location as a 2d latent variable. However with the current implementation an Auto*Normal will create 4 parameters instead of two. Allowing a variational distribution which is overparametrized and effectivelly higher dimensional/with more degrees of freedom than the prior. The right behaviour would be to fix scale to 1.
The proposed fix would be to have Auto*Normal switched automatically somehow to an AutoDelta wich would fit only the loc parameters of the reparametrized distribution.