If I understand correctly what is going on in auto guides such as AutoNormal, only the sample sites of the numpyro model get corresponding mean and width parameters associated to them. I am currently working with a model based on both sample and param sites, and I would like to model the posterior distribution of all these sites.
Hence, before trying to adapt the numpyro source code for what I am looking for, I wanted to make sure: is there some mathematical reason why auto guides do not create posterior parameters for param sites?
it doesn’t really make sense to ask for a posterior distribution for a param site—it doesn’t have a prior associated with it. param basically means “this is a point estimate quantity with no associated uncertainty”
I agree with you in the general case. However I have to use param sites as I am imposing some regularization on certain parameters in a way that cannot be expressed as a simple prior distribution (specifically, sparsity via l1-norm in the wavelet domain). But so far I don’t see why this should prevent me from sampling from these sites and model their posterior distribution within SVI.
Thanks for your answer though! The thing is that my use of these param sites is not really the normal use, which I agree with what you say should mean “no uncertainty”. I use these sites as a way to impose my “custom prior” by hand, using the numpyro’s factor statement.
Hmm that’s interesting. I completely missed this handler, but it may indeed help me. After optimizing the param sites, I could lift them and associate some analytical prior distribution to then sample using SVI. Thanks a lot!