Usage of pyro.param in model

The tutorial materials (e.g., SVI Part I: An Introduction to Stochastic Variational Inference in Pyro — Pyro Tutorials 1.8.1 documentation) show that pyro.param is a statement that should be included in the guide as it registers the variational parameters. But I saw in some examples that pyro.param is used in the model as well (e.g, scANVI: Deep Generative Modeling for Single Cell Data with Pyro — Pyro Tutorials 1.8.1 documentation). However, it is not clear to me what the use case of this statement is in the model as I could not see an explanation anywhere.

Could someone please explain when pyro.param should be used in the model?

Hi @dilara . When you use pyro.param in the model you can consider it as the maximum likelihood* estimation method and you get a point estimate for the given parameter. These are theta in the SVI Part I tutorial.

As far as I know pyro.param and pyro.sample statements can be mixed in the model with the difference that at inference you obtain a point estimate for the former (maximum likelihood* treatment) and posterior distribution for the latter (Bayesian treatment). For example, there is a method called Empirical Bayes where the parameters at the highest level of the model hierarchy are set to their most likely value (maximum likelihood treatment).

I personally use pyro.param for some parameters when I am developing a complex model (which is easier since they don’t require a prior distribution) and then later change them to pyro.sample if I want a full Bayesian treatment for my model.

Pyro also has an effect handler called lift that can be used to “lift” pyro.param statements to pyro.sample statements given a prior distribution for that parameter.

*Or maximum log evidence to be more precise when there are random variables in the model.

Hi @ordabayev, thank you so much for the explanation! I was not aware of the presence of lift.

I have a question about your comment here. Do you set the prior for pyro.sample based on the estimate you get from pyro.param?

Sometimes. As an example, there is a gain parameter in my model. It’s value can change from experiment to experiment (but always positive). Based on its pyro.param values from the fits in the past (and also other independent direct experimental measurements of gain) I know that the value of gain typically lies in the range between 5 and 50 so I set its prior as HalfNormal(50.0) for any future fits to roughly capture that range.

Thank you @ordabayev for the explanation! I also observed that using pyro.param instead of a prior worked better in my model.

I have a question about naming the parameters. Should the name of the parameters in the model and guide match, like they do for pyro.sample, or should it be avoided? For example, the names of the parameters match in guide and model in Tensor shapes in Pyro — Pyro Tutorials 1.8.1 documentation (see fun method).

Edit: I have just seen that you addressed this an hour ago at Pyro curve fitting parameter std estimate - #3 by ordabayev :slight_smile: Although, I am still a bit confused why the names match in this particular tutorial

In the tutorial it says Let’s look at a contrived model to see how these work in practice about writing a parallelizable code, not necessarily a proper model/guide pair from a Bayesian inference perspective (just my guess since I didn’t write that tutorial).

Having matching pyro.param s in the model and guide won’t produce an error and the code will run but it is about whether it makes sense from the math/inference point of view.

Edit: There are probably cases when it is ok and makes sense to have matching pyro.param s in model and the guide, just can’t think of any at the moment.

1 Like