PyroSample vs pyro.sample and other mild confusion

Misc.
1

Hi all. I’ve had fun messing around with some simple models but am missing some understanding on a few points that I’m struggling to find addressed in the docs.

  1. What is the difference between PyroSample and pyro.sample?
  2. If I’m using a PyroModule to represent my model (as in the Bayesian regression example), and I only have global variables, is there any distinction between putting PyroSample calls in init() vs in forward() ?
  3. Are there best practice recommendations about structuring code: e.g. should the model usually be a PyroModule? Should the guide be a member function of that class? This seems to vary across the different examples I’ve looked at.
  4. Are there any full examples using AutoGuideList ?
2

Hi @daknowles,

  1. pyro.sample() is the more standard pattern in both model functions and model PyroModules. PyroSample is intended for retrofitting PyTorch nn.Modules to be Bayesian, in the context of Bayesian neural nets. The idea is that these modules usually access some learnable nn.Parameter like self.bias. By replacing self.bias with a PyroSample and converting the nn.Module to a PyroModule (e.g. via to_pyro_module_()), you can achieve sampling at each call of the nn.Module.forward() method.
  2. Even if your model is a PyroModule, I’d recommend using pyro.sample() calls in the .forward() method. Again, PyroSample is a little cumbersome, and is really most useful for retrofitting.
  3. Re: best practices, opinions differ. I find it most flexible to create a separate model function (or PyroModule with PyroParams) and then a variety of guides, each typically a PyroModule or a subclass of an AutoGuide. This pattern is most useful if you want to try out different guides with a single model. If you end up settling a single guide, it can be useful to store your model and guide in a single nn.Module, but be careful to make that outer container an nn.Module and not a PyroModule (otherwise PyroParam and PyroSample statements may have name conflicts).
  4. For examples of AutoGuideList, I’d recommend searching through tests/infer/test_autoguide.py. Here’s the guide for the model I’m currently working on.

One piece of advice is to start with autoguides, e.g. start with an AutoNormal or AutoLowRankMultivariateNormal. Then maybe specialize a couple sites using an AutoGuideList: write a custom guide for those sites and use an AutoNormal for the remaining sites. Another piece of advice is to use reparametrizers, e.g. AutoReparam. We’re working on updated tutorials :slightly_smiling_face:

1 Like
3

Thanks, very helpful. I think part of the big picture I was missing was that (if I’m understanding currently) you could reasonably use pyro without ever touching any of the PyroModule/PyroSample machinery (in contrast to pytorch DL where I usually have everything in a Module). No neural networks are being harmed in the making of my current pyro model so I think I can do without PyroModule for now. It might be helpful to make that a little clearer in the docs somewhere.

The tutorials are good just clearly a little more targeted towards the ML researcher than the typical applied practitioner (in contrast to the Stan tutorials for example).

2 Likes