Hello, I am new to probabilistic programming and Pyro.

Is it possible to use a hyperprior to model the prior of some model using Pyro?

thanks

Hello, I am new to probabilistic programming and Pyro.

Is it possible to use a hyperprior to model the prior of some model using Pyro?

thanks

Sure, just sample your model’s hyperparameters from a prior distribution in your model definition in the usual way with `pyro.sample`

, as you would with any other component of your model. There’s no conceptual distinction in Bayesian inference between priors and hyperpriors.

If your model is already written as a `torch.nn.Module`

, you can also use `pyro.nn.PyroModule`

to avoid boilerplate code in your model logic:

```
class MyModel(torch.nn.Module):
def __init__(self, ...):
...
def forward(self, ...):
# actual model logic here
...
class MyPyroModel(MyModel, pyro.nn.PyroModule):
pass
my_module = MyPyroModel(...)
my_module.some_hyperparam = PyroSample(dist.Normal(0, 1))
```

1 Like

Would something like that work?

```
class BNN(PyroModule):
def __init__(self, input_size, hidden_size, output_size):
super(BNN, self).__init__()
self.fc1 = PyroModule[nn.Linear](input_size, hidden_size)
self.fc1.weight = PyroSample(dist.Normal(0., 1.).expand([hidden_size, input_size]).to_event(2))
self.fc1.bias = PyroSample(dist.Normal(0., 10.).expand([hidden_size]).to_event(1))
self.out = PyroModule[nn.Linear](hidden_size, output_size)
self.out.weight = PyroSample(dist.Normal(0., 1.).expand([output_size, hidden_size]).to_event(2))
self.out.bias = PyroSample(dist.Normal(0., 10.).expand([output_size]).to_event(1))
def forward(self, x, y_data=None):
output = self.fc1(x)
output = F.relu(output)
output = self.out(output)
lhat = F.log_softmax(output)
obs = pyro.sample("obs", dist.Categorical(logits=lhat), obs=y_data)
return obs
model = BNN(28*28, 1024, 10)
model.weight1 = PyroSample(dist.Normal(0., 1.))
```

Would something like that work?

Basically, yes - I’m not sure what `model.weight1`

is in this example, but if you’ve written your model as a subclass of `PyroModule`

as in your snippet and already set all the relevant (hyper)parameters to `PyroSample`

objects in your `BNN.__init__`

method, you don’t need to re-set any of those parameters to `PyroSample`

objects again outside of `__init__`

as in your final line.

OK, but i don’t see how only in the `BNN.__init__`

I would be doing a hierarchical prior. The way my code is, it is defining a prior over the nn parameters, weights ans bias. I’d like to define priors over the prior params. Sorry if i misunderstood you. For reference, i got this concept in this paper: http://www.jmlr.org/papers/volume1/tipping01a/tipping01a.pdf

I see, I believe you should be able to use the deferred version of `PyroSample`

, as in e.g. this thread on `PyroModule`

:

```
class BNN(PyroModule):
def __init__(self, input_size, hidden_size, output_size):
super(BNN, self).__init__()
self.fc1 = PyroModule[nn.Linear](input_size, hidden_size)
self.out = PyroModule[nn.Linear](hidden_size, output_size)
self.fc1.weight_scale = PyroSample(dist.InverseGamma(...))
# pass this PyroSample a lambda that takes self.fc1 and generates a Distribution
self.fc1.weight = PyroSample(lambda self: dist.Normal(0., self.weight_scale).expand([hidden_size, input_size]).to_event(2))
... # handle other parameters similarly
def forward(self, x, y_data=None):
output = self.fc1(x)
output = F.relu(output)
output = self.out(output)
lhat = F.log_softmax(output)
obs = pyro.sample("obs", dist.Categorical(logits=lhat), obs=y_data)
return obs
```

1 Like