 # Hyperprior

Hello, I am new to probabilistic programming and Pyro.

Is it possible to use a hyperprior to model the prior of some model using Pyro?

thanks

Sure, just sample your model’s hyperparameters from a prior distribution in your model definition in the usual way with `pyro.sample`, as you would with any other component of your model. There’s no conceptual distinction in Bayesian inference between priors and hyperpriors.

If your model is already written as a `torch.nn.Module`, you can also use `pyro.nn.PyroModule` to avoid boilerplate code in your model logic:

``````class MyModel(torch.nn.Module):
def __init__(self, ...):
...

def forward(self, ...):
# actual model logic here
...

class MyPyroModel(MyModel, pyro.nn.PyroModule):
pass

my_module = MyPyroModel(...)
my_module.some_hyperparam = PyroSample(dist.Normal(0, 1))
``````
1 Like

Would something like that work?

``````class BNN(PyroModule):

def __init__(self, input_size, hidden_size, output_size):
super(BNN, self).__init__()
self.fc1 = PyroModule[nn.Linear](input_size, hidden_size)
self.fc1.weight = PyroSample(dist.Normal(0., 1.).expand([hidden_size, input_size]).to_event(2))
self.fc1.bias = PyroSample(dist.Normal(0., 10.).expand([hidden_size]).to_event(1))
self.out = PyroModule[nn.Linear](hidden_size, output_size)
self.out.weight = PyroSample(dist.Normal(0., 1.).expand([output_size, hidden_size]).to_event(2))
self.out.bias = PyroSample(dist.Normal(0., 10.).expand([output_size]).to_event(1))

def forward(self, x, y_data=None):
output = self.fc1(x)
output = F.relu(output)
output = self.out(output)
lhat = F.log_softmax(output)
obs = pyro.sample("obs", dist.Categorical(logits=lhat), obs=y_data)
return obs

model = BNN(28*28, 1024, 10)
model.weight1 = PyroSample(dist.Normal(0., 1.))``````

Would something like that work?

Basically, yes - I’m not sure what `model.weight1` is in this example, but if you’ve written your model as a subclass of `PyroModule` as in your snippet and already set all the relevant (hyper)parameters to `PyroSample` objects in your `BNN.__init__` method, you don’t need to re-set any of those parameters to `PyroSample` objects again outside of `__init__` as in your final line.

OK, but i don’t see how only in the `BNN.__init__` I would be doing a hierarchical prior. The way my code is, it is defining a prior over the nn parameters, weights ans bias. I’d like to define priors over the prior params. Sorry if i misunderstood you. For reference, i got this concept in this paper: http://www.jmlr.org/papers/volume1/tipping01a/tipping01a.pdf

I see, I believe you should be able to use the deferred version of `PyroSample`, as in e.g. this thread on `PyroModule`:

``````class BNN(PyroModule):

def __init__(self, input_size, hidden_size, output_size):
super(BNN, self).__init__()
self.fc1 = PyroModule[nn.Linear](input_size, hidden_size)
self.out = PyroModule[nn.Linear](hidden_size, output_size)
self.fc1.weight_scale = PyroSample(dist.InverseGamma(...))
# pass this PyroSample a lambda that takes self.fc1 and generates a Distribution
self.fc1.weight = PyroSample(lambda self: dist.Normal(0., self.weight_scale).expand([hidden_size, input_size]).to_event(2))
...  # handle other parameters similarly

def forward(self, x, y_data=None):
output = self.fc1(x)
output = F.relu(output)
output = self.out(output)
lhat = F.log_softmax(output)
obs = pyro.sample("obs", dist.Categorical(logits=lhat), obs=y_data)
return obs
``````
1 Like