Hyperprior

Misc.
1

Hello, I am new to probabilistic programming and Pyro.

Is it possible to use a hyperprior to model the prior of some model using Pyro?

thanks

2

Sure, just sample your model’s hyperparameters from a prior distribution in your model definition in the usual way with pyro.sample, as you would with any other component of your model. There’s no conceptual distinction in Bayesian inference between priors and hyperpriors.

If your model is already written as a torch.nn.Module, you can also use pyro.nn.PyroModule to avoid boilerplate code in your model logic:

class MyModel(torch.nn.Module):
    def __init__(self, ...):
        ...

    def forward(self, ...):
        # actual model logic here
        ...

class MyPyroModel(MyModel, pyro.nn.PyroModule):
    pass

my_module = MyPyroModel(...)
my_module.some_hyperparam = PyroSample(dist.Normal(0, 1))
1 Like
4

Would something like that work?

class BNN(PyroModule):

def __init__(self, input_size, hidden_size, output_size):
    super(BNN, self).__init__()
    self.fc1 = PyroModule[nn.Linear](input_size, hidden_size)
    self.fc1.weight = PyroSample(dist.Normal(0., 1.).expand([hidden_size, input_size]).to_event(2))
    self.fc1.bias = PyroSample(dist.Normal(0., 10.).expand([hidden_size]).to_event(1))
    self.out = PyroModule[nn.Linear](hidden_size, output_size)
    self.out.weight = PyroSample(dist.Normal(0., 1.).expand([output_size, hidden_size]).to_event(2))
    self.out.bias = PyroSample(dist.Normal(0., 10.).expand([output_size]).to_event(1))
    
def forward(self, x, y_data=None):
    output = self.fc1(x)
    output = F.relu(output)
    output = self.out(output)
    lhat = F.log_softmax(output)
    obs = pyro.sample("obs", dist.Categorical(logits=lhat), obs=y_data)
    return obs

model = BNN(28*28, 1024, 10)
model.weight1 = PyroSample(dist.Normal(0., 1.))
5

Would something like that work?

Basically, yes - I’m not sure what model.weight1 is in this example, but if you’ve written your model as a subclass of PyroModule as in your snippet and already set all the relevant (hyper)parameters to PyroSample objects in your BNN.__init__ method, you don’t need to re-set any of those parameters to PyroSample objects again outside of __init__ as in your final line.

6

OK, but i don’t see how only in the BNN.__init__ I would be doing a hierarchical prior. The way my code is, it is defining a prior over the nn parameters, weights ans bias. I’d like to define priors over the prior params. Sorry if i misunderstood you. For reference, i got this concept in this paper: http://www.jmlr.org/papers/volume1/tipping01a/tipping01a.pdf

7

I see, I believe you should be able to use the deferred version of PyroSample, as in e.g. this thread on PyroModule:

class BNN(PyroModule):

def __init__(self, input_size, hidden_size, output_size):
    super(BNN, self).__init__()
    self.fc1 = PyroModule[nn.Linear](input_size, hidden_size)
    self.out = PyroModule[nn.Linear](hidden_size, output_size)
    self.fc1.weight_scale = PyroSample(dist.InverseGamma(...))
    # pass this PyroSample a lambda that takes self.fc1 and generates a Distribution
    self.fc1.weight = PyroSample(lambda self: dist.Normal(0., self.weight_scale).expand([hidden_size, input_size]).to_event(2))
    ...  # handle other parameters similarly
    
def forward(self, x, y_data=None):
    output = self.fc1(x)
    output = F.relu(output)
    output = self.out(output)
    lhat = F.log_softmax(output)
    obs = pyro.sample("obs", dist.Categorical(logits=lhat), obs=y_data)
    return obs
1 Like