Getting parameter values, model samples and likelihood simultaneously

1

Hi, I’m creating a simple linear regression example.

import pyro

def model(x, y=None):
    theta_0 = pyro.param("theta_0", torch.randn(1))
    theta_1 = pyro.param("theta_1", torch.randn(1))
    with pyro.plate("data", len(x)):
        return pyro.sample(
            "obs", pyro.distributions.Normal(x * theta_1 + theta_0, 1.0), obs=y
        )

For different runs, I want to draw samples from the model along with the theta_0 and theta_1 that generated the data.

For a given x, I can generate data as:

import torch
x = torch.linspace(-5.0, 5.0, 100)
y = x * 4 + 5 + torch.randn(100)
y_model= model(x)

and get the corresponding parameters as:

pyro.param("theta_0")
pyro.param("theta_1")

  1. I was wondering if there’s a simpler or better way to do this – get the thetas and the sampled data. My motivation is to show different samples of theta_0 and theta_1 and how they draw different lines.

  2. Secondly, how can I use my model to evaluate the log-likelihood of certain given observations given some parameter inputs theta_0 and theta_1?

1 Like
2

I looked at another post on the forum: Getting the meaning behind pyro.sample with obs - #2 by fehiepsi

Now, I can obtain the loglikelihood of the model.

import pyro

def model2(x, y=None, theta_0_val=None, theta_1_val=None):
    pyro.clear_param_store()
    if theta_0_val is not None:
        theta_0 = pyro.param("theta_0", torch.tensor(theta_0_val))
    else:
        theta_0 = pyro.param("theta_0", torch.randn(1))
    if theta_1_val is not None:
        theta_1 = pyro.param("theta_1", torch.tensor(theta_1_val))
    else:
        theta_1 = pyro.param("theta_1", torch.randn(1))
    with pyro.plate("data", len(x)):
        return pyro.sample(
            "obs", pyro.distributions.Normal(x * theta_1 + theta_0, 1.0), obs=y
        )


def ll(x, y, theta_0, theta_1):
    trace = pyro.poutine.trace(model2).get_trace(x, y, theta_0, theta_1)
    return trace.log_prob_sum()

I implemented the same function using PyTorch distributions.

def ll_torch(x, y, theta_0, theta_1):
    d = torch.distributions.Normal(loc = theta_0 + theta_1*x, scale=1.)
    return d.log_prob(y).sum()

And confirmed that I got the same answer from both implementations.

ll_torch(x, y, 2., 2.)

gives output

tensor(-2161.8367)

ll(x, y, 2., 2.)

gives output:

tensor(-2161.8367, grad_fn=<AddBackward0>)

Questions: I have the following questions now:

  1. Is there a better way to compute the loglikelihood over what I have done?
  2. Let us say I want to obtain MLE from scratch in Pyro (without setting a guide and then optimising ELBO). Conceptually, this is trivial and requires to use the optimise the log likelihood. In regular PyTorch, this would be as follows:
t0 = torch.tensor(0., requires_grad = True)
t1 = torch.tensor(0., requires_grad = True)


optim  = torch.optim.Adam([t0, t1], lr = 0.1)

for i in range(100):
    loss = -ll_torch(x, y, t0, t1)
    loss.backward()
    optim.step()
    optim.zero_grad()

For Pyro, do I need to clear the param store each time I update the parameters? Is there a way to update the values in the param store?

3
  1. Is there a better way to compute the loglikelihood over what I have done?
  2. Let us say I want to obtain MLE from scratch in Pyro (without setting a guide and then optimising ELBO)

Have you taken a look at the MLE and MAP estimation tutorial and the Custom SVI objectives tutorial? These should address your first and second questions respectively.