Out of sample predictions in Pyro

Misc.
1

I have a simple linear regression model in Pyro with the following code:

data
comments commits time
Alice 7500 25 4.5
Bob 10100 32 6.0
Cole 18600 49 7.0
Danielle 25200 66 12.0
Erika 27500 96 18.0 
slack_comments = torch.tensor(data.comments.values)
github_commits = torch.tensor(data.commits.values)
time = torch.tensor(data.time.values)

dims={
    "slack_comments": ["developer"],
    "github_commits": ["developer"],
    "time": ["developer"],
}

data_dict = {
    "developer": N,
    "time_since_joined": time
}

def model(developer, time_since_joined):
    b_sigma = abs(pyro.sample('b_sigma', dist.Normal(0, 300)))
    c_sigma = abs(pyro.sample('c_sigma', dist.Normal(0, 6)))
    b0 = pyro.sample("b0", dist.Normal(0, 200))
    b1 = pyro.sample("b1", dist.Normal(0, 200))
    c0 = pyro.sample("c0", dist.Normal(0, 10))
    c1 = pyro.sample("c1", dist.Normal(0, 10))

    with pyro.plate('developer', developer):
        slack = pyro.sample("slack_comments", dist.Normal(b0 + b1 * time_since_joined, b_sigma), obs=slack_comments)
        github = pyro.sample("github_commits", dist.Normal(c0 + c1 * time_since_joined, c_sigma), obs=github_commits)
        return slack, github


nuts_kernel = NUTS(model, jit_compile=True, ignore_jit_warnings=True)
mcmc = MCMC(nuts_kernel, num_samples=400, warmup_steps=400,
            num_chains=4, disable_progbar=True)
mcmc.run(**data_dict)
posterior_samples = mcmc.get_samples()
posterior_predictive = Predictive(model, posterior_samples).forward(**data_dict)
prior = Predictive(model, num_samples=150).forward(**data_dict)

I have the following questions:

  1. The posterior_predictive samples just seem to be the observed data called over again and again and not actually predictive samples. Why is this happening?

  2. The posterior_predictive samples have the shape of the observed data (there are 5 values passed), which makes sense, however when I want predictions on new time data with different shape (2 values passed) I still get the old shape of 5 values.

posterior_predictive

{'slack_comments': tensor([[ 7500, 10100, 18600, 25200, 27500],
         [ 7500, 10100, 18600, 25200, 27500],
         [ 7500, 10100, 18600, 25200, 27500],
         ...,
         [ 7500, 10100, 18600, 25200, 27500],
         [ 7500, 10100, 18600, 25200, 27500],
         [ 7500, 10100, 18600, 25200, 27500]]),
 'github_commits': tensor([[25, 32, 49, 66, 96],
         [25, 32, 49, 66, 96],
         [25, 32, 49, 66, 96],
         ...,
         [25, 32, 49, 66, 96],
         [25, 32, 49, 66, 96],
         [25, 32, 49, 66, 96]])}

predictions_dict = {
    "developer": 2,
    "time_since_joined": candidate_devs_time
}
predictions = Predictive(model, posterior_samples).forward(**predictions_dict)
predictions

{'slack_comments': tensor([[ 7500, 10100, 18600, 25200, 27500],
         [ 7500, 10100, 18600, 25200, 27500],
         [ 7500, 10100, 18600, 25200, 27500],
         ...,
         [ 7500, 10100, 18600, 25200, 27500],
         [ 7500, 10100, 18600, 25200, 27500],
         [ 7500, 10100, 18600, 25200, 27500]]),
 'github_commits': tensor([[25, 32, 49, 66, 96],
         [25, 32, 49, 66, 96],
         [25, 32, 49, 66, 96],
         ...,
         [25, 32, 49, 66, 96],
         [25, 32, 49, 66, 96],
         [25, 32, 49, 66, 96]])}

Am I doing something wrong? I had also used a guide but the results were same. I want to use the same trace/posterior samples to generate predictions (slack_comments and github_commits) data by passing new time data.

1 Like
2

@nitishp25 To get posterior_predictive, you should set the input developer and time_since_joined to None. Otherwise, the statement

pyro.sample("slack_comments", dist.Normal(b0 + b1 * time_since_joined, b_sigma), obs=slack_comments)

will return slack_comments for you. In particular, Predictive(model, posterior_samples)(None, None) will give you the expected result.

The Predictive utility does not know that you want to convert those observation statements in the model to sample statements. So we need to set obs=None at those statements (i.e. tell Pyro there is no observation data at those nodes). I think that that way, Predictive is flexible: all we need to do is to provide correct inputs. :wink:

2 Likes
3

Hi @fehiepsi, I ran:

posterior_predictive = Predictive(model, posterior_samples).forward(None, None)

but it gives the following error:

TypeError: unsupported operand type(s) for *: 'Tensor' and 'NoneType'

because time_since_joined is needed in:

with pyro.plate('developer', developer):
    slack = pyro.sample("slack_comments", dist.Normal(b0 + b1 * time_since_joined, b_sigma), obs=slack_comments)
1 Like
4

Oops, my mistake. I meant to make slack_comments and github_commits to None, kind of

def model(developer, time_since_joined, slack_comments=None, github_commits=None):
    ...

mcmc.run(**data_dict, slack_comments=slack_comments, github_commits=github_commits)
posterior_predictive = Predictive(...)(**data_dict)
# in the definition of `model`, slack_comments and github_commits will be None by default.
3 Likes
5

Thanks a lot for the help @fehiepsi! :smiley:

One more thing, is there any way to access constant data of the model outside the model?
For example, I have defined time inside the model now:

def model(developer, time_since_joined, slack_comments=None, github_commits=None):
    time = time_since_joined
    ...

Can I access it outside via the model or the sampler? Also, is it the right way to the declare constant data?

6

Maybe you need pyro.deterministics?

7

This is actually sampling this value and adding it to prior and posterior_predictive.

Actually, my purpose is to declare this data inside the model without any sampling to find a way to store it in Arviz inference data. There is a PyMC3 implementation of constant_data in InferenceData which stores the pm.data and pm.deterministic values. Can something similar be done in Pyro by avoiding the sampling and accessing this constant data (time in this case)?

8

I think the easiest way is to use arviz.from_dict, which allows you setting constant_data. Pyro does not have something similar to pymc3.Data.

1 Like
9

Oh thanks! I’m thinking of adding a constant_data argument to arviz.from_pyro only so that the user can pass the constant data just like the trace and posterior_predictive and then convert it to InferenceData. What do you think?

10

To be honest, I don’t know what constant_data does so I can’t tell if it is helpful. But if you want that feature, it is nice to have support for it. :slight_smile:

1 Like
11

Thanks for all the help! :smiley: