I’ve just started using Pyro and would like to know how get the uncertainty for a prediction in regression.
Using the tutorial Bayesian Regression - Introduction (Part 1) — Pyro Tutorials 1.8.4 documentation with the final snippet of code as follows how can you get the uncertainty (std/probability) for each prediction?
for i in range(20):
# guide does not require the data
sampled_reg_model = guide(None)
# run the regression model and add prediction to total
y_preds = y_preds + sampled_reg_model(x_data)
# take the average of the predictions
y_preds = y_preds / 20
running the model forward takes one sample from the posterior. since the distribution is parameterized, you can just just look in the param store. something like: pyro.param('name_of_std_param')
The following values are in the param store (Boston Housing data - 13 features). How do you calculate the uncertainty for each prediction based on this?
i think you might be confusing different things. the uncertainty is over parameters not outputs. a bayesian nn is a distribution over nns, so you sample an nn which give a point estimate given data. it seems like you’re looking for something like a GP?
[quote=“jpchen, post:4, topic:425”]
give a point estimate given data. it seems like you’re looking for something like a GP?
[/quote] The uncertainty related to each point prediction is given by the standard deviation (or entropy) of 20 different predictions you got from sampling the weights of you neural network, isn’t it ?
Yes, i agree that GP is an other way to get these uncertainty information.
oh sure, you can calculate the empirical std dev by sampling a bunch of nns and running on the same data point. but then at that point i think there are better models to use to get what you want.
Thanks for the comments and sorry for the late reply. Yeah perhaps a Gaussian Process is a better fit for measuring the uncertainty in each new prediction.
Excuse my ignorance to Bayesian methods but in the bayesian regression tutorial http://pyro.ai/examples/bayesian_regression.html4 which uses one linear layer (linear regression) so it will result in one set of weights / distribution of weights after training.
For a new test input is it possible to multiple the inputs by each weight / distribution to output a prediction with uncertainty?
sorry i don’t understand what you’re asking. what does multiply the inputs by each weight mean? the uncertainty learned in the tutorial is the global uncertainty, not local to each data point.
Least squares regression
In a regular regression problem once the model is trained and to make a prediction on new inputs you would multiply each feature by the corresponding weight and sum the values to output the final prediction for a given sample input.
weights (after training a model): w1=7, w2=18, w3=17
Multiplying the the weights by the the features for each input data sample will output a prediction (without any uncertainty information)
Probabilistic Linear Regression
weights (after training a model - linear model so only one set of weights):
w1_mean=7, w1_std=1
w2_mean=18, w2_std=4,
w3_mean=17, w3_std=1
Now that you have the mean value of each weight and standard deviation can you use this information to calculate the uncertainty of a prediction on the test data? i.e w2 has a large standard deviation so any input data with high values for this feature would result in high uncertainty in the prediction.
The second sample test input above has a value of 15 for feature_2 so maybe this would result in high uncertainty in the test prediction.
The first sample has a value of 0 for feature_2 so maybe this would result in a more confident prediction as the other two features effecting the prediction have a low standard deviation?
This is what I mean by using the mean and std of each weight to calculate the uncertainty specific to each new test input.
