Validation error is very small but the prediction accuracy rates are all 0?


I am trying to make predictions with a Pyro Bayesian neural network, and while the validation errors from the Stochastic Variational Inference during the training phase are very small (meaning the Bayesian network is very well trained), when I try to make predictions based on this Pyro neural network model, I am getting the accuracy rate (% of correct predictions from the test set) of 0%. I don’t know why this is happening.

My likelihood function is a multinomial likelihood, which I specified in my model statement as below:

# define likelihood function for our Bayesian layer.
class MyModel(PyroModule):
    def __init__(self,  model, name=""):
        super( ).__init__( )
        self.model = model

    def forward(self, input, true_answer = None, y=None):
        if true_answer != None:
            classification_scores = model(input)
            softmax_tensor = nn.Softmax(dim=-1)(classification_scores)
            if true_answer == torch.tensor([0]):
                 true_answer_tensor = torch.tensor([[1.,0.,0.,0.]])

            elif true_answer == torch.tensor([1]):
                 true_answer_tensor = torch.tensor([[0.,1.,0.,0.]])

            elif true_answer == torch.tensor([2]):
                 true_answer_tensor = torch.tensor([[0.,0.,1.,0.]])

            elif true_answer == torch.tensor([3]):
                 true_answer_tensor = torch.tensor([[0.,0.,0.,1.]])
            classification_scores = model(input)
            softmax_tensor = nn.Softmax(dim=-1)(classification_scores)
            true_answer_tensor = None
        # specify the multinomial likelihood for my `y`
        return pyro.sample("y",
                    dist.Multinomial(1, probs = softmax_tensor),
                    obs = true_answer_tensor)

and then, I am trying to make predictions like below:

# turn on the evaluation mode.
model = MyModel(original_model)
pred_obj = Predictive(model, guide=my_guide, num_samples = 100)
prediction = pred_obj.forward(input)
prediction_tensor = prediction["y"].squeeze(1)
prediction_tensor = torch.sum(prediction_tensor, dim=0)
my_prediction = torch.argmax(prediction_tensor).tolist()

I am thinking either my model definition and/or the way I use the Predictive function is wrong.
Is there any error in my code? It might have to do with the fact that I am not explicitly returning my_guide in my training function. Do I need to explicitly return the trained my_guide at the end of my training function, in order to make my Predictive() statement to properly refer to the trained my_guide (as opposed to my_guide before the training)?

Thank you,

Has this type of phenomenon happened with pyro models before (small validation error but low prediction accuracy)? If so, what are the common causes for this?

Thank you,

In my experience, I got this issue when e.g.

  • I forgot to scale the testing data
  • There are NaN in testing data
  • The model has dropout code

Do you use the same code for validation and testing?


Thank you for your reply…I am wondering what you mean by “scaling the testing data”…could you explain more on this? Also, for the last point you made, I am wondering what you exactly mean by “the model has dropout code”. Before the training AND prediction phase, I prevented all the dropouts in my neural network model… Thank you for your help.

By “scaling”, I meant “normalization” or “standardization” terms in data preprocessing (the one in this article). About dropout, yes, I meant neural network dropout layers.