Validation error is very small but the prediction accuracy rates are all 0?

Misc.
1

Hello,

I am trying to make predictions with a Pyro Bayesian neural network, and while the validation errors from the Stochastic Variational Inference during the training phase are very small (meaning the Bayesian network is very well trained), when I try to make predictions based on this Pyro neural network model, I am getting the accuracy rate (% of correct predictions from the test set) of 0%. I don’t know why this is happening.

My likelihood function is a multinomial likelihood, which I specified in my model statement as below:

# define likelihood function for our Bayesian layer.
class MyModel(PyroModule):
    
    def __init__(self,  model, name=""):
        super( ).__init__( )
        self.model = model

    def forward(self, input, true_answer = None, y=None):
    
        if true_answer != None:
            classification_scores = model(input)
            
            softmax_tensor = nn.Softmax(dim=-1)(classification_scores)
            
            if true_answer == torch.tensor([0]):
                 true_answer_tensor = torch.tensor([[1.,0.,0.,0.]])

            elif true_answer == torch.tensor([1]):
                 true_answer_tensor = torch.tensor([[0.,1.,0.,0.]])

            elif true_answer == torch.tensor([2]):
                 true_answer_tensor = torch.tensor([[0.,0.,1.,0.]])

            elif true_answer == torch.tensor([3]):
                 true_answer_tensor = torch.tensor([[0.,0.,0.,1.]])
            
        else:
            classification_scores = model(input)
            
            softmax_tensor = nn.Softmax(dim=-1)(classification_scores)
            true_answer_tensor = None
  
        # specify the multinomial likelihood for my `y`
        return pyro.sample("y",
                    dist.Multinomial(1, probs = softmax_tensor),
                    obs = true_answer_tensor)

and then, I am trying to make predictions like below:

# turn on the evaluation mode.
model = MyModel(original_model)
model.eval()
pred_obj = Predictive(model, guide=my_guide, num_samples = 100)
                                  
prediction = pred_obj.forward(input)
                                                 
prediction_tensor = prediction["y"].squeeze(1)
prediction_tensor = torch.sum(prediction_tensor, dim=0)
my_prediction = torch.argmax(prediction_tensor).tolist()

I am thinking either my model definition and/or the way I use the Predictive function is wrong.
Is there any error in my code? It might have to do with the fact that I am not explicitly returning my_guide in my training function. Do I need to explicitly return the trained my_guide at the end of my training function, in order to make my Predictive() statement to properly refer to the trained my_guide (as opposed to my_guide before the training)?

Thank you,

2

Has this type of phenomenon happened with pyro models before (small validation error but low prediction accuracy)? If so, what are the common causes for this?

Thank you,

3

In my experience, I got this issue when e.g.

  • I forgot to scale the testing data
  • There are NaN in testing data
  • The model has dropout code

Do you use the same code for validation and testing?

4

Hello,

Thank you for your reply…I am wondering what you mean by “scaling the testing data”…could you explain more on this? Also, for the last point you made, I am wondering what you exactly mean by “the model has dropout code”. Before the training AND prediction phase, I prevented all the dropouts in my neural network model… Thank you for your help.

5

By “scaling”, I meant “normalization” or “standardization” terms in data preprocessing (the one in this article). About dropout, yes, I meant neural network dropout layers.