Hi,
I’m pretty new to the whole idea of SVI. So I’m just wondering what happens if I define a neural network with dropout in it, and then put priors on the weights and use SVI to approximate the posterior. In the prediction process, is dropout still on? Because in normal neural nets, the network is set to .eval() when making predictions. But I don’t see this in the whole SVI setting.
A little late to reply here, but I was in search of the same question, so hopefully this can help someone down the road.
See the DMM example: Deep Markov Model — Pyro Tutorials 1.8.4 documentation The RNN guide for the hidden states uses dropout. You’ll still need to set .eval() (and back to .train() to continue training) just like in normal neural nets.
I’m not a Pyro admin, but here’s what I know. The nature of how the Bayesian neural network is created removes the need for dropout because it is self-regularizing.The purpose of dropout is to prevent a neural network from over fitting, but over fitting cannot happen for a bayesian network since the model weights are pulled from a shared learned probability distribution. This learned probability distribution is calculated by minimizing the KL divergence between it and the “true” distribution through tweaking the hyperparameters of the learned probability distribution. Read the paper Pyro is based on to figure out exactly why.
That said, SVI generates n number of neural networks using your architecture through the guide function, where n in the number of times you sample for model weights. I’m not sure if it includes the dropout layer when you sample.