Training and Evaluation performance issues

Misc.
1

I’m noticing some issues with performance when alternating between training and evaluation. Here is what I’m doing (for a VAE like model):

  • Train model for a few epochs
  • Test model using .eval() under with torch.nograd()
  • Train model some more (resetting .train())

Two observations:

  1. I’ve noticed the reconstructions can degrade
  2. The training loss increases (worse) initially than recovers after a few iterations

I’m using the latest stable versions of PyTorch and Pyro.

Are there any special commands to implement this right? Is it an issue that I’m executing this in a Jupyter Notebook?

Thanks for any help.

2

I have same problem with SS-VAE.
Can someone advise me something to fix that.
Even only torch.no_grad() cause me same bug.