- What tutorial are you running?
- What version of Pyro are you using?
- Please link or paste relevant code, and steps to reproduce.
I have been testing out a model that is somewhat similar to the GMM that is found in the tutorial https://pyro.ai/examples/gmm.html. Since I am using larger datasets, it is necessary to allow more iterations of svi.step() to run. However, I noticed that I started running into memory problems - it appeared that with many iterations of svi.step(), memory use gradually increases.
I tested this out with the exact code from the GMM tutorial to make sure it was not an issue with my model. The only 2 differences with the tutorial are that (A) the dataset was increased to 20,000 points instead of 5, and (B) the number of iterations in svi.step() are put to 100,000. Memory use increased as early as the 1000th point, and seemed to increase in jumps of ~0.01GB. I don’t know if the memory allocation stabilised (can’t test that on my machine - but it does crash it at 64GB which is my limit).
I tried 2 versions of Pyro - 1.8.1, and 1.8.0. The both result in the same issue.
I am wondering if this is an expected result, and if it is, what causes it? If it is unavoidable, what might be the best way to circumvent these memory problems?
Thank you very much, and any help is greatly appreciated.
P.S. Pyro is super cool.