Thank you @ordabayev for the explanation! I also observed that using pyro.param instead of a prior worked better in my model.
I have a question about naming the parameters. Should the name of the parameters in the model and guide match, like they do for pyro.sample, or should it be avoided? For example, the names of the parameters match in guide and model in Tensor shapes in Pyro — Pyro Tutorials 1.8.4 documentation (see fun method).
Edit: I have just seen that you addressed this an hour ago at Pyro curve fitting parameter std estimate - #3 by ordabayev 
Although, I am still a bit confused why the names match in this particular tutorial