Is there an established way to share parameter values between the model and guide when using SVI in numpyro? (This is often called “weight tying” in the context of autoencoders; see e.g. Building an Autoencoder with Tied Weights in Keras | by Laurence Mayrand-Provencher | Medium)
I can imagine a roundabout method which involves sampling a parametrized delta distribution in the guide and using a matching sample site to access the value in the model, but that feels like a bit of a hack…
Thanks!
I believe you can access the same parameter from both the model and the guide using numpyro.param("name").
Oh wow, that was simpler than I thought! I had the impression that only sample site values from the guide were replayed onto the model, but it seems that params somehow make it across as well. (Not sure I understand where that actually happens in the SVI source but I’m glad to see it “just works”…)
Thanks for the help =)
PS: For anyone trying to use this: you do still need to provide an initial value or initializer function for the param statement in the model (otherwise it defaults to None during tracing and breaks any downstream code that uses it), but it seems that the param statement used during the actual inference procedure is the one in the guide.
otherwise it defaults to None during tracing and breaks any downstream code that uses it
Could you create a FR for this? Or make a small pull request to allow replacing params in replay handler. 
Currently, in SVI, we haven’t replayed parameters yet. Thanks!
Can definitely create a FR, but I’m not quite sure what the desired behavior should be here. If we want to be able to run the model without a guide, it’ll need an initializer as it currently does - unless you want to think of more explicit ways to handle shared params?
Yes, I saw that parameters aren’t replayed, but somehow the model ends up sharing the value from the guide regardless if it has the same name. I’m guessing it works its way over via the messenger stack, but haven’t dug into the source enough to see where that happens…
The issue just happens in the initialization step. During svi updates, the logic is fine, I think.
If we want to be able to run the model without a guide, it’ll need an initializer as it currently does
You are right. It’s needed if we want to draw samples with init parameters. But I guess in practice, we always want to draw samples with optimized parameters. The desired behavior in SVI is to avoid initializing model’s params when users already specify them in the guide.
The desired behavior in SVI is to avoid initializing model’s params when users already specify them in the guide.
Got it. I’ll take a look later this afternoon and try to put something together.