nn.Parameter or PyroParam?

1

Hi, I’m currently working on a VAE for my master thesis and I have some questions I couldn’t find answers to.

I have a GRU in my guide and I’d like to register its initial state (h_0_guide) as a learnable parameter.

  1. Is there a standard way of initializing GRUs or RNNs in general in a guide? The only example I found initialized them with zeros.
  2. From what I understand the PyroParam is a wrapper around nn.Parameter allowing for adding constraints, is the way I’m using nn.Parameter correct if I don’t need any constraints for h_0 or should I use PyroParam instead? Are there any args or kwargs I should be taking into account?
    Any input is more than welcome!

My pseudocode:

class VAE(nn.Module):
def init():
self.h_o_guide = nn.Parameter(torch.randn(gru_hidden_dim_Guide), requires_grad=True)
def guide(…):
h_0_gru = self.h_0_guide.expand(self.gruGUIDE.num_layers*2, sequences.size(0), self.gruGUIDE.hidden_size).contiguous()
…
def model():
…

2
  1. You can pass an initialization tensor or callback (e.b. lambda: torch.randn(100)) to either pyro.param or PyroParam. Often when I want really fancy initialization (via another machine learning algorithm), I’ll create those initial values in my module’s .__init__() method and then use those values for initialization as in
    def __init__(self, ...):
    initial_value = ...expensive computation...
    self.my_param = PyroParam(initial_value)
  2. nn.Parameter is fine if you don’t need constraints.
3

Thank you so much, that was very helpful!