I tried to follow Bayesian Regression tutorial architecture to implement a more complex LSTM model.
In the model, I use two LSTM layers as encoder, with hidden units 128 and 64 separately. And I use pyro.random_module to lift the weights to stochastic variables. I use independent(1) in the code, like
priors_dist[layer_name] = pyro.distributions.Normal(weights_loc, weights_scale).independent(1)
.
However running the code I face RuntimeError: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 0
.
I found the error occurred at class Trace_ELBO
--> def _compute_log_r(model_trace, guide_trace)
–>log_r.add((stacks[name], log_r_term.detach()))
.
The reason is that at first layer the log_r_term size is (512, ), but second layer the log_r_term size change to (256, ), and in the log_r.add
function it will face shape mismatch error.
It seems that I have to use independent(2)
to eliminate the log_r_term shape mismatch. Is there some better ways to solve the problem?