Hi I am try to learn the Bayesian Neural network, find several simple examples using pyro to build it. But most of the examples seems have the problems. I have search the solutions, seems no direct solution for the problems.
Example 1, 2 has the same problem, example 3 said some wrong in Hidden layer. Only example 4 can run smoothly, but I have some doubts.
Example 1,2 has the same problem.
Same problem arise:
ValueError Traceback (most recent call last)
<ipython-input-25-70bcaa984a3f> in <module>()
6 for batch_id, data in enumerate(train_loader):
7 # calculate the loss and take a gradient step
----> 8 loss += svi.step(data[0].view(-1,28*28), data[1])
9 normalizer_train = len(train_loader.dataset)
10 total_epoch_loss_train = loss / normalizer_train
5 frames
/usr/local/lib/python3.7/dist-packages/pyro/infer/svi.py in step(self, *args, **kwargs)
126 # get loss and compute gradients
127 with poutine.trace(param_only=True) as param_capture:
--> 128 loss = self.loss_and_grads(self.model, self.guide, *args, **kwargs)
129
130 params = set(site["value"].unconstrained()
/usr/local/lib/python3.7/dist-packages/pyro/infer/trace_elbo.py in loss_and_grads(self, model, guide, *args, **kwargs)
129 loss = 0.0
130 # grab a trace from the generator
--> 131 for model_trace, guide_trace in self._get_traces(model, guide, args, kwargs):
132 loss_particle, surrogate_loss_particle = self._differentiable_loss_particle(model_trace, guide_trace)
133 loss += loss_particle / self.num_particles
/usr/local/lib/python3.7/dist-packages/pyro/infer/elbo.py in _get_traces(self, model, guide, args, kwargs)
168 else:
169 for i in range(self.num_particles):
--> 170 yield self._get_trace(model, guide, args, kwargs)
/usr/local/lib/python3.7/dist-packages/pyro/infer/trace_elbo.py in _get_trace(self, model, guide, args, kwargs)
56 """
57 model_trace, guide_trace = get_importance_trace(
---> 58 "flat", self.max_plate_nesting, model, guide, args, kwargs)
59 if is_validation_enabled():
60 check_if_enumerated(guide_trace)
/usr/local/lib/python3.7/dist-packages/pyro/infer/enum.py in get_importance_trace(graph_type, max_plate_nesting, model, guide, args, kwargs, detach)
48 graph_type=graph_type).get_trace(*args, **kwargs)
49 if is_validation_enabled():
---> 50 check_model_guide_match(model_trace, guide_trace, max_plate_nesting)
51
52 guide_trace = prune_subsample_sites(guide_trace)
/usr/local/lib/python3.7/dist-packages/pyro/util.py in check_model_guide_match(model_trace, guide_trace, max_plate_nesting)
252 if model_site["fn"].event_dim != guide_site["fn"].event_dim:
253 raise ValueError("Model and guide event_dims disagree at site '{}': {} vs {}".format(
--> 254 name, model_site["fn"].event_dim, guide_site["fn"].event_dim))
255
256 if hasattr(model_site["fn"], "shape") and hasattr(guide_site["fn"], "shape"):
ValueError: Model and guide event_dims disagree at site 'module$$$out.weight': 0 vs 1
Example 3:
https://alsibahi.xyz/snippets/2019/06/15/pyro_mnist_bnn_kl.html
Problem arise:
AttributeError: 'HiddenLayer' object has no attribute '_batch_shape'
Example 4:
class Model(PyroModule):
def __init__(self, h1=20, h2=20):
super().__init__()
self.fc1 = PyroModule[nn.Linear](1, h1)
self.fc1.weight = PyroSample(dist.Normal(0., 1.).expand([h1, 1]).to_event(2))
self.fc1.bias = PyroSample(dist.Normal(0., 1.).expand([h1]).to_event(1))
self.fc2 = PyroModule[nn.Linear](h1, h2)
self.fc2.weight = PyroSample(dist.Normal(0., 1.).expand([h2, h1]).to_event(2))
self.fc2.bias = PyroSample(dist.Normal(0., 1.).expand([h2]).to_event(1))
self.fc3 = PyroModule[nn.Linear](h2, 1)
self.fc3.weight = PyroSample(dist.Normal(0., 1.).expand([1, h2]).to_event(2))
self.fc3.bias = PyroSample(dist.Normal(0., 1.).expand([1]).to_event(1))
self.relu = nn.ReLU()
def forward(self, x, y=None):
x = x.reshape(-1, 1)
x = self.relu(self.fc1(x))
x = self.relu(self.fc2(x))
mu = self.fc3(x).squeeze()
sigma = pyro.sample("sigma", dist.Uniform(0., 1.))
with pyro.plate("data", x.shape[0]):
obs = pyro.sample("obs", dist.Normal(mu, sigma), obs=y)
return mu
Confused a bit. In my thoughts, the weight shouldn’t be set as the random variables in class ‘model’, , right? The weights as random variables should be set in the guild function instead.
If you know some other examples can work, feel free to post here, thanks!