# Sigmoid Belief Network - How to handle Discrete Latent variables with Pyro

#1

Tags: Discrete Latent Variables - Hybrid Latent Variables - Sigmoid Belief Network - Hybrid Guides

I am trying to construct a Sigmoid Belief Network (SBN) in Pyro. Two latent plates of variables (Z_1 and Z_2) explain an observed data set X. The weights are assumed to have a N(0,1) prior distribution and Z_1 follows a Bernoulli(0.5) prior distribution, Z_2 follows a Bernoulli(Z_1’W) distribution (note, elements of Z_2 have equal values for Z_1, but their own weight vector (W). Finally, elements of X follow a Bernoulli(Z_2’W) distribution.

I’m trying to rewrite the Sparse Gamma Deep Exponential Family example for this purpose, but Pyro is not able to handle latent discrete variables, according to an answer on yreddy’s earlier blogpost.

It remains, however, unclear to me what my code should look like. Currently my code looks as follows:
**
EDIT2: Added main programme to code

``````class SigmoidBeliefDEF(object):
def __init__(self):
# define the sizes of the layers in the deep exponential family
self.top_width = 2
self.bottom_width = 3
self.data_size = 5
# define hyperparameters that control the prior
self.p_z = torch.tensor(0.5)
self.mu_w = torch.tensor(0.0)
self.sigma_w = torch.tensor(1.0)

# 1
# define the model
def model(self, x):
x_size = x.size(0)
# 1.1
# sample the global weights
with pyro.plate("w_top_plate", self.top_width * self.bottom_width):
w_top = pyro.sample("w_top", Normal(self.mu_w, self.sigma_w))
with pyro.plate("w_bottom_plate", self.bottom_width * self.data_size):
w_bottom = pyro.sample("w_bottom", Normal(self.mu_w, self.sigma_w))

# 1.2
# sample the local latent random variables
# (the plate encodes the fact that the z's for different datapoints are conditionally independent)
with pyro.plate("data", x_size):
z_top = pyro.sample("z_top", Bernoulli(self.p_z).expand([self.top_width]).to_event(1))
# note that we need to use matmul (batch matrix multiplication) as well as appropriate reshaping
# to make sure our code is fully vectorized
w_top = w_top.reshape(self.top_width, self.bottom_width) if w_top.dim() == 1 else \
w_top.reshape(-1, self.top_width, self.bottom_width)
mean_bottom = torch.sigmoid(torch.matmul(z_top, w_top))
z_bottom = pyro.sample("z_bottom", Bernoulli(mean_bottom).to_event(1))

w_bottom = w_bottom.reshape(self.bottom_width, self.data_size) if w_bottom.dim() == 1 else \
w_bottom.reshape(-1, self.bottom_width, self.data_size)
mean_obs = torch.sigmoid(torch.matmul(z_bottom, w_bottom))

# observe the data using a Bernoulli likelihood
pyro.sample('obs', Bernoulli(mean_obs).to_event(1), obs=x)

def main(args):
dataset_path = Path(r"C:\Users\posc8001\Documents\DEF\Data\Simulation_1")
file_to_open = dataset_path / "small_data.csv"
f = open(file_to_open)
sigmoid_belief_def = SigmoidBeliefDEF()

# Specify hyperparameters of optimization
learning_rate = 0.2
momentum = 0.05

# Specify the guide
guide = AutoGuideList(sigmoid_belief_def.model)
hide=["assignment"])))
expose=["assignment"])))
guide = guide if args.auto_guide else sigmoid_belief_def.guide

# Specify Stochastic Variational Inference
svi = SVI(sigmoid_belief_def.model, guide, opt, loss=TraceMeanField_ELBO())

# we use svi_eval during evaluation; since we took care to write down our model in
# a fully vectorized way, this computation can be done efficiently with large tensor ops
svi_eval = SVI(sigmoid_belief_def.model, guide, opt,
loss=TraceMeanField_ELBO(num_particles=args.eval_particles, vectorize_particles=True))

# the training loop
for k in range(args.num_epochs):
loss = svi.step(data)

if k % args.eval_frequency == 0 and k > 0 or k == args.num_epochs - 1:
loss = svi_eval.evaluate_loss(data)
print("[epoch %04d] training elbo: %.4g" % (k, -loss))
``````

It is suggested to use `poutine.block` but it is unclear to me in which implementation I should use it in my code. Wrapping, for instance, the `pyro.sample("z_top",[args])` part in a `poutine.block` call didn’t do the trick. Is there anyone who might help me out? Either by elaborating on the use of `poutine.block` or by proposing an alternative solution?

**
EDIT:
I decided to add the error that is returned to me upon running the programme.

``````Traceback (most recent call last):
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\torch\distributions\constraint_registry.py", line 139, in __call__
factory = self._registry[type(constraint)]
KeyError: <class 'torch.distributions.constraints._Boolean'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\poutine\trace_messenger.py", line 147, in __call__
ret = self.fn(*args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\contrib\autoguide\__init__.py", line 190, in __call__
result.update(part(*args, **kwargs))
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\contrib\autoguide\__init__.py", line 377, in __call__
self._setup_prototype(*args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\contrib\autoguide\__init__.py", line 326, in _setup_prototype
self._unconstrained_shapes[name] = biject_to(site["fn"].support).inv(site["value"]).shape
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\torch\distributions\constraint_registry.py", line 143, in __call__
return factory(constraint)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\distributions\torch_distribution.py", line 226, in <lambda>
biject_to.register(IndependentConstraint, lambda c: biject_to(c.base_constraint))
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\torch\distributions\constraint_registry.py", line 142, in __call__
'Cannot transform {} constraints'.format(type(constraint).__name__))
NotImplementedError: Cannot transform _Boolean constraints

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/posc8001/Documents/DEF/Scipio_DEF/sigmoid_belief_network.py", line 216, in <module>
model = main(args)
File "C:/Users/posc8001/Documents/DEF/Scipio_DEF/sigmoid_belief_network.py", line 193, in main
loss = svi.step(data)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\infer\svi.py", line 99, in step
loss = self.loss_and_grads(self.model, self.guide, *args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\infer\trace_elbo.py", line 125, in loss_and_grads
for model_trace, guide_trace in self._get_traces(model, guide, *args, **kwargs):
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\infer\elbo.py", line 163, in _get_traces
yield self._get_trace(model, guide, *args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\infer\trace_mean_field_elbo.py", line 75, in _get_trace
model, guide, *args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\infer\trace_elbo.py", line 52, in _get_trace
"flat", self.max_plate_nesting, model, guide, *args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\infer\enum.py", line 42, in get_importance_trace
guide_trace = poutine.trace(guide, graph_type=graph_type).get_trace(*args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\poutine\trace_messenger.py", line 169, in get_trace
self(*args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\poutine\trace_messenger.py", line 153, in __call__
traceback)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\six.py", line 692, in reraise
raise value.with_traceback(tb)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\poutine\trace_messenger.py", line 147, in __call__
ret = self.fn(*args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\contrib\autoguide\__init__.py", line 190, in __call__
result.update(part(*args, **kwargs))
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\contrib\autoguide\__init__.py", line 377, in __call__
self._setup_prototype(*args, **kwargs)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\contrib\autoguide\__init__.py", line 326, in _setup_prototype
self._unconstrained_shapes[name] = biject_to(site["fn"].support).inv(site["value"]).shape
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\torch\distributions\constraint_registry.py", line 143, in __call__
return factory(constraint)
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\pyro\distributions\torch_distribution.py", line 226, in <lambda>
biject_to.register(IndependentConstraint, lambda c: biject_to(c.base_constraint))
File "C:\Users\posc8001\.virtualenvs\Scipio_DEF-F7b0vflQ\lib\site-packages\torch\distributions\constraint_registry.py", line 142, in __call__
'Cannot transform {} constraints'.format(type(constraint).__name__))
NotImplementedError: Cannot transform _Boolean constraints
Trace Shapes:
Param Sites:
Sample Sites:
data dist       |
value 20000 |
w_bottom_plate dist       |
value    15 |
w_top_plate dist       |
value     6 |

Process finished with exit code 1
``````

#2

It’s hard to say for sure what’s going wrong since the code that actually produced the error isn’t included in your post, but it looks from the stack trace like you’re trying to use SVI with a continuous autoguide. The error is telling you that you can’t use a continuous variational distribution for a discrete latent variable. You’ll need to write your own guide or use a combination of a continuous and discrete autoguide (`pyro.contrib.autoguide.AutoDiscreteParallel`) via `pyro.contrib.autoguide.AutoGuideList`.

Note that inference in models like this with lots of discrete latent variables is difficult, especially variational inference, and if you want to work with a larger version of the model you may need to write your own guide and use `pyro.infer.TraceGraph_ELBO` with neural baselines to reduce gradient variance. See the SVI gradient estimator tutorial and one of the references there on amortized variational inference in SBNs for more background.

Reparameterization of latent variables failed
#3

Hi @eb8680_2,

Thanks for your fast reply. I updated my question to include the main programme.

Thank you as well for pointing out the `AutoGuideList` option. I forgot to mention it in my initial question, but I attempted to implement it. It still returns the aforementioned error though. Did I use a wrong implementation? (see my implementation below)

``````# Specify the guide
guide = AutoGuideList(sigmoid_belief_def.model)