I have a question about dependency tracking in Pyro. I think my confusion stems from the technical definition of dependence and how batch dimensions work. I am trying to reconcile what I’ve read in the “Aside: Dependency tracking in Pyro” in SVI Part III. One point of confusion for me is that in the tensor shape tutorial when it first talks about batch_shapes, it says
.batch_shapedenote independent random variables, whereas indices over
.event_shapedenote dependent random variables (ie one draw from a distribution).
But then later on, it seems to be saying that even when something has been declared to be a dimension in batch_shape (as opposed to event_shape), if you want the variables in the dimension to be independent, you have to further annotate it with plate (on dev, 0.3).
In trying to understand, I’ve formed a working hypothesis. Even though it is likely incorrect, maybe spelling out my current understanding would be helpful in pinpointing where I’ve gone wrong: Variables in a single batch dimension (-1) have the potential to be independent (as in they are not draws from a multivariable distribution with a defined covariance matrix), but you still need to further annotate to tell Pyro they are in fact independent. A Pyro program shares batch dimensions (-1, -2) across the whole program (across different sample statements). Variables assigned to different dimensions (ie -1 and -2 are considered independent), but within a single dimension (-2), the variables in there (the size of the dimension) are dependent unless further annotated. The prior sentence is true even if the variables come from separate sample statements.
I’ve created a few situations with corresponding questions that I think would help clarify:
a = Bernoulli(0.5).sample(2)
- Are a and a independent?
- Also, Is this the same as doing?
a1 = Bernoulli(0.5).sample(); a2 = Bernoulli(0.5).sample();
- If the above are not independent- is there a probabilistic explanation? Is it that a can depend on a, but not the other way around? Is it that there could be a common ancestor (as in a bayes network) that we have not specified?
with pyro.plate("my_plate", 2): a = Bernoulli(0.5).sample()
- Are a and a independent here?
with pyro.plate("my_plate1", 1, dim=-1): a = Bernoulli(0.5).sample() with pyro.plate("my_plate2", 1, dim=-2): b = Bernoulli(0.5).sample()
- Are a and be independent from one another here?
with pyro.plate("my_plate1", 2): a = Bernoulli(0.5).sample() with pyro.plate("my_plate2", 2): b = Bernoulli(0.5).sample()
- Are a and a independent from one another here?
- What about a and b?