I’m interested in implementing the finite version of the Infinite Overlapping Mixture Model paper by Heller and Ghahramani (http://mlg.eng.cam.ac.uk/zoubin/papers/HelGha07over.pdf) where the mixture model is defined as follows:
Here, unlike typical mixture models, the mixture is defined as a product of k components that belong to the exponential family. z_ik is a binary vector (which might have multiple 1s) and c is the normalization constant. (Note that this is not same as the product of random variables)
I’m thinking of possible ways to do this in Pyro, but before that, I’d like to show you also the derivation of this product for exponential family distributions. As authors state in the paper:
Product of exp. fam. distributions is a new distribution of the same family with a natural parameter that is the sum of the natural parameters of the components. Now I wonder if this can be elegantly implemented in Pyro.
I’m not familiar with Pyro but I think it’s very cool that all exp.fam. distributions in PyTorch have
_natural_params property which I can use for this purpose, however, this property uses “native” parameters of the distributions e.g.
self.scale in the Normal case and
self.probs in the Bernoulli case.
I can write classes for the distributions I need by inheriting from existing classes of Pyro, but I was wondering if there is an easier way to define a generic class like ProductDistribution which takes a list of exp.fam. distribution objects and behaves like the product I mentioned above.