Pyro Bayesian GPLVM SVI with minibatching

Tutorials
1

Hello everyone,
I am new to probabilistic programming and am currently working with the Gaussian process latent variable model (gplvm) as outlined in the link below.

http://docs.pyro.ai/en/0.2.1-release/_modules/pyro/contrib/gp/models/gplvm.html

However the problem with this model is that is does not support mini batching with large data sets. I went through the SVI tutorial part 2 but am having a little trouble adding mini batching in the current implementation to scale it up to large data sets. How do i incorporate the following lines used for SVI part 2 in the model and the guide?

  for i in pyro.irange("data_loop", len(data), subsample_size=self.batch_size):
             pyro.sample("obs_{}".format(i), dist.MultivariateNormal(X_loc[i],X_scale_tril[i]), obs=data[i])

If data is a 150 x 4 tensor how should i sample mini batch of this high dimensional data in the above statement and how to incorporate it in the model and guide. Any help on this would be appreciated.

def model(self):
self.set_mode(“model”, recursive=False)

    # sample X from unit multivariate normal distribution
    zero_loc = self.X_loc.new_zeros(self.X_loc.shape)
    C = self.X_loc.shape[1]
    Id = torch.eye(C, out=self.X_loc.new_empty(C, C))
    X_name = param_with_module_name(self.name, "X")
    X = pyro.sample(X_name, dist.MultivariateNormal(zero_loc, scale_tril=Id)
                                .independent(zero_loc.dim()-1))
    self.base_model.set_data(X, self.y)
    self.base_model.model()

def guide(self):
self.set_mode(“guide”, recursive=False)

    # sample X from variational multivariate normal distribution
    X_loc = self.get_param("X_loc")
    X_scale_tril = self.get_param("X_scale_tril")
    X_name = param_with_module_name(self.name, "X")
    X = pyro.sample(X_name,
                    dist.MultivariateNormal(X_loc, scale_tril=X_scale_tril)
                        .independent(X_loc.dim()-1))

    self.base_model.set_data(X, self.y)
    if self._call_base_model_guide:
        self.base_model.guide()
2

@aneeqr You are right that the current version does not support mini-batch. Indeed, it is not clear how to do it using the current gplvm api. I have no idea how to set up guide for the latent X to incorporate mini-batch training and inference.

If you come up with a right way to draw the latent minibatch X, then you can use vsgp model. It supports mini_batch, no need to use iarange. You just need to use .set_data with the right X_minibatch and y_minibatch.

1 Like
3

Hey,
Thanks although the problem statement I have is more on unsupervised learning. I am trying to play around with the current implementation of the gplvm as that’s my main model of interest but just to set my concepts clear when we talk of minibatching for un-supervised learning, does this means reducing the batch size of the Y high dimensional data or is it related to the lower dimensional representation representation X, in case of the current gplvm can anyone confirm the global and local variational parameters? And how do we scale the results? Also can anyone direct me to any probabilistic program that has this current variational gplvm with SVI and supports mini-batching? I couldn’t find anything on Edward and Stan and PyMC3 or anything like a black box that I could use. There are implementations of this on GPy but that’s more manually calculated and I really want to see the same achieved via a probabilistic program.
Thank you
Aneeq

4

@aneeqr Could you please point out some reference? I don’t understand GPy’s implementation so I can’t say much.

About unsupervised: yes, we use GPLVM for unsupervised learning.
About mini-batch: I think calculation for high dimensional data is not a big problem. Big problem comes from a large amount of data. So in my opinion, mini-batch learning means that we just want to deal with a small amount of data for each step.
About scaling results: vsgp does it for you (using poutine.scale).
About other probabilistic programs: I have no idea.

As I said, VSGP already supports mini-batch learning. All you need is to feed data for them using set_data method. I would use Normal distribution instead of MultivariateNormal distribution (because MultivariateNormal’s scale_tril is not compatible to mini-batch selection) and change the implementation of gplvm’s guide and model to something like this:

# first, for each step, set minibatch_indices
X_minibatch = pyro.sample(..., dist.Normal(x_loc[minibatch_indices], x_scale[minibatch_indices]))
y_minibatch = y[minibatch_indices]
self.base_model.set_data(X_minibatch, y_minibatch)
5

Thanks!
I was talking about this model.
http://gpy.readthedocs.io/en/deploy/_modules/GPy/models/bayesian_gplvm_minibatch.html#BayesianGPLVMMiniBatch

The current implementation of Pyro supports for gplvm with SVI but doesnt do minibatching. Thanks for the feedback!

6

@aneeqr I don’t understand what GPy does, so I can’t say much. It seems that it works like the suggestion in my last comment. Could you please confirm it? If so, you can modify the code of Pyro’s gplvm to do mini-batch.