Transforming input X with GPLVM

Hi,

I have a dataset of 250 examples x 300 features, each normally distributed. Each assigned a binary class label. I followed the example given here: http://pyro.ai/examples/gplvm.html

I planned to make this example work on the input dataset X. It actually appears to have produced a somewhat expected result:

In my attempt to classify held out data I get an error:

 gplvm.forward(test_data)

ValueError: Train data and test data should have the same shape of features, but got torch.Size([2]) and torch.Size([300]).

This makes sense. I know I’m not using this method as it is meant to be used but I was wondering if it was possible to make it work. Is there a way to run inference over the test_data? You can assume i used the same train util as in cell 7 of the tutorial.

Hi @gdmacmillan, to my knowledge (please correct me if I am wrong, I am happy to read further references to see how people uses GPLVM to classify held out data), after training GPLVM, we get good hyperparameters for the data. Then when new data is provided, we train GPLVM again with these fixed hyperparameters to learn X. Does it make sense to you?

To fix hyperparameters, you can print out all named parameters print(gplvm.named_parameters() and set requires_grad_(False) for each parameter (exept X). For example,

gplvm.Xu.requires_grad_(False)

To set new data, you can use gplvm.set_data(X_new, y_new) where y_new is held out set and X_new is new parameter (with correct new shape). Then you can redefine prior/guide for X if you want.

Hi Fehiepsi, your explanation makes sense. I am trying this but a little confused on how I am supposed to enter the new data. I tried:

test_data = torch.tensor(test_df[features].values, dtype=torch.get_default_dtype())
# transpose data to correct its shape
y_test = test_data.t()
X_test_prior_mean = torch.zeros(y_test.size(1), 2) # shape: 19750 x 2
X_test = Parameter(X_test_prior_mean.clone())

for _, param in gplvm.named_parameters():
    param.requires_grad_(True)

gplvm.set_data(X_test, y_test)

losses = gp.util.train(gplvm, num_steps=4000)

But i get the error:
KeyError: “attribute ‘X’ already exists”

Another note

The example defines the model as:

First, we need to define the output tensor y. To predict values for all 48 genes, we need 48 Gaussian processes. So the required shape for y is num_GPs x num_data = 48 x 437.

My setup was to treat each feature as a process and I have 300 features and 250 observations. With this my shape for y becomes 300 x 250.

The set_data method only does the job self.X = X. ~This is possible for my PyTorch version 1.1.0.~ Maybe the error happens because the old prior/guide of X are available. How about adding this line to your code

del gplvm._priors['X']

Edit I think that the problem is X is moved to buffer (no longer Parameter) when we set prior. Could you try this instead

del gplvm.X
del gplvm._priors['X']
gplvm.set_data(X_test, y_test)
...

Ah no unfortunately. Here is the entire stack trace:


KeyError Traceback (most recent call last)
in ()
----> 1 losses = gp.util.train(gplvm, num_steps=4000)

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/util.py in train(gpmodule, optimizer, loss_fn, retain_graph, num_steps)
175 losses = []
176 for i in range(num_steps):
–> 177 loss = optimizer.step(closure)
178 losses.append(torch_item(loss))
179 return losses

/opt/conda/lib/python3.6/site-packages/torch/optim/adam.py in step(self, closure)
56 loss = None
57 if closure is not None:
—> 58 loss = closure()
59
60 for group in self.param_groups:

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/util.py in closure()
169 def closure():
170 optimizer.zero_grad()
–> 171 loss = loss_fn(gpmodule.model, gpmodule.guide)
172 torch_backward(loss, retain_graph)
173 return loss

/opt/conda/lib/python3.6/site-packages/pyro/infer/trace_elbo.py in differentiable_loss(self, model, guide, *args, **kwargs)
106 loss = 0.
107 surrogate_loss = 0.
–> 108 for model_trace, guide_trace in self._get_traces(model, guide, *args, **kwargs):
109 loss_particle, surrogate_loss_particle = self._differentiable_loss_particle(model_trace, guide_trace)
110 surrogate_loss += surrogate_loss_particle / self.num_particles

/opt/conda/lib/python3.6/site-packages/pyro/infer/elbo.py in _get_traces(self, model, guide, *args, **kwargs)
166 else:
167 for i in range(self.num_particles):
–> 168 yield self._get_trace(model, guide, *args, **kwargs)

/opt/conda/lib/python3.6/site-packages/pyro/infer/trace_mean_field_elbo.py in _get_trace(self, model, guide, *args, **kwargs)
65 def _get_trace(self, model, guide, *args, **kwargs):
66 model_trace, guide_trace = super(TraceMeanField_ELBO, self)._get_trace(
—> 67 model, guide, *args, **kwargs)
68 if is_validation_enabled():
69 _check_mean_field_requirement(model_trace, guide_trace)

/opt/conda/lib/python3.6/site-packages/pyro/infer/trace_elbo.py in _get_trace(self, model, guide, *args, **kwargs)
50 “”"
51 model_trace, guide_trace = get_importance_trace(
—> 52 “flat”, self.max_plate_nesting, model, guide, *args, **kwargs)
53 if is_validation_enabled():
54 check_if_enumerated(guide_trace)

/opt/conda/lib/python3.6/site-packages/pyro/infer/enum.py in get_importance_trace(graph_type, max_plate_nesting, model, guide, *args, **kwargs)
40 against it.
41 “”"
—> 42 guide_trace = poutine.trace(guide, graph_type=graph_type).get_trace(*args, **kwargs)
43 model_trace = poutine.trace(poutine.replay(model, trace=guide_trace),
44 graph_type=graph_type).get_trace(*args, **kwargs)

/opt/conda/lib/python3.6/site-packages/pyro/poutine/trace_messenger.py in get_trace(self, *args, **kwargs)
167 Calls this poutine and returns its trace instead of the function’s return value.
168 “”"
–> 169 self(*args, **kwargs)
170 return self.msngr.get_trace()

/opt/conda/lib/python3.6/site-packages/pyro/poutine/trace_messenger.py in call(self, *args, **kwargs)
145 args=args, kwargs=kwargs)
146 try:
–> 147 ret = self.fn(*args, **kwargs)
148 except (ValueError, RuntimeError):
149 exc_type, exc_value, traceback = sys.exc_info()

/opt/conda/lib/python3.6/site-packages/pyro/contrib/autoname/scoping.py in _fn(*args, **kwargs)
73 def _fn(*args, **kwargs):
74 with type(self)(prefix=self.prefix, inner=self.inner):
—> 75 return fn(*args, **kwargs)
76 return _fn
77

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/models/sgpr.py in guide(self)
160 @autoname.scope(prefix=“SGPR”)
161 def guide(self):
–> 162 self.set_mode(“guide”)
163
164 def forward(self, Xnew, full_cov=False, noiseless=True):

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/parameterized.py in set_mode(self, mode)
211 for module in self.modules():
212 if isinstance(module, Parameterized):
–> 213 module.mode = mode
214
215 @property

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in setattr(self, name, value)
581 buffers[name] = value
582 else:
–> 583 object.setattr(self, name, value)
584
585 def delattr(self, name):

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/parameterized.py in mode(self, mode)
226 self._register_param(name)
227 for name in self._priors:
–> 228 self._register_param(name)
229
230 def _sample_from_guide(self, name):

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/parameterized.py in _register_param(self, name)
270 p_unconstrained = self._parameters["{}_unconstrained".format(name)]
271 p = transform_to(self._constraints[name])(p_unconstrained)
–> 272 self.register_buffer(name, p)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in register_buffer(self, name, tensor)
118 raise KeyError(“buffer name can’t be empty string “””)
119 elif hasattr(self, name) and name not in self._buffers:
–> 120 raise KeyError(“attribute ‘{}’ already exists”.format(name))
121 elif tensor is not None and not isinstance(tensor, torch.Tensor):
122 raise TypeError("cannot assign ‘{}’ object to buffer ‘{}’ "

KeyError: “attribute ‘X’ already exists”

Thanks for posting the entire stack! Could you try again the solution in my last comment? (I just edited it) If it still gives problems, I’ll run the example again by myself and get back to you.

del gplvm.X
del gplvm._priors['X']
gplvm.set_data(X_test, y_test)

Thank you for your reply and the help!

Deleting the data and the priors seems to have worked. I was able to run the utils.train method for 1000 steps. I am running this on the CPU and my test data contains 19750 examples so it was very slow… next step is to speed it up with GPU.

For the plot instead of using variable x loc for the mean and standard deviation of the posterior. I was able to use the X sample points by calling

gplvm.mode = "guide"
X = gplvm.X.detach().numpy()

then:

plt.figure(figsize=(8, 6))
plt.scatter(X[:, 0], X[:, 1])

this is the result:

This is a weird result as it does have pattern but it’s not what I was expecting and I can’t quite comprehend it.

It is interesting and to be honest, I don’t know much about how to use GPLVM to classify test set. The above suggestion might not work. The best reference I can find is this paper where authors suggest to maximize log likelihood of p(X_new, y_new | X, y) (section 5.1).

Because you want to have X aligned around 0 and 1, some sort of prior should be placed at these places. But I don’t know how to go further from that. :frowning:

@gdmacmillan Do you have further progress/reference on using GPLVM to classify new data? I am curious and interested in this problem. :slight_smile:

Following the arguments in the above paper, I think that we can process along the following line

class Predict(gp.Parameterized):
    def __init__(self, y_new):
        self.X_new = nn.Parameter(...)  # with correct shape
    def model(self):
        # conditional on past data
        y_new_loc, y_new_var = gplvm(X_new)
        pyro.sample("y_new", dist.Normal(y_new_loc, y_new_var.sqrt()), obs=y_new)
    def guide(self):
        pass

and train this model to learn X_new. This way, what we are doing is to maximize log likelihood of
p(y_new | X_new, X, y, hyperparameters). If you set prior and guide for X_new, you come up with the same approach with the above paper, which is to maximize log likelihood of p(y_new, X_new | X, y, hyperparameters). Please let me know if it works for your problem. I don’t have your data so I can’t test it, but this approach is more reasonable to me. :slight_smile:

@fehiepsi I’m sorry for the late reply. I believe you are on the right track and your approach certainly seems reasonable to me but I must admit I am not the best for understanding this problem. I have just started to look into Gaussian Processes and probabilistic modeling as a whole this year so I am still relatively new to the math and language. I will be trying to improve on the latent variable unsupervised learning model but first I went back to the drawing board and started to take a look at supervised learning with sparse gp’s. Anyways since not much code was out there showing how to train gaussian process logistic regression, i thought i would put together a short kernel:

https://www.kaggle.com/gdmacmillan/gp-logistic-regression

fyi, this is the same data as I tried to use with GPLVM

Yup, logistic regression is a better approach to this problem. But I am still curious about how to transfer a gplvm model to new dataset. Thanks for sharing your data! I’ll play with it and let you know if I can come up with a working version.^^

@fehiepsi Any luck with developing GPLVM for this binary dataset or any more unsupervised classification approaches for this problem?

Hi @gdmacmillan, I have not had time to jump into this yet. Will find some time this weekend to see how to tackle this problem. :slight_smile:

Hi @fehiepsi,

I got back to the problem and tried to follow your advice. I’m first running the tutorial Gaussian Process Latent Variable Model (http://pyro.ai/examples/gplvm.html) and then run

class Predict(gp.parameterized.Parameterized):
    def __init__(self, y_new, gplvm):
        super(Predict, self).__init__()
        self.y_new = y_new
        self.gplvm = gplvm
        self.X_new = Parameter(torch.zeros(1,2))
        
    def model(self):
        # conditional on past data
        y_new_loc, y_new_var = self.gplvm(self.X_new)        
        pyro.sample("y_new", dist.Normal(y_new_loc[:,0], y_new_var[:,0,0].sqrt()).independent(1), obs=self.y_new[:,0])
        
    def guide(self):
        pass    

## Test to get backwards (from Y to X) with any random sample from previous Y.
y_new = torch.tensor(df.values[[42],:], dtype=torch.get_default_dtype()).t()
p = Predict(y_new, gplvm)
losses = gp.util.train(p, num_steps=100)

When I run the last line, I get the error that ‘SGPR/SparseGPRegression/X’ is in guide but not in the model. I’m not sure how to solve this, since I want to optimize X_new and consider gplvm as fixed.

Would be nice if you can help me here, and also check the code above.

Here is the complete traceback:

~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/util.py:182: UserWarning: Found non-auxiliary vars in guide but not model, consider marking these infer={'is_auxiliary': True}:
{'SGPR/SparseGPRegression/X'}
  guide_vars - aux_vars - model_vars))
Traceback (most recent call last):

  File "<ipython-input-74-3b6a156082b6>", line 24, in <module>
    losses = gp.util.train(p, num_steps=100)

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/contrib/gp/util.py", line 177, in train
    loss = optimizer.step(closure)

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/torch/optim/adam.py", line 58, in step
    loss = closure()

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/contrib/gp/util.py", line 171, in closure
    loss = loss_fn(gpmodule.model, gpmodule.guide)

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/infer/trace_elbo.py", line 109, in differentiable_loss
    loss_particle, surrogate_loss_particle = self._differentiable_loss_particle(model_trace, guide_trace)

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/infer/trace_mean_field_elbo.py", line 95, in _differentiable_loss_particle
    guide_site = guide_trace.nodes[name]

KeyError: 'SparseGPRegression/X'

Many thanks!

@gammatam @gdmacmillan I make an effort at https://gist.github.com/fehiepsi/8046cdc3b84b01568962ed1ac64cc2a8, where a few more points are required

  • freeze learned parameters
for param in gplvm.parameters():
    param.requires_grad_(False)
  • Fix X
# get a sample and detach
#gplvm.mode = "guide"
#gplvm.X = gplvm.X.detach()
# or use the prediction mean
gplvm.X = gplvm.X_loc
  • Remove prior
del gplvm._priors['X']

It seems to me that the model can somehow do its job (despite that I set initial position for all of them at 0) but not as good as if we set priors.