Transforming input X with GPLVM

gdmacmillan · May 7, 2019, 2:25am

Hi,

I have a dataset of 250 examples x 300 features, each normally distributed. Each assigned a binary class label. I followed the example given here: http://pyro.ai/examples/gplvm.html

I planned to make this example work on the input dataset X. It actually appears to have produced a somewhat expected result:

In my attempt to classify held out data I get an error:

 gplvm.forward(test_data)

ValueError: Train data and test data should have the same shape of features, but got torch.Size([2]) and torch.Size([300]).

This makes sense. I know I’m not using this method as it is meant to be used but I was wondering if it was possible to make it work. Is there a way to run inference over the test_data? You can assume i used the same train util as in cell 7 of the tutorial.

fehiepsi · May 7, 2019, 4:47am

Hi @gdmacmillan, to my knowledge (please correct me if I am wrong, I am happy to read further references to see how people uses GPLVM to classify held out data), after training GPLVM, we get good hyperparameters for the data. Then when new data is provided, we train GPLVM again with these fixed hyperparameters to learn X. Does it make sense to you?

To fix hyperparameters, you can print out all named parameters print(gplvm.named_parameters() and set requires_grad_(False) for each parameter (exept X). For example,

gplvm.Xu.requires_grad_(False)

To set new data, you can use gplvm.set_data(X_new, y_new) where y_new is held out set and X_new is new parameter (with correct new shape). Then you can redefine prior/guide for X if you want.

gdmacmillan · May 7, 2019, 6:37pm

Hi Fehiepsi, your explanation makes sense. I am trying this but a little confused on how I am supposed to enter the new data. I tried:

test_data = torch.tensor(test_df[features].values, dtype=torch.get_default_dtype())
# transpose data to correct its shape
y_test = test_data.t()
X_test_prior_mean = torch.zeros(y_test.size(1), 2) # shape: 19750 x 2
X_test = Parameter(X_test_prior_mean.clone())

for _, param in gplvm.named_parameters():
    param.requires_grad_(True)

gplvm.set_data(X_test, y_test)

losses = gp.util.train(gplvm, num_steps=4000)

But i get the error:
KeyError: “attribute ‘X’ already exists”

gdmacmillan · May 7, 2019, 6:41pm

Another note

The example defines the model as:

First, we need to define the output tensor y. To predict values for all 48 genes, we need 48 Gaussian processes. So the required shape for y is num_GPs x num_data = 48 x 437.

My setup was to treat each feature as a process and I have 300 features and 250 observations. With this my shape for y becomes 300 x 250.

fehiepsi · May 7, 2019, 7:22pm

The set_data method only does the job self.X = X. ~This is possible for my PyTorch version 1.1.0.~ Maybe the error happens because the old prior/guide of X are available. How about adding this line to your code

del gplvm._priors['X']

Edit I think that the problem is X is moved to buffer (no longer Parameter) when we set prior. Could you try this instead

del gplvm.X
del gplvm._priors['X']
gplvm.set_data(X_test, y_test)
...

gdmacmillan · May 7, 2019, 7:33pm

Ah no unfortunately. Here is the entire stack trace:

KeyError Traceback (most recent call last)
in ()
----> 1 losses = gp.util.train(gplvm, num_steps=4000)

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/util.py in train(gpmodule, optimizer, loss_fn, retain_graph, num_steps)
175 losses =
176 for i in range(num_steps):
→ 177 loss = optimizer.step(closure)
178 losses.append(torch_item(loss))
179 return losses

/opt/conda/lib/python3.6/site-packages/torch/optim/adam.py in step(self, closure)
56 loss = None
57 if closure is not None:
—> 58 loss = closure()
59
60 for group in self.param_groups:

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/util.py in closure()
169 def closure():
170 optimizer.zero_grad()
→ 171 loss = loss_fn(gpmodule.model, gpmodule.guide)
172 torch_backward(loss, retain_graph)
173 return loss

/opt/conda/lib/python3.6/site-packages/pyro/infer/trace_elbo.py in differentiable_loss(self, model, guide, *args, **kwargs)
106 loss = 0.
107 surrogate_loss = 0.
→ 108 for model_trace, guide_trace in self._get_traces(model, guide, *args, **kwargs):
109 loss_particle, surrogate_loss_particle = self._differentiable_loss_particle(model_trace, guide_trace)
110 surrogate_loss += surrogate_loss_particle / self.num_particles

/opt/conda/lib/python3.6/site-packages/pyro/infer/elbo.py in _get_traces(self, model, guide, *args, **kwargs)
166 else:
167 for i in range(self.num_particles):
→ 168 yield self._get_trace(model, guide, *args, **kwargs)

/opt/conda/lib/python3.6/site-packages/pyro/infer/trace_mean_field_elbo.py in _get_trace(self, model, guide, *args, **kwargs)
65 def _get_trace(self, model, guide, *args, **kwargs):
66 model_trace, guide_trace = super(TraceMeanField_ELBO, self)._get_trace(
—> 67 model, guide, *args, **kwargs)
68 if is_validation_enabled():
69 _check_mean_field_requirement(model_trace, guide_trace)

/opt/conda/lib/python3.6/site-packages/pyro/infer/trace_elbo.py in _get_trace(self, model, guide, *args, **kwargs)
50 “”"
51 model_trace, guide_trace = get_importance_trace(
—> 52 “flat”, self.max_plate_nesting, model, guide, *args, **kwargs)
53 if is_validation_enabled():
54 check_if_enumerated(guide_trace)

/opt/conda/lib/python3.6/site-packages/pyro/infer/enum.py in get_importance_trace(graph_type, max_plate_nesting, model, guide, *args, **kwargs)
40 against it.
41 “”"
—> 42 guide_trace = poutine.trace(guide, graph_type=graph_type).get_trace(*args, **kwargs)
43 model_trace = poutine.trace(poutine.replay(model, trace=guide_trace),
44 graph_type=graph_type).get_trace(*args, **kwargs)

/opt/conda/lib/python3.6/site-packages/pyro/poutine/trace_messenger.py in get_trace(self, *args, **kwargs)
167 Calls this poutine and returns its trace instead of the function’s return value.
168 “”"
→ 169 self(*args, **kwargs)
170 return self.msngr.get_trace()

/opt/conda/lib/python3.6/site-packages/pyro/poutine/trace_messenger.py in call(self, *args, **kwargs)
145 args=args, kwargs=kwargs)
146 try:
→ 147 ret = self.fn(*args, **kwargs)
148 except (ValueError, RuntimeError):
149 exc_type, exc_value, traceback = sys.exc_info()

/opt/conda/lib/python3.6/site-packages/pyro/contrib/autoname/scoping.py in _fn(*args, **kwargs)
73 def _fn(*args, **kwargs):
74 with type(self)(prefix=self.prefix, inner=self.inner):
—> 75 return fn(*args, **kwargs)
76 return _fn
77

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/models/sgpr.py in guide(self)
160 @autoname.scope(prefix=“SGPR”)
161 def guide(self):
→ 162 self.set_mode(“guide”)
163
164 def forward(self, Xnew, full_cov=False, noiseless=True):

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/parameterized.py in set_mode(self, mode)
211 for module in self.modules():
212 if isinstance(module, Parameterized):
→ 213 module.mode = mode
214
215 @property

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in setattr(self, name, value)
581 buffers[name] = value
582 else:
→ 583 object.setattr(self, name, value)
584
585 def delattr(self, name):

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/parameterized.py in mode(self, mode)
226 self._register_param(name)
227 for name in self._priors:
→ 228 self._register_param(name)
229
230 def _sample_from_guide(self, name):

/opt/conda/lib/python3.6/site-packages/pyro/contrib/gp/parameterized.py in _register_param(self, name)
270 p_unconstrained = self._parameters[“{}_unconstrained”.format(name)]
271 p = transform_to(self._constraints[name])(p_unconstrained)
→ 272 self.register_buffer(name, p)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in register_buffer(self, name, tensor)
118 raise KeyError(“buffer name can’t be empty string ""”)
119 elif hasattr(self, name) and name not in self._buffers:
→ 120 raise KeyError(“attribute ‘{}’ already exists”.format(name))
121 elif tensor is not None and not isinstance(tensor, torch.Tensor):
122 raise TypeError("cannot assign ‘{}’ object to buffer ‘{}’ "

KeyError: “attribute ‘X’ already exists”

fehiepsi · May 7, 2019, 7:50pm

Thanks for posting the entire stack! Could you try again the solution in my last comment? (I just edited it) If it still gives problems, I’ll run the example again by myself and get back to you.

del gplvm.X
del gplvm._priors['X']
gplvm.set_data(X_test, y_test)

gdmacmillan · May 8, 2019, 1:53pm

Thank you for your reply and the help!

Deleting the data and the priors seems to have worked. I was able to run the utils.train method for 1000 steps. I am running this on the CPU and my test data contains 19750 examples so it was very slow… next step is to speed it up with GPU.

For the plot instead of using variable x loc for the mean and standard deviation of the posterior. I was able to use the X sample points by calling

gplvm.mode = "guide"
X = gplvm.X.detach().numpy()

then:

plt.figure(figsize=(8, 6))
plt.scatter(X[:, 0], X[:, 1])

this is the result:

This is a weird result as it does have pattern but it’s not what I was expecting and I can’t quite comprehend it.

fehiepsi · May 8, 2019, 5:56pm

It is interesting and to be honest, I don’t know much about how to use GPLVM to classify test set. The above suggestion might not work. The best reference I can find is this paper where authors suggest to maximize log likelihood of p(X_new, y_new | X, y) (section 5.1).

Because you want to have X aligned around 0 and 1, some sort of prior should be placed at these places. But I don’t know how to go further from that.

fehiepsi · May 13, 2019, 8:18pm

@gdmacmillan Do you have further progress/reference on using GPLVM to classify new data? I am curious and interested in this problem.

fehiepsi · May 13, 2019, 11:20pm

Following the arguments in the above paper, I think that we can process along the following line

class Predict(gp.Parameterized):
    def __init__(self, y_new):
        self.X_new = nn.Parameter(...)  # with correct shape
    def model(self):
        # conditional on past data
        y_new_loc, y_new_var = gplvm(X_new)
        pyro.sample("y_new", dist.Normal(y_new_loc, y_new_var.sqrt()), obs=y_new)
    def guide(self):
        pass

and train this model to learn X_new. This way, what we are doing is to maximize log likelihood of
p(y_new | X_new, X, y, hyperparameters). If you set prior and guide for X_new, you come up with the same approach with the above paper, which is to maximize log likelihood of p(y_new, X_new | X, y, hyperparameters). Please let me know if it works for your problem. I don’t have your data so I can’t test it, but this approach is more reasonable to me.

gdmacmillan · May 14, 2019, 4:55am

@fehiepsi I’m sorry for the late reply. I believe you are on the right track and your approach certainly seems reasonable to me but I must admit I am not the best for understanding this problem. I have just started to look into Gaussian Processes and probabilistic modeling as a whole this year so I am still relatively new to the math and language. I will be trying to improve on the latent variable unsupervised learning model but first I went back to the drawing board and started to take a look at supervised learning with sparse gp’s. Anyways since not much code was out there showing how to train gaussian process logistic regression, i thought i would put together a short kernel:

fyi, this is the same data as I tried to use with GPLVM

fehiepsi · May 14, 2019, 3:07pm

Yup, logistic regression is a better approach to this problem. But I am still curious about how to transfer a gplvm model to new dataset. Thanks for sharing your data! I’ll play with it and let you know if I can come up with a working version.^^

gdmacmillan · May 20, 2019, 2:12pm

@fehiepsi Any luck with developing GPLVM for this binary dataset or any more unsupervised classification approaches for this problem?

fehiepsi · May 21, 2019, 1:31am

Hi @gdmacmillan, I have not had time to jump into this yet. Will find some time this weekend to see how to tackle this problem.

gammatam · June 19, 2019, 6:03am

Hi @fehiepsi,

I got back to the problem and tried to follow your advice. I’m first running the tutorial Gaussian Process Latent Variable Model (Gaussian Process Latent Variable Model — Pyro Tutorials 1.8.4 documentation) and then run

class Predict(gp.parameterized.Parameterized):
    def __init__(self, y_new, gplvm):
        super(Predict, self).__init__()
        self.y_new = y_new
        self.gplvm = gplvm
        self.X_new = Parameter(torch.zeros(1,2))
        
    def model(self):
        # conditional on past data
        y_new_loc, y_new_var = self.gplvm(self.X_new)        
        pyro.sample("y_new", dist.Normal(y_new_loc[:,0], y_new_var[:,0,0].sqrt()).independent(1), obs=self.y_new[:,0])
        
    def guide(self):
        pass    

## Test to get backwards (from Y to X) with any random sample from previous Y.
y_new = torch.tensor(df.values[[42],:], dtype=torch.get_default_dtype()).t()
p = Predict(y_new, gplvm)
losses = gp.util.train(p, num_steps=100)

When I run the last line, I get the error that ‘SGPR/SparseGPRegression/X’ is in guide but not in the model. I’m not sure how to solve this, since I want to optimize X_new and consider gplvm as fixed.

Would be nice if you can help me here, and also check the code above.

Here is the complete traceback:

~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/util.py:182: UserWarning: Found non-auxiliary vars in guide but not model, consider marking these infer={'is_auxiliary': True}:
{'SGPR/SparseGPRegression/X'}
  guide_vars - aux_vars - model_vars))
Traceback (most recent call last):

  File "<ipython-input-74-3b6a156082b6>", line 24, in <module>
    losses = gp.util.train(p, num_steps=100)

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/contrib/gp/util.py", line 177, in train
    loss = optimizer.step(closure)

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/torch/optim/adam.py", line 58, in step
    loss = closure()

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/contrib/gp/util.py", line 171, in closure
    loss = loss_fn(gpmodule.model, gpmodule.guide)

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/infer/trace_elbo.py", line 109, in differentiable_loss
    loss_particle, surrogate_loss_particle = self._differentiable_loss_particle(model_trace, guide_trace)

  File "~/.conda/envs/pyro/lib/python3.6/site-packages/pyro/infer/trace_mean_field_elbo.py", line 95, in _differentiable_loss_particle
    guide_site = guide_trace.nodes[name]

KeyError: 'SparseGPRegression/X'

Many thanks!

fehiepsi · June 21, 2019, 3:42am

@gammatam @gdmacmillan I make an effort at gplvm_predict.ipynb · GitHub, where a few more points are required

freeze learned parameters

for param in gplvm.parameters():
    param.requires_grad_(False)

Fix X

# get a sample and detach
#gplvm.mode = "guide"
#gplvm.X = gplvm.X.detach()
# or use the prediction mean
gplvm.X = gplvm.X_loc

Remove prior

del gplvm._priors['X']

It seems to me that the model can somehow do its job (despite that I set initial position for all of them at 0) but not as good as if we set priors.

mirumirai · November 26, 2020, 7:34pm

@fehiepsi I’m also trying to use a new y but I’m running into an error “submodule gplvm has executed outside of supermodule” when running the gist. Could you help me out?

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-37-fee319757233> in <module>
     18 
     19 y_new = y
---> 20 pred_module = Predict(y_new, gplvm)
     21 losses = gp.util.train(pred_module, num_steps=2000)
     22 # let's plot the loss curve after 2000 steps of training

<ipython-input-37-fee319757233> in __init__(self, y_new, gplvm)
      3         super(Predict, self).__init__()
      4         self.y_new = y_new
----> 5         self.gplvm = gplvm
      6         self.X_new = Parameter(torch.zeros(y_new.shape[1], gplvm.Xu.shape[1]))
      7 

C:\ProgramData\Anaconda3\envs\general\lib\site-packages\pyro\contrib\gp\parameterized.py in __setattr__(self, name, value)
    112                 self.autoguide(name, dist.Delta)
    113                 value = PyroSample(partial(_get_sample_fn, name=name))
--> 114         super().__setattr__(name, value)
    115 
    116     def autoguide(self, name, dist_constructor):

C:\ProgramData\Anaconda3\envs\general\lib\site-packages\pyro\nn\module.py in __setattr__(self, name, value)
    482             except AttributeError:
    483                 pass
--> 484             self.add_module(name, value)
    485             return
    486 

C:\ProgramData\Anaconda3\envs\general\lib\site-packages\pyro\nn\module.py in add_module(self, name, module)
    378         """
    379         if isinstance(module, PyroModule):
--> 380             module._pyro_set_supermodule(_make_name(self._pyro_name, name), self._pyro_context)
    381         super().add_module(name, module)
    382 

C:\ProgramData\Anaconda3\envs\general\lib\site-packages\pyro\nn\module.py in _pyro_set_supermodule(self, name, context)
    401         for key, value in self._modules.items():
    402             if isinstance(value, PyroModule):
--> 403                 assert not value._pyro_context.used, \
    404                     "submodule {} has executed outside of supermodule".format(name)
    405                 value._pyro_set_supermodule(_make_name(name, key), context)

AssertionError: submodule gplvm has executed outside of supermodule

fehiepsi · November 27, 2020, 6:14am

Hi @mirumirai, I just updated the gist to make it work for the recent Pyro versions. It seems to me that the above approach works as expected.

mirumirai · November 27, 2020, 11:03am

It’s working - thanks a ton!