clearly the lengthscale is much smaller than 100 here. Does SVI change the values of the lengthscale, even though I have specified it, with no prior? It seems like I am doing something wrong here.
I even tried putting a uniform prior on the lengthscale with a lower bound of 100, and I get a similar result:
ls = pyro.sample('ls', Uniform(torch.DoubleTensor([100.]).to(device), torch.DoubleTensor([200]).to(device)))
OK, that makes sense. Shouldn’t a uniform prior with a large lower bound also prevent the lengthscale from getting small, though? I’ve tried all sorts of strong priors on the lengthscale, but they don’t seem to do any good.
ls = pyro.sample('ls', Gamma(torch.DoubleTensor([20.]).to(device), torch.DoubleTensor([0.5]).to(device)))
amp = pyro.sample('amp', Gamma(torch.DoubleTensor([2.]).to(device), torch.DoubleTensor([0.5]).to(device)))
K = gp.kernels.RBF(
input_dim=1,
variance=amp,
lengthscale=ls
)
K.lengthscale_unconstrained.requires_grad_(False)
K.variance_unconstrained.requires_grad_(False)
It was not clear how to use PyroSample in the context of how I am using the GP (as a latent variable rather than in a GPRegression or a custom model class. This clearly does not work:
ls = pyro.nn.PyroSample(Gamma(torch.DoubleTensor([20.]).to(device), torch.DoubleTensor([0.5]).to(device)))
amp = pyro.nn.PyroSample(Gamma(torch.DoubleTensor([2.]).to(device), torch.DoubleTensor([0.5]).to(device)))
K = gp.kernels.RBF(
input_dim=1,
variance=amp,
lengthscale=ls
)
@fonnesbeck I think this notebook might be helpful for you. It shows how to set priors, set autoguide, add random effects, inspect parameters, customize optimizers… for a gp model.
It is hard to guess what’s wrong without some code to replicate the error. I think you might need to use pyro_method like GPR model. You can also find an explanation for it in the above notebook.
I’m going to try an emulate the regression example and use PyroModule. Its not clear what needs to be defined in __init__ as PyroSample and what is in forward using pyro.sample. I will try all of the GP parameters in the former and the non-GP parameters in the latter.
My model is a little unique because I am using GPs as latent functions with MultivariateNormal variables, and combining them as an expected value for a likelihood, rather than GPRegression, which is only a marginal GP. So, I have several GPs that I am combining.
I think I discoverd the cause of this error: I have multiple kernels in my model (since I have multiple latent GPs). If I comment out the other kernels, it runs fine. Strange that this occurs even though I am using the K. namespace (the other kernel is K_w).
So, any time you have two kernels in a model, it seems like this issue will come up. There appears to be a namespace issue here, as the parameters of K1 and K2 should be distinct.
I see. I think all you need to do is to make your model a pyro module, with submodules K1, K2. Like PyTorch nn Module, parameter names of those submodules will be “K1.lengthscale”, “K2.lengthscale”. We designed Pyro module as a subclass of PyTorch nn module, so constructing, naming,… are very similar to PyTorch ways.
I did manage to get the model converted to a PyroModule, but it appears to run about 10x slower. Is there known overhead for using a module? The only change I made to how the model is run was to add a retain_graph=True argument to Trace_ELBO to avoid the RuntimeError I was getting without it. Would that be responsible for a slowdown?
I think so, some results from google search mentioned that in pytorch, using retain_graph=True is ways slower than the default option. I guess it is better to try to resolve the RuntimeError.
Is there known overhead for using a module
When I converted the old gp modules to PyroModules, I didn’t observe such regression (if I remember correctly).
Yeah, its not obvious why the RuntimeError is occurring, as I have just re-expressed my existing model (which does not raise an error) as a PyroModule. The running logic is straight from the docs:
N_STEPS = 1000
model = VenueModel(days, venue_list)
guide = autoguides.AutoDiagonalNormal(model)
optimizer = pyro.optim.Adam({"lr": 0.01})
svi = SVI(model, guide, optimizer, loss=Trace_ELBO())
pyro.clear_param_store()
for j in range(N_STEPS):
loss = svi.step(X, y)
if not j % 10:
print("[iteration %04d] loss: %.4f" % (j + 1, loss / len(y)))
The failure occurs at the call to step:
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
145 Variable._execution_engine.run_backward(
146 tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 147 allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
148
149
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.
and the only way I can get around it is to set retain_graph=True.
Well, I tried to make a runnable simple example that fails, but cannot. However, I notice that if I build up my model incrementally, its when I add the following GP components to the __init__ and forward methods that results in the error:
So again, there is much more to the model than this, but removing these components allows the model to run without error. Does anything jump out at you as being problematic?
