I am trying to implement a custom loss function with has two different parts, but I am not sure how to put them together.
→ The first part of the loss function is the ELBO, for which I want to use loss = pyro.infer.Trace_ELBO().
→ The second part is a l2 regularizer described as:
def L2_regularizer(my_parameters, lam=torch.tensor(1.)):
reg_loss = 0.0
for param in my_parameters:
reg_loss = reg_loss + param.pow(2.0).sum()
return lam*reg_loss
@fritzo thanks for the quick reply. I have three follow up questions.
Shouldn’t the pyro.factor("regularizer", -L2_regularizer(x_loc)) statement be inside the guide instead of the model? I am saying this because -L2_regularizer(x_loc) inside the model will always return zero (since x_loc = torch.zeros(N*3,)).
If I want to add another part to this loss, let say a loss function L3_func(...), can I simply use the pyro.factor as pyro.factor("regularizer2", -L3_func(...))?
I am having some trouble trying to define a custom objective function. Based on the custom objective function documentation, I understand that I can define the total loss as:
loss_fn = pyro.infer.Trace_ELBO().differentiable_loss loss = loss_fn(model, guide) + L2_regularizer(my_parameters).
What I am not sure is where/how to define my_parameters and how to pass the data through the model and guide. For my case, in the guide, I have two sets of parameters, namely, x_loc and x_scale. Do I define them outside the guide? as follows,
Gosh you’re right, my model regularizes the wrong x_loc . I guess this is a little trickier in the guide. I think we’ll need to negate it (again, since the loss is \mathbb E_q [\log q - \log p] and we’re moving the factor from the model p to the guide q) and specify has_rsample=True. Does this work for you?
# optimizer
optimizer = torch.optim.Adam(my_parameters, {"lr": 0.001, "betas": (0.90, 0.999)})
for i in range(num_steps):
loss = loss_fn(data=data_obs)
loss.backward()
optimizer.step()
optimizer.zero_grad()
What I am confused about is how to get the my_parameters in optimizer defined above? For my case, the parameters are defined in the guide x_loc and x_scale.