Customize loss function

Hi :smile: I’m trying a simple example of SVI and would like to understand how I can affect it’s loss function
I’m trying make change_loss affect loss function in some way - e.g if it equals 0.3 then to make loss 30% more, if it’s -0.2 make loss value less on 20%

is there a way how I can do so? I might be do something totally wrong here :see_no_evil:, but here is my example:

import numpy as np
import torch
from torch.distributions import constraints
import pyro
import pyro.infer
import pyro.optim as optim
import pyro.distributions as dist

N = 10
mu_ = 10.
sigma_= 2.
X = np.random.normal(mu_, sigma_, N)
X = np.array(X, dtype=np.float32)

def model(X):
    change_loss = torch.FloatTensor(1).uniform_(-0.5, 0.5)
    # how to make change_loss affect elbo ?
    mu = pyro.sample("mu", dist.Uniform(torch.tensor(-25.), torch.tensor(+25.)))
    tau = torch.tensor( 1 /4)
    with pyro.plate("observed_data", size=len(X)):
        sample = pyro.sample("gaussian_data", dist.Normal(mu, 1/ torch.sqrt(tau)), obs=X)
    return sample

def guide(X):
    mean_loc = torch.randn((1))
    mean_scale = torch.tensor(0.001)
    mu_loc = pyro.param("guide_mu_mean", mean_loc)
    mu_scale = pyro.param("guide_mu_scale", mean_scale, constraint=constraints.positive)
    mu = pyro.sample("mu", dist.Normal(mu_loc, mu_scale))


adam_params = {"lr": 0.003, "betas": (0.95, 0.999)}
optimizer = optim.Adam(adam_params)

svi = pyro.infer.SVI(model=model,

losses = []
X_ = torch.tensor(X, dtype=dtype)

for t in range(10000):
    loss = svi.step(X_)
    if t%100==0:
        print (t, "\t", loss)

can you please explain your goal? because i have difficulty following your logic.
also does this help?

I am just curious how can I change the svi.step(X_) on each iterations, so it will include other function’s results (e.g. change_loss) into account? Because for example, I may have my sample = pyro.sample("gaussian_data", dist.Normal(mu, 1/ torch.sqrt(tau)), obs=X) and I may have a sample2 that will depend on sample's output at each epoch, but I need these two samples to be in the right form, so I want my loss function to include not only accuracy of first sample , but accuracy of both samples at the same time

that’s what I’m tying to do :smile:
I’m just trying things, so there mb a wrong logic :sweat_smile:

yes I saw Scaling the Loss, but not sure if this is it what I’m looking for, as I couldn’t find a proper example of usage…

i’m afraid i’m having trouble understanding exactly what you want to do. you’ll need to be more precise if you want us to be able to help you. it sounds like it might be something very “non-bayesian”…

like I’m having difficulties to understand how I can influence the expected likelihood (first term in elbo) with my custom values
here was a question which I guess similar to what I’m looking for
so there would be a possibility to use both ELBO and MSE
it would be great if it were more examples on how to do so

sorry but you will need to be much more specific about exactly what your goal is if you want to increase the probability that someone on this forum is able to answer your question or otherwise help you.

1 Like