Mini batching & Renyi_ELBO not supported?

Hi!

First, thanks for your attention :). I would like to confirm if Renyi_ELBO is certainly not working with mini_batches, since I have a model that works with mini_batches and it did not throw an error/warning when I plugged it in. So…the conclusion would be that is just learning from batches separately?So at the end of the training I would get the results only for the last batch?
Or has it been upgraded and it can handle mini_batches now?
Thanks again :slight_smile:

@artistworking IIUC, Renyi_ELBO works with mini_batches as long as you scale your likelihood properly, for example

def model(batch_x, batch_y):
    sigma = pyro.sample("sigma", dist.Exponential(1))
    with pyro.poutine.scale(scale=num_full_data / batch_size):
        return pyro.sample("obs", dist.Normal(batch_x, sigma), obs=batch_y)

For reference, you can take a look at formula (9) of https://arxiv.org/pdf/1602.02311.pdf. IMO, that warning is not necessary. :slight_smile:

1 Like

Thank you very much for your reply :slight_smile: So just to double check, if we have quite a large dataset, say 50000 data points, and our mini_batches are of size 200, then, following the formula we will multiply the ‘subset average likelihood’ by 250?
Or there is some further reduction to that number within the poutine.scale?
Is that a reasonable number?
It seems high, but honestly I do not know. Thanks !!! :slight_smile:

Yes, it is a reasonable number and it is advised to scale likelihood whenever you use subsampling with global random variables. You can check this tutorial for more information about global/local latent variables and when to scale/when not. In either case, I think that Renyi_ELBO will work just like Trace_ELBO.

1 Like

Thank you very much for the info and help. This has made me realize we have to pay more attention to the likelihood scaling. Thanks and have a nice day!! :slight_smile:

1 Like