Using the following simple model, I get quite a bit of variance in the PSIS Diagnostic.
import numpy as np
from torch import tensor
from pyro import distributions as dist
from pyro import sample, factor, plate
from pyro.optim import Adam
from pyro.infer import autoguide, SVI, Trace_ELBO, TraceMeanField_ELBO
from pyro.infer.autoguide import init_to_median
from pyro.infer.importance import psis_diagnostic
n = 1000
mu = 3.14
sigma = 2.7
y = tensor(np.random.normal(mu, sigma, n)).float()
def model(y):
mu = sample("mu", dist.Normal(0, 1))
sigma = sample("sigma", dist.Gamma(1, 1))
with plate("data"):
y = sample('y', dist.Normal(mu, sigma), obs=y)
optimizer = Adam({"lr": 5e-2})
guide = autoguide.AutoDiagonalNormal(model, init_loc_fn=init_to_median())
svi = SVI(model, guide, optimizer, loss=TraceMeanField_ELBO())
for i in range(5000):
if i % 100 == 0:
print(i)
svi.step(y)
psis_diagnostic(model, guide, y, num_particles=1000)
Repeated several times gives
In [102]: psis_diagnostic(model, guide, y, num_particles=1000)
Out[102]: 0.12475437759230361
In [103]: psis_diagnostic(model, guide, y, num_particles=1000)
Out[103]: 0.03406547836026321
In [104]: psis_diagnostic(model, guide, y, num_particles=1000)
Out[104]: 0.15812618171283743
In [105]: psis_diagnostic(model, guide, y, num_particles=1000)
Out[105]: 0.24965923846260274
In [106]: psis_diagnostic(model, guide, y, num_particles=1000)
Out[106]: 0.08239155431661215
In [107]: psis_diagnostic(model, guide, y, num_particles=1000)
Out[107]: -0.13488150137448549
In [108]: psis_diagnostic(model, guide, y, num_particles=1000)
Out[108]: -0.16388070392834722
In [109]: psis_diagnostic(model, guide, y, num_particles=1000)
Out[109]: -0.0156825685265973
Is this expected behavior or a limitation of using PSIS?