Thanks! That makes sense. I based my code on an example I found, and found it strange since indeed I didn’t see the parameter definitions in the tutorial model functions.
In the new model function I also added sigma as a sample (with corresponding parameters in the guide function such that it would also do Bayesian regression on those.
Then the stds on parameters using SVI are still smaller, but based on my current understanding SVI doesn’t sample the true posterior, just approximates it, and perhaps that’s the issue here. MCMC with NUTS matches just with the result from Least-Squares.
If I don’t set sigma as a distribution (i.e. fix the modelled noise on the data), MCMC fails completely to find a good solution.
def model(x,y,guess):
ωμ = torch.tensor(guess[0])
ωσ = torch.tensor(0.2)
ω = pyro.sample("ω", dist.Normal(ωμ, ωσ))
ϕμ = torch.tensor(guess[1])
ϕσ = torch.tensor(1.)
ϕ = pyro.sample("ϕ", dist.Normal(ϕμ,ϕσ))
bμ = torch.tensor(guess[2])
bσ = torch.tensor(0.2)
b = pyro.sample("b", dist.Normal(bμ, bσ))
values = torch.sin(ω*x+ϕ)+b
sigma_μ = torch.tensor(1.)
sigma_σ = torch.tensor(0.1)
sigma = pyro.sample("sigma", dist.Normal(sigma_μ, sigma_σ))
with pyro.plate("data", len(y)):
pyro.sample("obs", dist.Normal(0, sigma), obs = values-y)