Difference `pyro.param` vs `pyro.sample`

Misc.
1

I wonder what is the difference between pyro.sample and pyro.param when declaring variables in my probabilistic model or the guide? E.g.,

def model(data):
    x = pyro.param("x", torch.zeros(A, B))
    ...

vs

def model(data):
    x = pyro.sample("x", dist.Normal(torch.zeros(A), torch.ones(B)))
    ...

If I understand it correctly, during SVI the variable s will be “learned” in both models. Moreover, I can use x to declare/specify other variables in my model. So the difference is not clear to me.

Thanks in advance!

2

parameters corresponds to point estimates and sample statements correspond to latent variables. latent variables are governed by distributions and inference yields a distribution over the inferred values of the latent variable (not a point estimate).

in other words sample statements correspond to quantities with associated uncertainty while parameters have no uncertainty about them.

1 Like
3

Thanks for the quick reply.
To clarify: assume I want to estimate x_i = A * z_i, where x is my input of dimension N x F (samples x features).
Hence, with latent dimension K:

  • A should be a parameter, because it is fixed across samples:
    A = pyro.param("A", torch.ones(F, K))

  • z_i is a latent variable and therefore needs to be sampled:
    z = pyro.sample("z", dist.Normal(torch.zeros(K,1), torch.ones(K,1)))

Is that correct?

4

there is no notion of correctness here. there are all kinds of modeling assumptions you could make: A could be a parameter or it could be a latent variable, etc

1 Like
5

Thanks, the original question is answered!

6

I am still confused.

You can define a latent variable in the model (i.e. using a pyro.sample statement) and using a AutoDelta guide. Inference will give you a single point estimate for the latent variables in your model.

So I don’t think it is correct to say that:

in other words sample statements correspond to quantities with associated uncertainty while parameters have no uncertainty about them.

I think the correct statement is that:

In the model:

  1. pyro.sample is for quantities that have a prior.
  2. pyro.params is for quantities with no prior.

The difference is that the inference procedure will optimize different quantities.

In the guide:

  1. pyro.sample is for all quantities that have a pyro.sample in the model (excluding the observed ones). Possibly including a sampling from a Delta distribution if you only want a point estimate.
  2. pyro.param are for all the rest.

See MLE and MAP Estimation — Pyro Tutorials 1.8.4 documentation for an example where this is tangentially discussed.

7

when i wrote the below i was implicitly assuming the user is doing variational inference (e.g. with an AutoNormal guide) and not something like MLE or MAP

parameters corresponds to point estimates and sample statements correspond to latent variables. latent variables are governed by distributions and inference yields a distribution over the inferred values of the latent variable (not a point estimate).

in other words sample statements correspond to quantities with associated uncertainty while parameters have no uncertainty about them.

2 Likes