Hi everyone, I was wondering if this is possible in NumPyro:
If I’m sampling from a model in which I already know a mean + standard deviation for some of the parameters. Can I fix those parameters to a Normal(mean, standard deviation) when sampling, rather than setting those parameters as a prior?
I’m working with very high dimensional ODE models so it would be wonderful if I can reduce the dimensionality somewhat without resorting to fixing these known parameters as point estimates.
Thank you!
not if i understand your question correctly. if you want to take uncertainty of a given latent variable into account you need to include it in your model and have your inference algorithm (e.g. HMC) explicitly consider it. you can lower the dimension of the latent space by fixing certain latent variables to point estimates but doing so will of course mean you’re not taking any uncertainty into account in those particular latent variables.
Hi Martin,
Thanks for your reply!
I think you mostly understood my question correctly. I’m imagining a situation where one might know a parameter point estimate and corresponding uncertainty experimentally.
So, for instance, if I have a model with 100 unknown parameters, and I already know that 20 of these parameters are normal(0.1, 1.0), or whatever, wouldn’t it be much more efficient to fix these distributions in my model so that I’ll sample from them with MCMC but not have to random walk these distributions as well? Especially if the data set I’m using doesn’t contain much information about these parameters.
Or would you in this case still suggest I include these parameters as priors and let Bayes figure it out? Of course I could also include them as known point estimates but that would disregard the uncertainty associated with them.
The problem is I’m working with high dimensional ODE models and would like get a posterior in finite time and without too many identifiability issues etc.
i think you may misunderstand how MCMC works. regardless of what prior you choose for a given latent variable (whether it’s something simple like a Normal distribution or something more complex), the way MCMC generates samples from the latent space is by moving around that latent space. you can’t move around (i.e. explore uncertainty in) a given latent variable unless it’s part of your latent space.
Hi Martin,
Thanks for your comment. You may be right, I find it difficult to picture how MCMC would deal with such a parameter.
So would you suggest I include such background knowledge as a prior then? I’m guessing that would be the most correct approach but it may not be feasible given the number of parameters in my model.
Alternatively I could perhaps fix a parameter that I already know the distribution of to its mean during MCMC but then also sample from it when running posterior predictive simulations (next to the posterior identified by MCMC which then does not include that specific parameter). That may significantly bias my results though.
What do you think?
including it into the prior is generally the better approach.
Alternatively I could perhaps fix a parameter that I already know the distribution of to its mean during MCMC but then also sample from it when running posterior predictive simulations (next to the posterior identified by MCMC which then does not include that specific parameter). That may significantly bias my results though.
yes you could also do something like this although it’s a bit of a “hack”. still it might work well in practice if all you care about is e.g. predictions
Thanks! That’s good to know. I think i’ll experiment with both and report back here 
Thanks for the idea! I didn’t yet find a clear example of how the HMCGibbs work but the documentation seems to suggest to me that I can basically pass samples for some of the parameters to the NUTS optimizer.
Is it necessary that these samples are obtained by Gibbs sampling?
Specifically, I have an ODE system that also contains a few algebraic equations (so its actually a DAE system). To me, it seems like a reasonable (and efficient) approach to sample the parameters in these algebraic equations for each equation independently (as a simple regression), and then pass the corresponding posteriors to the ODE system optimization.
Another reason for doing it like this is that I don’t have much data for some of the variables. So the sampling of some of the algebraic (regression) equations will be based on, for instance, only N=50, while for the ODE variables I have N=500.
I’m definitely also curious to see how well the sampler would work if I pass the posteriors of the parameters in the algebraic equations as a prior to the sampler of the ODE system.
Assume that you have a generative model p(x, y, data) where x and y are latent variables and you can further assume that p(x | data) = p(x | y, data) (i.e. conditioned on data, x does not depend on y). If you know what p(x | data) is (I assumed this from your comment: “I already know that 20 of these parameters are normal(0.1, 1.0)”), then you can use HMCGibbs with gibbs_fn will generate a new x from p(x | data). At each MCMC step, after having a new x, HMC will give you a new sample y from p(y | x, data). The pattern will be
def gibbs_fn(rng_key, gibbs_sites, hmc_sites):
return {"x": dist.Normal(0.1, 1.0).sample(rng_key, sample_shape=(20,))}
kernel = HMCGibbs(NUTS(model), gibbs_fn=gibbs_fn, gibbs_sites=['x'])
Ah this is very interesting, thanks a lot! I’m implementing the steps iteratively and i’ll definitely try this somewhere in hopefully the coming week.