Do we have some solution similar to one of these
- spike-and-slab
- horseshoe prior
Hi @edwinng, by solution, do you need some concrete examples or just want to know if it is possible to do it? Pyro/NumPyro supports models with discrete latent variables, so spike-and-slab models should be easy to implement and do inference with. For horseshoe prior, you can see some hierarchical versions of it in this SVI example or this MCMC example.
Edit: I added a feature request for spike-and-slab models.
@fehiepsi i am just asking for a possible solution of sparse regression. It seems the scripts you provided will be sufficient to start with for me. I will take a look on them first. Of course, if there are more concrete tutorials, it is even better!
Edwin
I’m reading through the github thread on the feature request. Do I understand correctly that a spike-and-slab prior is possible to implement in Pyro, but not yet in Numpyro? If so, I’d definitely try this in Pyro.
the “feature request” in numpyro isn’t for a feature per se; it’s for a tutorial dedicated to the topic.
a spike-and-slab prior should be implementable in both pyro and numpyro.
note, however, that spike-and-slab priors are not the only sparsity-inducing priors. both pyro and numpyro have examples that use a horseshoe-like prior to induce sparsity in linear regression coefficients.
Thank you — I’m experimenting with pyro. In the particular niche of the literature I’m in right now, these mixture priors are standard. However, I think that some of these continuous shrinkage priors may be easier to implement. I’ll start there.
@vasa If you want to try spike-and-slab, then using numpyro is better because it will be faster. The code at that issue should work with the current state of this PR. But if the number of variables d is large (say > 20), running time will be slow because the enumeration size grows like 2^d for that model. Using continuous versions would be better in that case, until we have an inference algorithm that does not require enumeration to deal with discrete latent variable (e.g. metropolis within gibbs)
Thank you for the recommendation @fehiepsi. I’d really love to try out numpyro, but it won’t be for this model. I’ve done some tests, and it looks like I’ll pursue the continuous shrinkage priors in pyro for now. I have many discrete parameters in this model. The classic models I am seeing implement tailored Gibbs samplers that allow them to handle this kind of structure efficiently (well, I assume they do).
So far the finnish horseshoe is kind of working. I’m having some issues with stability during SVI, and there is more tuning to be done here.
Sure, it is better to not use enumeration in your case. For continuous versions, either svi or mcmc should work with both pyro and numpyro. If speed is not a problem, then just go with your favorite backend.
1 Like
It’s a large and strange model in statistical genetics. Essentially the famous eight schools example, just on steroids. I’m refining ~750K correlated parameter estimates. People have worked out some nice likelihood and prior functions in the literature and I thought it would be interesting to see a PPL implementation. I chose Pyro for the sophisticated SVI framework and it’s a proven a powerful tool set.