Stein variational inference - a simple example?

Misc.
1

Hi everyone,

I’m trying to apply SVGD to my problem but having a hard time to even get it working for a toy example (linear regression type). The comments in:
https://docs.pyro.ai/en/1.6.0/inference_algos.html#pyro.infer.svgd.SVGD
don’t really show how one would write a model specifically for SVGD. Has anyone gotten SVGD to work and could share an example model definition?

Thank you

2

hello, i implemented infer.svgd and never had much luck with it. this may be in part because the implemented svgd variants do not use all the latest tricks. however i think in practice it just happens to be a pretty finicky class of algorithms, given the dependence on the hard-to-set kernel hyperparameters and the possibility of hard optimization (e.g. getting stuck in bad local optima). people seem to get it to work in research papers but i suspect a lot of those results involve lots of fine tuning. it’s also possible there’s a bug in the implementation but that seems somewhat unlikely given the relatively simple nature of the implementation. unfortunately i haven’t any particularly useful advice for getting svgd to work outside of toy problems.

3

Regarding failure modes, see Section 4 of this technical report I wrote.

Precedent bad examples also exist but were not really highlighted so well.
A potential solution is annealing:
https://openreview.net/pdf?id=pw2v8HFJIYg

1 Like