Stein variational inference - a simple example?

Hi everyone,

I’m trying to apply SVGD to my problem but having a hard time to even get it working for a toy example (linear regression type). The comments in:
don’t really show how one would write a model specifically for SVGD. Has anyone gotten SVGD to work and could share an example model definition?

Thank you

hello, i implemented infer.svgd and never had much luck with it. this may be in part because the implemented svgd variants do not use all the latest tricks. however i think in practice it just happens to be a pretty finicky class of algorithms, given the dependence on the hard-to-set kernel hyperparameters and the possibility of hard optimization (e.g. getting stuck in bad local optima). people seem to get it to work in research papers but i suspect a lot of those results involve lots of fine tuning. it’s also possible there’s a bug in the implementation but that seems somewhat unlikely given the relatively simple nature of the implementation. unfortunately i haven’t any particularly useful advice for getting svgd to work outside of toy problems.

Regarding failure modes, see Section 4 of this technical report I wrote.

Precedent bad examples also exist but were not really highlighted so well.
A potential solution is annealing:

1 Like