I have a SVI model that seems quite prone to finding local minima. I’ve taken to re-running the model several times with different seeds to understand the range of possible errors and to choose between possible results.
AutoDelta to create my guide function, and this is an issue as the first svi step (using JIT) can take a few minutes. This means if I run a reasonable number of seeds, I can be spending an hour or more just on the first svi step of each model.
What I’m wondering is if there is a way to snap the state of the model after the first SVI step, and then re-randomize the parameters, avoiding the JIT recompiles. With the re-initialized params I’m hoping to run just my SVI again to get another elbo trace, and throw that result in my bucket of possibilities.
Ideally I’d use the carefully chosen
init_loc_fn s to regenerate starting tensors, just with a new rng seed. By stumbling around I can eventually run something that looks like:
guide.parts._init_loc_fn(site) # guide is an AutoGuideList
but it feels like there must be a more straightforward way. My main question is:
- How can I reinitialize existing parameters using the init_loc_fn?
A few other noob questions:
Is there a better way to get the sample sites than running a big trace/block combo every time, a la
with poutine.trace() as sample_capture: # We use block here to allow tracing to record parameters only. with poutine.block(expose_fn=lambda msg: msg["type"] == "sample"): guide(inputs)
Is manually adjusting the dict elements of
pstore.get_state()a reasonable way to override parameters? (I saw calling
pyro.param()before the guide makes them is best, but what if they are already in the store?)
I can’t quite get a clear picture of the .
unconstrained() aspect when it comes to params / samples.
- It seems that the .
unconstrained() output tensor is the actual site that everything uses as key, is that correct?
- If I want to override values in the tensor, does that mean that I have to set values in the unconstrained space? How do I map values in the constrained space into that space?
- What’s the general practice for how and when to use .
- It seems that the .
Thanks for all the work on such a great environment!