Can someone explain what kind of models would require the optimizer to be dynamic? And why is this different for pyro (as opposed to pytorch)?
My guess was that this is because parameters in pyro are your latent variables, and so given data, you may have more latent variables if you are working with a plate model. But its just a guess.
pyro doesn’t know a parameter exists until it hits a param statement. if the param statement is in an if/else block that is stochastic because the branching depends on a random variable (i.e. a sample statement) then pyro may not encounter that random variable until its run the model 4792 times. at which point it can create an optimizer for the newly encountered parameter. in other words pyro doesn’t know what parameters it will encounter upfront before running the program.
I understand your statement, but I cannot quite understand why this is specific to pyro and not a basic pytorch feature: Any model including conditionals will presumably have the same problem, and pyro could just initialize all possible variables (or do you have a good example of where this is not feasible?)
this isn’t so much specific to pyro as specific to handling a broad class of models, including those with stochastic control flow, possibly without countably infinite parameters. in order to initialize all possible variables pyro would need to do static analysis of python code, not an easy task.
Got it. Do you have a tutorial or example project doing something like this (potentially infinite parameters)? Having a bit of trouble imagining the model that looks like this.