So imagine I am trying to use a Uniform distribution (
sigma) in my guide, and this uniform distribution should be bound by [low, high].
sigma_low = pyro.param('sigma_low',lambda: torch.tensor(0.),constraint=constraints.interval(0,1))
sigma_high = pyro.param('sigma_high',lambda: torch.tensor(1.),constraint=constraints.interval(0,1))
sigma = pyro.sample('sigma',dist.Uniform(sigma_low,sigma_high).expand([n]).to_event(1))
But I often get errors saying the sigma_low can not be higher than sigma_high, which makes sense. But I am wondering how I can add this further constraint in the guide.
Would be appreciated some hints here.
Thanks a lot,
high have to be bounded by 0 and 1? If not you can try ordered_vector.
Thanks so much for the reply and I am sorry for not being able to understand it immediately. Would you mind sharing a simple example of how to utilize the ordered_vector?
For me, I am hoping to model
sigma as a uniform distribution bounded by 0 and 1, so I am trying to use the
sigma_high to achieve that. Are there better ways to achieve what I am looking for in the guide function?
If you want to just have
sigma variable from a Uniform distribution bounded by 0 and 1 then you can just have:
sigma = pyro.sample('sigma', dist.Uniform(0, 1))
Unless I’m missing something.
Sorry for the confusion, I meant to say,
sigma is a variational distribution in the guide function, defined by two parameters, which are its lower bound and higher bound. And through variational inference, we are learning these two parameters. However, this
sigma has to be within [0,1], so I am trying to figure out how I can enforce this constraint in my guide function.
you can just use elementary transformations. e.g. let
x=high/low then put a
(0,1) constraint on
delta and a
(1, \infty) constraint on
x. then you can solve for
low and get
high=delta * x/(x-1). alternatively you can define
y=x-1 and put a positivity constraint on
y and write
high=delta * (y+1)/y
Thanks so much for the instructions, I guess the actual code for your suggestion should be like below:
delta = pyro.param('delta',lambda:torch.tensor(0.),constrain=constrains.interval(0,1))
x = pyro.param('x',lambda:torch.tensor(1.),constrain=constrains.interval(1,float('inf')))
low = 1/(x-1)*delta
high = x/(x-1)*delta
sigma = pyro.sample('sigma',dist.Uniform(low,high))
But I wonder how it can guarantee that both
high are within [0,1] itself? For example, if high=1.2, low=0.5, it should comply with the constrain defined above but the
high can not be 1.2 right?
Very appreciated for your help here,
whoops sorry i meant the following; the point is you can use elementary transformations and elementary constraints to achieve this:
low = x * (1 - d)
high = low + d
btw it may not be a great idea to have a guide like this because variational inference may not be well defined if the support of the model and guide differ at a particular sample site (here presumably
sigma). it might be better to e.g. use a
Beta distribution which assigns mass to all of the interval
(0, 1) but also allows concentration near any point of that interval.
Thanks again for the prompt response!
I agree I think beta distribution might be a better idea here given
sigma must be within [0,1], and I probably should use it instead. But just go back to this uniform example for a second, it seems that d (0,1) and x (0,1) still can not exactly achieve the goal, doesn’t it?
Say high=0.8, and low=0.4, which are two valid bound values, then x=high/low=2, which violates the constraint for x?
Sorry for keeping bugging you but I am just hoping to get a better understanding of it.
i changed the definition of
x for 1000 pairs of values satisfying unit interval constraints and compute
high and you’ll see
for _ in range(10**4):
d = torch.rand(1).item()
x = torch.rand(1).item()
l = x * (1 - d)
h = l + d
assert l > 0.0 and l < 1.0
assert h > 0.0 and h < 1.0
assert l < h
Thanks so much for the code and clarifications!