How to enforce one parameter is larger than another?

Misc.
1

Hello,

So imagine I am trying to use a Uniform distribution (sigma) in my guide, and this uniform distribution should be bound by [low, high].

sigma_low = pyro.param('sigma_low',lambda: torch.tensor(0.),constraint=constraints.interval(0,1))
sigma_high = pyro.param('sigma_high',lambda: torch.tensor(1.),constraint=constraints.interval(0,1))
sigma = pyro.sample('sigma',dist.Uniform(sigma_low,sigma_high).expand([n]).to_event(1))

But I often get errors saying the sigma_low can not be higher than sigma_high, which makes sense. But I am wondering how I can add this further constraint in the guide.

Would be appreciated some hints here.

Thanks a lot,
Frank

2

Hi. Do low and high have to be bounded by 0 and 1? If not you can try ordered_vector.

3

Hi @ordabayev,

Thanks so much for the reply and I am sorry for not being able to understand it immediately. Would you mind sharing a simple example of how to utilize the ordered_vector?

For me, I am hoping to model sigma as a uniform distribution bounded by 0 and 1, so I am trying to use the sigma_low and sigma_high to achieve that. Are there better ways to achieve what I am looking for in the guide function?

Thanks again,
Frank

4

If you want to just have sigma variable from a Uniform distribution bounded by 0 and 1 then you can just have:

sigma = pyro.sample('sigma', dist.Uniform(0, 1))

Unless I’m missing something.

5

Hi @ordabayev,

Sorry for the confusion, I meant to say, sigma is a variational distribution in the guide function, defined by two parameters, which are its lower bound and higher bound. And through variational inference, we are learning these two parameters. However, this sigma has to be within [0,1], so I am trying to figure out how I can enforce this constraint in my guide function.

Thanks again,
Frank

6

you can just use elementary transformations. e.g. let delta=high-low and x=high/low then put a (0,1) constraint on delta and a (1, \infty) constraint on x. then you can solve for high and low and get low=delta/(x-1) and high=delta * x/(x-1). alternatively you can define y=x-1 and put a positivity constraint on y and write low=delta/y and high=delta * (y+1)/y

7

Hi @martinjankowiak,

Thanks so much for the instructions, I guess the actual code for your suggestion should be like below:

delta = pyro.param('delta',lambda:torch.tensor(0.),constrain=constrains.interval(0,1))
x = pyro.param('x',lambda:torch.tensor(1.),constrain=constrains.interval(1,float('inf')))
low = 1/(x-1)*delta
high = x/(x-1)*delta
sigma = pyro.sample('sigma',dist.Uniform(low,high))

But I wonder how it can guarantee that both low and high are within [0,1] itself? For example, if high=1.2, low=0.5, it should comply with the constrain defined above but the high can not be 1.2 right?

Very appreciated for your help here,
Frank

8

whoops sorry i meant the following; the point is you can use elementary transformations and elementary constraints to achieve this:

  • d in (0, 1)
  • x in (0, 1)
  • low = x * (1 - d)
  • high = low + d

btw it may not be a great idea to have a guide like this because variational inference may not be well defined if the support of the model and guide differ at a particular sample site (here presumably sigma). it might be better to e.g. use a Beta distribution which assigns mass to all of the interval (0, 1) but also allows concentration near any point of that interval.

9

Hi @martinjankowiak,

Thanks again for the prompt response!

I agree I think beta distribution might be a better idea here given sigma must be within [0,1], and I probably should use it instead. But just go back to this uniform example for a second, it seems that d (0,1) and x (0,1) still can not exactly achieve the goal, doesn’t it?

Say high=0.8, and low=0.4, which are two valid bound values, then x=high/low=2, which violates the constraint for x?

Sorry for keeping bugging you but I am just hoping to get a better understanding of it.

Many thanks
Frank

10

i changed the definition of x and d. choose d and x for 1000 pairs of values satisfying unit interval constraints and compute low and high and you’ll see

1 Like
11 
import torch

for _ in range(10**4):
    d = torch.rand(1).item()
    x = torch.rand(1).item()
    l = x * (1 - d)
    h = l + d
    assert l > 0.0 and l < 1.0
    assert h > 0.0 and h < 1.0
    assert l < h
1 Like
12

Thanks so much for the code and clarifications!