Stochastic Block Model using Pyro

#1

Hi guys!

Might be a longer post, but I want to be specific :slight_smile:

I’m currently doing a research internship at the Technical University Munich (TUM) and working on Stochastic Block Models (SBMs). Since I am new to the topic of inference, I followed an example SBM implementation in the probabilistic framework edward1 (see here). This succeeded and was a nice way to start getting familiar with probabilistic modeling itself. However, Pyro seems to be more flexible and generally more user friendly for our purposes as to why we want to implement the SBM using Pyro.

However, I ran into some issues regarding parameter concentration values, dimensions, etc. and am having a difficult time debugging. In principle, I used the Pyro documentation to get familiar with models and guides and in particular this example on SVI.

I basically tried porting my SBM implementation of the famous karate club data set from edward to Pyro. To make things/info more accessible to you, have a look at my code from my GitHub directory (using Python 3.7 as base interpreter):

-> My probabilistic SBM approach (formulas) can be found here
-> My initial SBM implementation using edward can be found here
-> My attempt at a Pyro version can be found here.
Note, I have defined two guides guide_1 and guide_2. The first one matches more or less what I have found in the examples, the second one resembles more the solution based on edward.

As far as I have understood:

  • The model defines the general probabilistic structure and assumed random variable distribution
  • The guide defines the variational parameters, initializes them and prepares them for inference
  • SVI is then chosen as the inference algorithm

My questions:

  1. Have I correctly defined the model and guide?
  2. Have I correctly addressed the observed data and connected it to the model adjacency matrix? Is this the way how to do it?
  3. I tried debugging the code, however I run into value and dimension issues. Anyone see a fix?
  4. Optional: Can the SBM be defined more easily using a pyro.plate structure?

I am very curious and seem to understand more every day. I would greatly appreciate any help!

Thank you :slight_smile:

#2

Whoa, I am especially interested in SBM and graphon/graphing in general and also would like to learn and apply my knowledge to this field. @aladinD I am happy to take a look this weekend if this thread is still alive until then.

If anyone else has inputs for this, please jump in. This should be a cool topic to discuss. :slight_smile:

Edit: this reference is also relevant

2 Likes
#3

Yes, this would be great!

There is a greater use case for SBMs and it would be optimal to solve most of the problem sets using Pyro :wink:

#4

@aladinD Just looking at your implementation, here are some suggestions

Have I correctly defined the model and guide?

For the model, there are a few things missing:

  • You need to declare the event/batch dimensions of your priors. You can read tensor shapes tutorial for more information. If you unsure what to do, just simply call .to_event() for each prior. For example
Eta = pyro.sample("Eta", dist.Beta(0, 1).expand([K, K]).to_event())
  • It seems that you missed declaring shape (N,) for Z. You can use dist.Multinomial(...).expand([N]).to_event() if not sure what to do.

For the guides:

  • In the model, Eta has the shape K, K while in guide1 Eta has the shape K.
  • In the model, pi has the shape K while in guide1 pi has the shape K, K.
  • The categorical distribution in the guide1 has different support w.r.t. to the Multinomial distribution in the model.
  • All priors in model2 are not distributions. I guess you wanted to use dist.Delta(torch.nn.Sigmoid(eta_sigmoid_q)) at those places?
  • In model2, the support of Z is a vector of integers, while the output of softmax is a simplex. I am not sure what you wanted to do here. In the case of K=2, I guess you want to round the outputs of softmax to integers? (e.g. (0.3, 0.7) -> (0, 1)?)

Have I correctly addressed the observed data and connected it to the model adjacency matrix? Is this the way how to do it?

The svi.step(data) function needs to have the same signature as the definition of model/guide. So you need to provide data, nodes, K there (though I am not sure what nodes do in your model). In addition,

SVI(model=SBM_Model(data, nodes, K),
    guide=SBM_Guide_1(data, nodes, K),...)

should be

SVI(model=SBM_Model, guide=SBM_Guide_1,...)

I tried debugging the code, however I run into value and dimension issues. Anyone see a fix?

There are many things that are misspecified in your implementation, as pointed out above. I suggest working with the SVI tutorials again will be very helpful for you to understand how to do SVI inference in Pyro.

Can the SBM be defined more easily using a pyro.plate structure?

If you want to use it, you can do something like

with plate("N", N):
    Z = pyro.sample("Z", dist.Multinomial(total_count=1, probs=pi))

but it seems to me that plate is not needed for your model. Using .to_event() is enough in my opinion.

#5

@fehiepsi Thank you very much for your reply and sorry for this rather late feedback!

I have clearly misunderstood some of Pyro’s fundamentals (model and notation wise) and am now working on an updated version of my SBM. I will post it shortly, so we can review it together! :smiley:

1 Like