How to integrate pyro model to c++

Misc.
1

Is there any c++ API in pyro? There is a model (several .p files) trained in pyro. I want to use it for prediction in c++. What should I do?

Thanks

2

Hi @sj2019, great question!

tl;dr torch.jit.save your trained model from Python, then torch::jit::load() it into C++.

My team is also exploring ways to serve Pyro models in C++. While I haven’t used the torch.jit.save() ; torch::jit::load() path myself, I’ve used torch.jit.trace() to speed up models. If you do this, I’d recommend using PyTorch modules and pyro.module() rather than directly storing params in the param store via pyro.param(), because PyTorch modules are a bit easier to torch.jit.save(). I believe a pattern that should work is (again YMMV as I haven’t tried this):

class Model(nn.Module):
    ...
    def forward(self, ...):
        ...

class Guide(nn.Module):
    ...
    def forward(self, ...):
        ...

@torch.jit.ScriptModule
class Predictor(nn.Module):
    def __init__(self, model, guide):
        super().__init__()
        self.model = model
        self.guide = guide

    # This might need to be torch.jit.traced, I'm not sure.
    def forward(self, data):
        tr = poutine.trace(self.guide).get_trace(data)
        return poutine.replay(self.model, tr)(data)

...train using SVI...

# Finally save the trained predictor.
predictor = Predictor(...)
torch.jit.save(predictor, "predictor.torch")

Then I believe you can load your model with something like:

auto predictor = torch::jit::load("predictor.torch")

Let me know how this goes. We could definitely use a tutorial on how to do this, so we welcome feedback about anything you learn. I’ve added this question as a request for tutorial: FR Add tutorial on how to serve trained models from C++ · Issue #2054 · pyro-ppl/pyro · GitHub

Cheers,
Fritz

1 Like
3

There is another way to embed python into C++: using pybind11. It start a python interpreter in the process, and run python program in C++. It also support auto-transform from python/numpy data structure to C++ list data structure.

1 Like
4

@sj2019 - We have a preliminary example available on Add example for model serving using ScriptModule by neerajprad · Pull Request #2073 · pyro-ppl/pyro · GitHub. Please feel free to review / provide feedback. There are a few design issues that we need to further discuss, and this should be available in our 1.0 release soon. In the meantime, I hope that you will be able to adapt that for your needs.

5

Thank you! By the way, when is the expected release date of version 1.0?

6

I was told that the release will be around this weekend. :wink: