Gaussian Process Mixture of Experts

Misc.
1

Hi,

I have time series data where the generating process switches back and forth between two types. I’d like to fit a model to the data, and capture the times the switch occurs. I’m thinking a “Gaussian Process Mixture of Experts” would work, where I have two GPs, each modelling one of the two types of data, and a “gating” function that switches between the two.

Here’s some references I’ve found:

  1. GitHub - aidanscannell/mogpe: Mixtures of Gaussian Process Experts in GPflow/TensorFlow - This seems to do what I want, but in Tensorflow. It uses a GP with Binomial likelihood as the gating function.
  2. https://proceedings.neurips.cc/paper_files/paper/2001/file/9afefc52942cb83c7c1f14b2139b09ba-Paper.pdf - One of the earlier papers, using Dirichlet process as the gating function.

I had a quick go at using the Pyro GP module, with a GP gating function, but didn’t have much luck. Before going further, I’d like to check if anyone has implemented this in Pyro, or if anyone has any tips?

Any suggestions for alternative ways to model are also welcome.

Thanks!
Paul

2

Have you read about change point models? I don’t know how feasible/easy one would be to build in Pyro (or if it’s the proper model for your problem, but it sounds like it would be), but here’s some links that might help in thinking about the problem

Wikipedia: Change detection - Wikipedia
Stan example: 7.2 Change Point Models | Stan User’s Guide
Paper which applies such a model: https://www.biorxiv.org/content/10.1101/2024.04.12.589190v1.full.pdf

3

An alternative way to model this would be using switching kalman filters (and variants)

Switching Kalman Filters (Murphy 1998)

The Kalman filter and variants are pretty straightforward to implement differentiably, but if you want that model depends on what you know or assume about the processes. E.g. if you want to actually model the forward dynamics of the process and if you assume there is a continuous statespace. It’s not necessary to also have actions as part of the model.

4

Hi @Paul,
How were you able to use the Mixture of GP experts from the Github repo? I am getting an error when trying to use it.
I was trying to implement the repo for the Mixture of Gaussian Process Experts (MOGPE) for a project (I am trying to approximate non-linear SDEs with GPs) but I found an error with the line No module named ‘mogpe.custom_types’. Did you have any similar issues? If so, how did you fix this?
Thanks