I am trying to create a Multivariate normal distribution from a mean and covariance matrix obtained with SWAG.
When using pyro.distributions to create the distribution I got a positive definiteness error such as the following:
ValueError: Expected parameter covariance_matrix (Tensor of shape (2570, 2570)) of distribution MultivariateNormal(loc: torch.Size(), covariance_matrix: torch.Size([2570, 2570])) to satisfy the constraint PositiveDefinite(), but found invalid values: tensor([[ 4.8379e-01, -1.0609e-02, 2.0726e-01, ..., -3.8059e+00, 4.2559e+00, -2.2484e+00], [-1.0609e-02, 1.2820e-03, -1.3205e-02, ..., 1.9880e-01, -1.9384e-01, 1.0893e-01], [ 2.0726e-01, -1.3205e-02, 6.3967e-01, ..., -3.7263e+00, 3.7177e+00, -2.1286e+00], ..., [-3.8059e+00, 1.9880e-01, -3.7263e+00, ..., 1.4046e+02, -7.0172e+01, 3.8460e+01], [ 4.2559e+00, -1.9384e-01, 3.7177e+00, ..., -7.0172e+01, 1.5973e+02, -4.2080e+01], [-2.2484e+00, 1.0893e-01, -2.1286e+00, ..., 3.8460e+01, -4.2080e+01, 4.6706e+01]], device='cuda:0')
I have checked the constraints for positive definitness and all of the them are fulfilled by my covariance matrix.
Here is a sample of the checks I performed:
import numpy as np cov = torch.load('cov_mat') cov = cov.cpu().numpy() np.allclose(cov,cov.T) # Returned true np.linalg.eig(cov) #Returned only positive eigenvalues np.linalg.cholesky(cov) #Was feasible
I thus tried to use torch.distributions.MultivariateNormal instead of the corresponding pyro implementation, which from what I have read is simply a wrapper class of the corresponding pytorch class.
Doing so, I had absolutely no issue creating the distribution and could sample from it without any issues.
I then tried to import pyro and then redo the same with pyro.distributions.MultivariateNormal, but it failed again.
In short after a bit of testing, I noticed that simply importing pyro without using it would make torch.distributions.MultivariateNormal throw the same error as pyro.distributions.MultivariateNormal.
So the following code only works with a fresh start of python if “import pyro” is commented out:
import torch import pyro cov = torch.load('/cov_mat') dis = torch.distributions.MultivariateNormal(torch.zeros(len(cov)).to('cuda'), cov) print(dis.sample())
I’ve tried running the following tests for positive definiteness which are the ones run in pytorch:
both return True, meaning that my covariance matrix is indeed positive definite.
Does anyone know why simply importing pyro without using it makes the torch code fail? And why the pyro implementation does not work, while the pytorch one does (without the import of pyro)?
I am thinking that precision might be the issue. So could it be that a simple import of pyro somehow changes the precision?