Frustrated with memory error

Hello,

I’m having a hard time running a poisson regression with just one predictor variable. I keep getting a memory error.

Traceback (most recent call last):

  File "<ipython-input-15-a0536389f5f9>", line 4, in <module>
    elbo = svi.step(x, y)

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\pyro\infer\svi.py", line 128, in step
    loss = self.loss_and_grads(self.model, self.guide, *args, **kwargs)

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\pyro\infer\trace_elbo.py", line 126, in loss_and_grads
    for model_trace, guide_trace in self._get_traces(model, guide, args, kwargs):

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\pyro\infer\elbo.py", line 170, in _get_traces
    yield self._get_trace(model, guide, args, kwargs)

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\pyro\infer\trace_elbo.py", line 53, in _get_trace
    "flat", self.max_plate_nesting, model, guide, args, kwargs)

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\pyro\infer\enum.py", line 55, in get_importance_trace
    model_trace.compute_log_prob()

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\pyro\poutine\trace_struct.py", line 216, in compute_log_prob
    log_p = site["fn"].log_prob(site["value"], *site["args"], **site["kwargs"])

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\distributions\independent.py", line 88, in log_prob
    log_prob = self.base_dist.log_prob(value)

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\distributions\poisson.py", line 63, in log_prob
    return (rate.log() * value) - rate - (value + 1).lgamma()

RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 950208322568 bytes. Buy new RAM!

The size of the data is 344,000 rows of 1’ and 0’ for the predictor variable and a long tensor for the y variable.

I basically copied/pasted the code from the regression template example.

def model(x_data, y_data):
    a = pyro.sample("mean", dist.Normal(0., 10.))
    b = pyro.sample("edit_coeff", dist.Normal(0., 1.))
    sigma = pyro.sample("sigma", dist.Uniform(0., 10.))
    mean = a + b * x_data
    rate = mean.exp()
    with pyro.plate("data", len(x)):
        pyro.sample("obs", dist.Poisson(rate = rate).independent(1), obs=y_data)  
        
def guide(x_data, y_data):
    a_loc = pyro.param('a_mean', torch.tensor(0.))
    a_scale = pyro.param('a_std', torch.tensor(1.),
                         constraint=constraints.positive)
    sigma_loc = pyro.param('sigma_loc', torch.tensor(1.),
                             constraint=constraints.positive)
    weights_loc = pyro.param('weights_loc', torch.randn(3))
    weights_scale = pyro.param('weights_scale', torch.ones(3),
                               constraint=constraints.positive)
    a = pyro.sample("a", dist.Normal(a_loc, a_scale))
    b = pyro.sample("edit_coeff", dist.Normal(weights_loc[0], weights_scale[0]))
    sigma = pyro.sample("sigma", dist.Normal(sigma_loc, torch.tensor(0.05)))
    mean =  a + b * x_data     
    
def summary(samples):
    site_stats = {}
    for site_name, values in samples.items():
        marginal_site = pd.DataFrame(values)
        describe = marginal_site.describe(percentiles=[.05, 0.25, 0.5, 0.75, 0.95]).transpose()
        site_stats[site_name] = describe[["mean", "std", "5%", "25%", "50%", "75%", "95%"]]
    return site_stats

# Utility function to print latent sites' quantile information.
def summary(samples):
    site_stats = {}
    for site_name, values in samples.items():
        marginal_site = pd.DataFrame(values)
        describe = marginal_site.describe(percentiles=[.05, 0.25, 0.5, 0.75, 0.95]).transpose()
        site_stats[site_name] = describe[["mean", "std", "5%", "25%", "50%", "75%", "95%"]]
    return site_stats

svi = SVI(model,
          guide,
          optim.Adam({"lr": .05}),
          loss=Trace_ELBO())


pyro.clear_param_store()
num_iters = 1000 
for i in range(num_iters):
    elbo = svi.step(x, y)
    if i % 500 == 0:
        logging.info("Elbo loss: {}".format(elbo))

Should I be doing something else to save on memory? Even my 32GB gpu runs out of memory where I’ve ran pymc3 models bigger than this one fine.

I think you need to remove this.

1 Like

It didn’t run but it at least it moved to another error. Why did that fix it?

I think more information about the shapes of x_data, y_data and len(x), is needed. But I just guess that x_data.shape == y_data.shape == (len(x),). If so,

dist.Poisson(rate = rate)

will have batch_shape = x_data.shape = (len(x),), event_shape = ().

dist.Poisson(rate = rate).independent(1)

will have batch_shape = (), event_shape = (len(x),).

Under plate("data"),

pyro.sample("obs", dist.Poisson(rate = rate).independent(1))

“obs” site will have batch_shape = (len(x),), event_shape = (len(x),). So this site will have shape (len(x), len(x)). Is this what you intended?

Without independent(1), obs site will have shape (len(x),), which is the shape of y (if I understand the shapes of your data correctly).

it moved to another error

Your model has Uniform(0, 10) prior for sigma but you have Normal guide for it. I think it will lead to errors when the guide provides some samples outside of the interval (0, 10). It is better to use guides having the same support as priors.

Thank you. I’m having trouble moving to pyro from pymc3 which no longer works on the computer my job provides. The shape of my x_data is torch.Size([344639, 1]) and the shape of my y is torch.Size([344639, 1]). I’m just starting with one X variable until I learn this more.

Here is the code I’ve changed. I think a NB model will better suite the data.

def model(x_data, y_data):
    a = pyro.sample("intercept", dist.Normal(0., 1.))
    b = pyro.sample("edit_coeff", dist.Normal(0., 1.))
    sigma = pyro.sample("sigma", dist.Uniform(0., 5.))
    mean = a + b * x_data
    rate = mean.exp()
    with pyro.plate("data", len(x)):
        pyro.sample("obs", dist.NegativeBinomial(total_count = 5, probs = .029), obs=y_data)  
        
def guide(x_data, y_data):
    a = pyro.sample("intercept", dist.Normal(0, 1))
    b = pyro.sample("edit_coeff", dist.Normal(0, 1))
    sigma = pyro.sample("sigma", dist.Normal(0, 5))
    mean =  a + b * x_data     
    
def summary(samples):
    site_stats = {}
    for site_name, values in samples.items():
        marginal_site = pd.DataFrame(values)
        describe = marginal_site.describe(percentiles=[.05, 0.25, 0.5, 0.75, 0.95]).transpose()
        site_stats[site_name] = describe[["mean", "std", "5%", "25%", "50%", "75%", "95%"]]
    return site_stats

svi = SVI(model,
          guide,
          optim.Adam({"lr": .05}),
          loss=Trace_ELBO())


pyro.clear_param_store()
num_iters = 1000 
for i in range(num_iters):
    elbo = svi.step(x, y)
    if i % 500 == 0:
        logging.info("Elbo loss: {}".format(elbo))

And this is where I get:

RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 475104161284 bytes. Buy new RAM!

Is there a better way to write out the model? I just want to start with a linear regression and eventually move to hierarchical model.

I think you need to specify dim=-2 here, otherwise, it will use the default value dim=-1 for your data plate. Btw, you don’t need to have mean = a + b * x_data in the guide and you need to define some variational parameters so that SVI can optimize them. Otherwise, SVI will do nothing.

1 Like

The dim =-2 and it’s running! I’m so happy. Thank you. Why does that work? I’m not sure I understand what’s going on?

I can’t believe I might finally get to use a bayesian model in my job. I’m pumped.

I copied from the tutorial, https://pyro.ai/examples/bayesian_regression_ii.html which had the mean in the guide. Is that wrong?

What would I need for the SVI outside of :
svi = SVI(model, guide, optim.Adam({"lr": .05}), loss=Trace_ELBO())

Did I mention I was happy! Thanks for sticking with me on this!

About the mean, it is not wrong. That computation is just not needed for anything else. Do you want to submit a PR in github to remove that line in the tutorial? :wink:

Your observation has two batch dimensions: -1 and -2 where -1 dim has size 1 and -2 dim has size 344639. So the data plate should be -2.

If you want batch dimension -1 to be data dim, then you can use dist.NegativeBinomial(...).to_event(1) to move the last dimension to event dimensions. More details can be found in tensor shapes tutorial.

submitted

1 Like