How to debug loo calculation?

numpyro
1

Hi everyone,

I estimate a hierarchical model on 200 decisions from 300 subjects with 4 types of parameters (with a subject and a condition level) and everything looks great so far: I get R-hat of 1 (sometimes 1.01) on all parameters and only 1 divergence popped up during estimation.

However, I have the following problem: The LOO calculation fails. I get many RuntimeWarnings such as "

  • divide by zero encountered in divide b_ary [/](https://file+.vscode-resource.vscode-cdn.net/)= prior_bs * ary[int(n [/](https://file+.vscode-resource.vscode-cdn.net/) 4 + 0.5) - 1]
    or
  • divide by zero encountered in divide b_ary += 1 [/](https://file+.vscode-resource.vscode-cdn.net/) ary[-1]
    and ultimately the following UserWarning:
  • UserWarning: Estimated shape parameter of Pareto distribution is greater than 0.7 for one or more samples. You should consider using a more robust model, this is because importance sampling is less likely to work well if the marginal posterior and LOO posterior are very different. This is more likely to happen with a non-robust model and highly influential observations.

Conversely, The Output results in “nan”:

Computed from 12000 posterior samples and 60000 observations log-likelihood matrix.

         Estimate       SE
elpd_loo      nan      nan
p_loo         nan        -

There has been a warning during the calculation. Please check the results.
------

Pareto k diagnostic values:
                         Count   Pct.
(-Inf, 0.5]   (good)     54086   90.1%
 (0.5, 0.7]   (ok)        1762    2.9%
   (0.7, 1]   (bad)       1638    2.7%
   (1, Inf)   (very bad)  2514    4.2%

How do I proceed from here? The WAIC calculation works, but gives me a similar Warning:

UserWarning: For one or more samples the posterior variance of the log predictive densities exceeds 0.4. This could be indication of WAIC starting to fail. 
See http://arxiv.org/abs/1507.04544 for details
  warnings.warn(
Computed from 12000 posterior samples and 60000 observations log-likelihood matrix.

          Estimate       SE
elpd_waic -14359.38   133.33
p_waic      629.21        -

There has been a warning during the calculation. Please check the results.

What do you recommend? Thank you!

Best
N

2

@nmwitzig Are you computing with arviz or some other outside package? Because if so that would be the appropriate place to ask.

3

Hi martinjankowiak,

you’re right: I use arviz, but figured that maybe people here may had similar issues - given I use numpyro for model fitting; but yeah: probably better to head over to arviz discussion forums.

Thank you!
Best
N

4

if all the samples returned by numpyro are finite/not-nan then whatever is going on has to do with the details of computing loo