Simulations with infitite values for weighted binomial GAM, interpreting plots

Question

Simulations with infitite values for weighted binomial GAM, interpreting plots

tim-lauer opened this issue 6 months ago · 3 comments

Hi,

thanks for the great library. I am a beginner, hope it is ok to ask this question here.

I am working with a binomial GAM with an imbalanced DV (0: 254410, 1: 3999).

gam_model1= gam(DV_binary~ s(IV1_continuous) +
                               IV2_binary +
                               s(Cov1_continous) +
                               s(Cov2_continuous) +
                               s(Cov3_continuous) +
                               s(Cov4_continuous), 
               family = binomial, 
               data = df)
               
> summary(gam_model1)

Family: binomial 
Link function: logit 

Formula:
DV_binary ~ s(IV1_continuous) + IV2_binary + s(Cov1_continous) + 
    s(Cov2_continuous) + s(Cov3_continuous) + s(Cov4_continuous)

Parametric coefficients:
                      Estimate Std. Error z value Pr(>|z|)    
(Intercept)           -4.75387    0.03481 -136.56   <2e-16 ***
IV2_binaryTRUE  2.01337    0.08296   24.27   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Approximate significance of smooth terms:
                               edf Ref.df  Chi.sq p-value    
s(IV1_continuous)           8.925  8.998 1662.29  <2e-16 ***
s(Cov1_continous) 6.873  7.019   38.12  <2e-16 ***
s(Cov2_continous) 3.957  4.472  199.40  <2e-16 ***
s(Cov3_continous)     8.798  8.986  270.09  <2e-16 ***
s(Cov4_continous)     8.391  8.895  384.38  <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

R-sq.(adj) =  0.0323   Deviance explained = 12.1%
UBRE = -0.85936  Scale est. = 1         n = 258409

I don't have any experience interpreting the DHARMa plots (any help would be appreciated), but I figured that the predictive performance of the model - though not my main focus - was too low to be a good model (Precision: 0.35, Recall: 0.003, F1: 0.005).

I tried to account for the imbalance in the DV by using weights inversely proportional to class frequencies (which BTW only slightly improved predictive performance, Precision: 0.03, Recall: 0.75, F1: 0.07), but now, I cannot create the DHARMa plots anymore. I tried using different weights, but each time I get the error:

> simulationOutput <- simulateResiduals(fittedModel = gam_model1)
Registered S3 method overwritten by 'GGally':
  method from   
  +.gg   ggplot2
Registered S3 method overwritten by 'mgcViz':
  method from  
  +.gg   GGally
Simulations from your fitted model produce infinite values. Consider if this is sensible
Error in securityAssertion("Simulations from your fitted model produce NA values. DHARMa cannot calculated residuals for this. This is nearly certainly an error of the regression package you are using",  : 
  Message from DHARMa: During the execution of a DHARMa function, some unexpected conditions occurred. Even if you didn't get an error, your results may not be reliable. Please check with the help if you use the functions as intended. If you think that the error is not on your side, I would be grateful if you could report the problem at https://github.com/florianhartig/DHARMa/issues 

 Context: Simulations from your fitted model produce NA values. DHARMa cannot calculated residuals for this. This is nearly certainly an error of the regression package you are using
In addition: There were 50 or more warnings (use warnings() to see the first 50)

Surprisingly, when I tried to downsample the majority class of the DV (-> same count as minory class) as a quick test without weighting, the predictive performance was much better (F1=0.7), and the DHARMa plots worked just fine (and looked better from what I can tell):

So my questions would be:

Any suggestions on how I can get the DHARMa plots to work for my weighted GAM?
Any suggestions on how to deal with the imbalance that seems to be an issue here (without throwing away data)?
(How can the Residual vs. predicted plot of the initial unweighted model be interpreted?)

Any help, even a one-liner, would be greatly appreciated. Thanks.

Best,
Tim

Answer 1 · 2024-06-24T09:15:12.000Z

Hi,

your question sound as if you are coming from a ML background. They don't really make sense for a statistical analysis. I'm not sure what you are trying to achieve:

If you are trying to build a predictive model, forget about the residual checks and just do whatever as you would do in a ML study
However, if you are trying to run a statistic analysis (where you want to get correct p-values etc), then the residual checks become important but class-imbalance is not something people usually worry about and you definitely don't want to weigh your data in this kind of analysis. If this is your goal, I would recommend to read a stats book on GLMs to understand assumptions of the logistic regression etc. I'm not sure if it's helpful alone, as they don't cover the theory, but my lecture notes on practical regression analysis are here https://theoreticalecology.github.io/AdvancedRegressionModels/

Answer 2 · 2024-07-04T06:43:35.000Z

Thanks! Yes, I have some ML background, but here my goal is to do statistical analysis (good to know that one would refrain from using weights for this).

Thanks a lot for the lecture notes! If I may: would you think that the first (unweighted) model provides more or less correct p-values from the plots? It's hard for me to tell how severe the problems are (while considering that I would not want to add more variables - I have specific hypotheses about the included ones - or remove outliers unless really necessary).

Best,
Tim

Answer 3 · 2024-07-04T07:10:29.000Z

Hi Tim,

in statistical models, you cannot just re-weigh your data. That doesn't mean your model and p-values are correct. You have to ensure that you have the right model according to other strategies, which is what is discussed in courses about regression modelling.

Best
F