Effect sizes change from v0.2.5 to v0.3.0

[timothysandhu]

Hi Joses,

I previously posted in the Google group - am reposting the issue here.

I am just wondering if I’m missing something with the updated version of the package.

I am using estimation statistics to analyse the difference in scores between before and after an intervention for two groups (case and control)

Back in July (using version 0.2.5) I ran a series of multi-paired plots using the v2_code attached below (adapted from the vignettes helpfully provided on the package page).

I returned to the analysis late last year, and updated to version 0.3.0, and I used the v3_code attached (again this was adapted from the provided vignettes).

However, the results provided very different paired mean difference plots, despite the Tufte slopegraphs being identical. I have verified this more recently by running the above code on the same machine and installing either version 0.2.5 or 0.3.0.

As far as I can see, the only difference in the code is the use of the mean_diff function in the code used for version 0.3.0. Are there any additional arguments I need to provide that would correct for this difference, or am I missing something?

I have dug a little deeper, and it seems that the mean differences and the mean of the bootstraps are the same across versions (the plots seem to show this too) - but the variance/shape of the bootstrap distribution and hence the confidence intervals are different in v.0.3.0.

I noticed that the bootstrap is now done using boot not simpleboot, could my difference be related to that?

I would be really grateful for any support with this,

Thanks so much,

Best wishes,

Tim

v2_code.txt

v3_code.txt

v_2_plot.pdf

v_3_plot.pdf

[timothysandhu]

Hi Joses,

Hope you're well!

Just wondering if there had been any update on this. We are in the final stages of a manuscript and would love to be able to include these results.

Do let me know if you need any more information about the issue.

Thanks again for taking a look.

Best,

Tim

[josesho]

It looks to me like the default ylims for your dataset are different between 0.2.5 and 0.3.0.

Could you try

plot(multi.two.group.paired.mean_diff, 
       effsize.ylim = c(-0.5, 1))

to normalise the way plots from either 0.2.5 or 0.3.0 turn out?

[timothysandhu]

Plots attached from both versions with the above effsize.ylim specification.

Thanks!

Tim
v_2_plot_effsize.pdf
v_3_plot_effsize.pdf

[josesho]

OK, looks like a bug, let me get back to you!

[timothysandhu]

Thanks so much Joses! Let me know if you need anything else that would help.

[timothysandhu]

Hi Joses,

Hope you're well!

Just wondering if there's anything else you need from me that might help with the bug?

Thanks so much again for taking a look.

Best,

Tim

[josesho]

This is likely the same as #94. Still working on it!

[timothysandhu]

Thanks for the update Joses - it looks as if the issues are the same. I will see if I can find anything in the boot package for groups with unequal N. Cheers!

[josesho]

cf #107

[timothysandhu]

Hi Joses, thanks for letting me know about this update. Should I edit the effectsize.R code manually, or should I wait for v.3.1 to be releaed (apologies, I'm not very Github literate!). Cheers.

[josesho]

Hi @timothysandhu ,

We're not quite ready to ship v0.4 yet, but you can install the dev version branch v0.3.9999 with

devtools::install_github("ACCLAB/dabestr",  ref = "v0.3.9999")

and let us know if you face any issues!

[timothysandhu]

Hi! Thanks for that. I've run through all my datasets and the differences between confidence intervals in v0.2.5 and v0.3.9999 only show up at the second decimal place (usually +- 0.02). Whereas v0.3.0 produced wildly different results. Thanks to you all for getting to the bottom of this! :)

[josesho]

All thanks to @FellowFish !