Insufficient power

[crossxwill]

Using the pwr package, I determined the optimal sample size of 38. However, when I ran simulations using this sample size, I only get a power of 70%. What's going on?

The null is p = 15%. The true data-generating process is p = 5%.

library(dplyr)
library(pwr)

set.seed(123)

pwr_n <- pwr.p.test(h = ES.h(0.05, 0.15),  # Ho: p = 0.15
                    n = NULL,
                    sig.level = 0.10,
                    power = 0.80,
                    alternative = "less")

pwr_n$n
# [1] 38.0115

conf_90 <- function(i){
  set.seed(i)
  
  x <- rbinom(n=round(pwr_n$n), size=1, prob=0.05) # True p = 0.05
  
  out <- binom.test(sum(x), length(x), conf.level=0.90, alternative = "less")
  # out <- prop.test(sum(x), length(x), conf.level=0.90, alternative = "less")
  
  
  return(out$conf.int)
  
}

intervals <- lapply(1:10000, conf_90)

df <- as.data.frame(do.call(rbind, intervals))

names(df) <- c("lower", "upper")

# Ho: p = 0.15

df$Type2Error <- df$upper >= 0.15

head(df)

# power

1 - sum(df$Type2Error) / nrow(df)

# [1] 0.6977

[crossxwill]

https://stats.stackexchange.com/questions/500771/insufficient-power-from-pwr-p-test

[heliosdrm]

As replied in stackexchange, the function in pwr calculates the power by approximation. It must be taken into account that the whole package reproduces the calculations of power described in Cohen's book, as it is referred to throughout the whole documentation, with all their advantages -- and also their drawbacks, of course.