February 21, 2022

Issues With Null Hypothesis Significance Testing

According to Andrew Gelman:

  1. Overestimation of effect sizes. The statistical significance filter,” by which estimates are much more likely to be reported if they are two standard errors from zero, introduces a bias which can be huge.
  2. Estimates in the wrong direction.
  3. Extraction of statistically significant patterns from noise.
  4. Incentives for small samples and noisy measures.

Source: http://www.stat.columbia.edu/~gelman/research/published/incrementalism_3.pdf