This Book Is So Hard It’s Usually Only Read by PhDs—But It'll Transform How You Think

Summary

The video presents five lessons from psychologist Jacob Cohen’s Statistical Power Analysis for the Behavioral Sciences. Its central message is that research findings should not be accepted or dismissed based on a single statistic. To judge them, consider whether a study could detect an effect, how large and practically important that effect is, and how much uncertainty and how many alternative explanations remain.

Five key principles

1. Ask whether the study could have detected the effect

Statistical power is the probability that a study will detect an effect of a particular size if that effect exists. A study that finds no statistically significant result may have missed a real effect because its sample was too small, its measurements were imprecise, or the effect was difficult to detect.

Cohen’s analysis of 70 studies published in 1962 found that, on average, they had an 18% chance of detecting a small effect, a 48% chance of detecting a medium effect, and an 83% chance of detecting a large effect.

Practical questions:

  - How many participants were studied?
  - What effect could the study detect?
  - Does a null result rule out an effect, or could the study simply have been unable to find it?

1. Separate statistical significance from practical importance

Statistical significance generally concerns how surprising the data would be under a null hypothesis, often one of no difference. It does not establish that a result is useful, important, or valid.

Very large samples can make tiny effects statistically significant. Smaller studies can produce promising but uncertain estimates that do not cross the usual significance threshold.

Practical questions:

  - What was the actual difference—in minutes, dollars, symptoms, or lives?
  - How uncertain is the estimate?
  - Would a difference of that size matter in real life?

1. Decide what evidence is needed before collecting data

Cohen’s framework links four elements: effect size, number of observations, the threshold for false alarms, and the probability of detecting an effect if it exists. Once three are specified, the fourth is determined.

An a priori power analysis uses this relationship to plan a study in advance, rather than recruiting whoever is available and interpreting the results afterward. A planned study should define what effect would matter, how likely it should be to detect that effect, and how many participants or observations are needed.

When reviewing research: Look for a justified sample size and a stated target effect. If these are missing, the study is not automatically worthless, but its results may come from an analysis of whatever data happened to be available.

1. Treat standardized effect-size labels as starting points, not verdicts

Cohen’s d conventions—roughly 0.20 for small, 0.50 for medium, and 0.80 for large—offer a shared vocabulary and can help with study planning.

Cohen intended these labels as rough guides, not universal judgments of importance. The same standardized label can correspond to consequences that matter very differently depending on the context.

Practical questions:

  - What does the effect mean in ordinary units?
  - How many people does it affect?
  - Does it accumulate over time?
  - What does it cost to achieve?
  - How reliable are the measurements?

Judging whether an effect is worthwhile involves contextual and value-based judgments, not statistics alone.

1. Do not treat statistical thresholds or standardized measures as dogma

Cohen criticized the mechanical practice of labeling results “significant” if p is below 0.05 and dismissing them if it is above. Results with p = 0.049 and p = 0.051 can represent nearly identical evidence.

A p-value is not the probability that the hypothesis is true or that the effect exists. Statistical significance also does not prove causation: other differences between groups may explain an association.

Evidence should update confidence in proportion to the study design and findings, rather than being treated as a final verdict.

A Cohen-style research audit

When assessing a claim from research or the news, ask:

• Could the study detect the effect it was looking for?
• How large was the effect, and how precise was its estimate?
• What does the effect mean in conventional, real-world units?
• Would it make a practical difference, given its costs and consequences?
• What alternative explanations remain, especially for causal claims?
• Has the finding been replicated in other samples?

Replication and the limits of published findings

The video describes a 2015 effort in which researchers attempted to replicate 100 studies from three leading psychology journals. While 97% of the original studies had reported statistically significant results, 36% of the replications did; the average replicated effect was about half the original.

The project authors cautioned that no single measure determines whether a result has replicated, and differences can have multiple causes. Still, the findings illustrate why results may look less certain when tested again, particularly with larger samples.

The video also notes that published studies can create a misleading impression of certainty: weak or inconclusive findings may be described as evidence of no effect, small effects may be presented as breakthroughs, and unsuccessful studies may never be published.

Speakers and sources featured

• Stephen Petro — The video’s narrator and presenter.
• Jacob Cohen — Psychologist and author of Statistical Power Analysis for the Behavioral Sciences; his ideas and research are the focus of the presentation.
• Cohen’s graduate-student colleague — Mentioned in Cohen’s account of a small study that failed to find a significant result; not named in the subtitles.
• The authors of the 2015 replication project — Discussed in relation to attempts to replicate 100 psychology studies; not individually named in the subtitles.