Your first model gives p = 0.08. You notice three participants rushed through the survey and drop them: p = 0.06. You add gender as a covariate because “it makes theoretical sense”: p = 0.04. None of these steps is fraud. Yet the probability of finding a “significant” result when no real effect exists has climbed well above the 5% you believe you are working with.
That is p-hacking: trying multiple analyses and stopping at the one that crosses p < 0.05. Statisticians describe the broader problem as the “garden of forking paths”: every reasonable decision is a fork, and if forks are chosen after seeing the data, the p-value loses its intended meaning.
Why a few small choices matter so much
An alpha of 0.05 means that, with no true effect, you have about a 5% chance of a significant result on one pre-specified test. Run 20 independent tests and the chance that at least one comes out below 0.05 is about 64%. Real analytic choices are not fully independent, so the true figure is lower, but well-known simulations in psychology showed that combining just a handful of flexible choices can multiply the false-positive rate several times over.
Seven common habits and their fixes
| Habit | What it looks like | Transparent alternative |
|---|---|---|
| Optional stopping | Checking p after every 20 participants and stopping when it drops | Fix sample size in advance with a power analysis, or use a sequential design with correction |
| Post hoc outlier removal | Trying several cut-offs and keeping the convenient one | Pre-specify the rule; report results with and without exclusions |
| Flexible covariates | Adding and removing controls until the coefficient is significant | Choose covariates on theoretical grounds before analysis |
| Outcome switching | Measuring five outcomes, reporting one | Name the primary outcome; report every outcome measured |
| Post hoc subgroups | “The effect only appears in second-year female students” | Label as exploratory, correct for multiple comparisons, seek replication |
| HARKing | Writing hypotheses after seeing results and presenting them as predictions | Separate confirmatory hypotheses from exploratory findings |
| Generous rounding | Reporting p = 0.054 as “p = 0.05” or “marginally significant” | Report exact p-values with effect sizes and confidence intervals |
HARKing: telling the story backwards
HARKing, Hypothesizing After the Results are Known, means writing the introduction so that an unexpected finding looks as if it had been predicted all along. It is tempting because the paper reads more smoothly, and supervisors sometimes encourage it with good intentions: “tighten up the narrative”. The problem is that readers then overrate the evidence, because a confirmed prediction carries more weight than a chance discovery.
Honest wording does not weaken the paper: “We did not predict this effect. One possible explanation is… The finding should be tested in an independent sample.”
Multiple comparisons and how to correct
When you test many hypotheses at once, some significant results will appear by chance:
- Bonferroni: divide alpha by the number of tests. Simple and conservative, fine when tests are few.
- Holm: a stepwise version that is always at least as powerful as Bonferroni.
- Benjamini–Hochberg: controls the false discovery rate, suited to many tests where a small share of false positives is acceptable, as in genomics.
More important than the method: disclose how many tests you actually ran. Correcting for 5 tests when you tried 30 solves nothing.
Report so readers can judge for themselves
- Effect sizes (d, r, odds ratios…) with 95% confidence intervals, not just p-values.
- Every outcome measured, every condition, every exclusion criterion.
- A sentence such as: “We report how we determined our sample size, all data exclusions, all manipulations and all measures in the study.” Many psychologists use this as a standard commitment.
- Sensitivity analyses: with and without outliers, with alternative covariate sets.
Multiverse analysis: turning a weakness into a strength
When several data-processing choices are equally defensible, run them all and report the distribution of results instead of picking one. This is known as multiverse analysis or a specification curve. If the effect appears under most reasonable choices, your conclusion is far stronger than any single model could make it. If it appears in only a few, you and your readers need to know.
A self-check before writing the results
- Was the analysis plan written before I saw the data? If not, have I labelled later decisions as exploratory?
- How many models did I try before settling on this one? Did I record them?
- Was the sample size fixed in advance or decided while watching results?
- Does the main result survive changing one reasonable choice?
- Have I reported every outcome I measured?
A practical tip: keep an analysis log with the date and every model you ran, including the ones that “failed”. It takes five minutes a session and lets you answer reviewers, and yourself, honestly.
Your next step: for your current analysis, write down three defensible alternatives for outlier handling and covariates, run all three and see whether the conclusion changes. The answer tells you how confidently you can write your discussion.
Câu hỏi thường gặp
What is p-hacking?
It is trying multiple ways of analysing data and reporting the one that reaches statistical significance, which inflates the false-positive rate. It often happens unintentionally through small decisions made after seeing the data.
Is removing outliers p-hacking?
Not if the rule is set in advance and applied consistently. It becomes a problem when you try several cut-offs and keep the one that gives the result you want.
What does HARKing mean?
Hypothesizing After the Results are Known: presenting a hypothesis formed after seeing the data as if it had been predicted. The honest alternative is to say the finding was unexpected and needs further testing.
When should I use a Bonferroni correction?
When you test several hypotheses that could each support a conclusion. With many tests, false discovery rate methods such as Benjamini–Hochberg are often more appropriate.
Can I call p = 0.06 marginally significant?
It is better not to. Report the exact p-value alongside the effect size and confidence interval so readers can judge the strength of the evidence.
