A meta-analysis of a teaching method reported an average effect of 0.45 standard deviations, with a narrow confidence interval and a tiny p-value. The forest plot showed individual studies ranging from −0.2 to +1.3, and I² was 85%. The 0.45 was arithmetically correct, but it averaged very different things. The more important question was why the studies disagreed, and the paper never answered it.
Meta-analysis is the statistical combination of results from several studies. Done well, it gives a more precise estimate than any single study. Done hastily, it dresses heterogeneous evidence in false certainty. This article focuses on the statistics: how pooling works and how to read the results.
Step one: put every study on the same scale
Primary studies report results in many forms: means and standard deviations, proportions, odds ratios, correlations. To pool them you need a common effect size:
| Outcome type | Common effect size | Note |
|---|---|---|
| Continuous, same scale | Mean difference (MD) | Easiest to interpret, keeps original units |
| Continuous, different scales | Standardised mean difference (SMD, Hedges’ g) | Hedges’ g corrects small-sample bias |
| Binary | Odds ratio (OR), risk ratio (RR) | Pool on the log scale, present on the original scale |
| Correlation | r, pooled after Fisher’s z transformation | Back-transform to r for reporting |
Many studies do not report everything you need, so you compute effect sizes from t, F or p values, or from figures. Record how each one was calculated; this is where data entry errors most often creep in.
Weights: bigger studies speak louder
Meta-analysis does not take a simple average. Each study is weighted by its precision, usually the inverse of its variance: one divided by the squared standard error.
A hypothetical example with three studies:
- Study A: g = 0.20, SE 0.10 → weight 100.
- Study B: g = 0.50, SE 0.20 → weight 25.
- Study C: g = 0.35, SE 0.15 → weight about 44.
Pooled estimate = sum of (g × weight) divided by sum of weights ≈ 48.1 / 169.4 ≈ 0.28. The pooled SE = 1 divided by the square root of the summed weights ≈ 0.077, so the 95% confidence interval runs from about 0.13 to 0.43. The simple average is 0.35; the pooled value is lower because Study A, the most precise, pulls it towards its own estimate.
Fixed or random effects?
| Fixed-effect model | Random-effects model | |
|---|---|---|
| Assumption | All studies estimate one true effect | True effects vary between studies around a mean |
| Question answered | What is the common effect in these studies? | What is the mean of the distribution of effects? |
| Weights | Large studies dominate | More balanced between large and small studies |
| Confidence interval | Narrower | Wider when there is heterogeneity |
In social science, education and public health, where studies differ in populations, settings and measures, a single true effect is rarely plausible; random effects is usually the default. Do not choose the model after seeing which gives a nicer result. With few studies, consider better between-study variance estimators (such as REML) and the Hartung–Knapp adjustment for confidence intervals.
Reading a forest plot
The forest plot is the centrepiece of every meta-analysis:
- One row per study: the square is the estimate, its size proportional to weight, the horizontal line its confidence interval.
- A vertical line marks no effect (0 for differences, 1 for ratios).
- The diamond at the bottom is the pooled estimate; its width is the confidence interval.
- Many plots also show a prediction interval: the range of effects you might see in a new study. It is often the most useful figure for practitioners, and often ignored.
Look at how much the horizontal lines overlap before you look at the diamond. If the confidence intervals barely overlap, the story is heterogeneity, not the average.
Heterogeneity: Q, I² and tau²
- The Q test checks whether variation exceeds what chance would produce. It has low power with few studies, so a non-significant p does not prove homogeneity.
- I² is the proportion of observed variation due to real differences between studies rather than sampling error. In the three-study example it is only about 3%.
- Tau² is the variance of true effects, on the same scale as the effect size. It tells you the absolute size of the heterogeneity.
A common misunderstanding: I² measures a proportion, not how much effects differ. With many large, precise studies, I² can be high even when true effects differ only slightly. Report tau and the prediction interval too, rather than labelling I² low, moderate or high.
Explaining heterogeneity: subgroups and meta-regression
When heterogeneity is substantial, look for its sources using variables specified in advance: education level, intervention length, study quality. Subgroup analysis compares pooled estimates between groups; meta-regression uses continuous moderators. Both are study-level observations, prone to confounding, and with few studies their power is very low. Treat them as suggestive, not as causal evidence.
Publication bias and funnel plots
Small studies with non-significant results are less likely to be published, so the set you pool may be skewed towards large effects.
- Funnel plot: effect size on the horizontal axis, precision on the vertical. Without bias, points form a symmetrical funnel. An empty lower corner (missing small studies with small effects) is a warning sign.
- Egger’s test checks asymmetry; it needs roughly ten or more studies to be meaningful.
- Asymmetry is not necessarily publication bias; small studies may genuinely differ (different populations, more intensive interventions).
- Adjustment methods such as trim-and-fill show how sensitive results are, but they cannot recover missing data.
The best protection is searching grey literature, theses and trial registries from the start.
In practice: a statistics checklist for a meta-analysis
- Choose an effect size suited to the outcome; record how each was computed and have a second person check.
- Choose the model before seeing results and justify it; default to random effects when settings vary.
- Draw the forest plot; report the pooled estimate, confidence interval and prediction interval.
- Report Q, I² and tau; if heterogeneity is large, run pre-specified subgroup analyses.
- Run sensitivity analyses: leave one study out at a time, drop high risk-of-bias studies.
- Draw a funnel plot and test asymmetry when there are enough studies.
Tools: the metafor and meta packages in R, and the meta-analysis modules in JASP and jamovi for those who prefer menus.
Next step: open a meta-analysis in your field, ignore the pooled number and look at the forest plot first. Ask whether the studies agree, how wide the prediction interval is and whether the authors explained the heterogeneity. You will read meta-analyses quite differently afterwards.
Câu hỏi thường gặp
What is a meta-analysis?
A statistical method that combines effect sizes from several studies, weighting them by precision, to obtain an overall estimate and assess how much studies differ.
Should I use a fixed-effect or random-effects model?
When studies differ in populations, settings or measures, random effects is usually more appropriate. Choose before seeing the results and state your reasons.
What I² value counts as high heterogeneity?
Do not rely on thresholds alone. I² measures the proportion of variation due to real differences, not its size; also report tau and the prediction interval.
How do you read a forest plot?
Each row is a study, with a square for the estimate and a line for its confidence interval; the diamond at the bottom is the pooled estimate. Check the overlap before the diamond.
Does an asymmetric funnel plot prove publication bias?
No. Small studies may genuinely differ in population or intervention. Treat asymmetry as a signal to investigate and use sensitivity analyses.
