Bayesian Statistics for p-Value Users: Priors, Credible Intervals and Bayes Factors

A doctoral student ran an experiment with 40 participants, found no difference, p = 0.38, and wanted to write that “the intervention has no effect”. Her supervisor reminded her that a large p-value does not demonstrate absence of an effect; it only says the data were not enough to reject the null. She asked whether there was a way to say the data support “no effect”. There is, and it is one reason many researchers turn to Bayesian statistics.

This article is not trying to convert you. The goal is to help you read a Bayesian result in a paper, recognise when the approach helps, and report it properly if you use it.

The core difference in one sentence

A p-value answers: “If the null hypothesis were true, how unusual would data this extreme be?” Bayesian analysis answers: “Given these data and what we knew beforehand, how believable are different values of the parameter?” The second is closer to what readers usually want to know, which is exactly why p-values are so often misread as if they answered it.

Three ingredients: prior, likelihood, posterior

The prior draws the most suspicion: “you are putting subjective belief into the analysis”. The practical answer is that priors must be declared openly, justified, and tested against alternatives in a sensitivity analysis. With moderate or large samples, data usually overwhelm reasonable priors and Bayesian results sit close to familiar ones.

Credible intervals: read them the way you always wanted to

A 95% credible interval contains 95% of the posterior probability for the parameter, given the chosen model and prior. You may say: “under this model and prior, there is a 95% probability the effect lies between 0.05 and 0.48.” A frequentist confidence interval cannot be read that way, although many people do.

The posterior also answers useful questions directly: what is the probability the effect is above zero, or above a practically meaningful threshold such as 0.2 standard deviations?

Bayes factors: comparing two hypotheses

A Bayes factor (BF) tells you how many times more likely the data are under one hypothesis than another.

• BF₁₀ = 6: the data are six times more likely under the effect hypothesis than under the null.
• BF₁₀ = 0.2, i.e. BF₀₁ = 5: the data favour the null fivefold. A p-value cannot tell you this.
• BF close to 1: the data cannot distinguish the hypotheses; more data are needed.

Many sources offer verbal labels (weak, moderate, strong evidence). They help orientation but should not become hard cut-offs like p < 0.05. Bayes factors also depend noticeably on the prior for the effect hypothesis; report it and try a few different widths.

When a Bayesian approach is especially useful

• You want to talk about evidence for the null, as in the opening example.
• You have credible prior information from earlier studies or a meta-analysis and want to combine it systematically.
• Complex models, such as multilevel models with few groups, where standard fitting often fails; mild priors help stabilise them.
• You want to monitor data as they arrive and stop when evidence is sufficient, within a pre-planned framework.

What Bayesian statistics cannot do is turn weak data into strong conclusions. With 40 participants the posterior is still wide; it simply says so more honestly.

Three common misunderstandings

• “Bayesian analysis does not need sample size planning”: false. Small samples give wide posteriors; you still plan sample size, just around precision or strength of evidence.
• “Pick the prior that gives a nice result”: that is manipulation, no better than p-hacking. Priors should be decided, ideally preregistered, before analysis.
• “Bayes factors and credible intervals are the same thing”: no. One compares two hypotheses; the other describes plausible parameter values, and they can leave different impressions.

Tools to get started

JASP has an SPSS-like interface and runs t-tests, ANOVA, correlation and regression both ways, which makes it ideal for learning. In R, brms and rstanarm let you write Bayesian models with syntax close to lme4. Python users often choose PyMC. Whatever you use, check convergence diagnostics (R-hat close to 1, adequate effective sample size) before reading results.

In practice: reporting a Bayesian analysis

1. State the model and software, with versions.
2. Declare the prior for each parameter and why you chose it.
3. Report convergence diagnostics: R-hat, effective sample size, number of chains and iterations.
4. Report the posterior median or mean with a 95% credible interval and, where useful, the probability that the effect exceeds a meaningful threshold.
5. If using Bayes factors, name the two hypotheses compared and the prior for the effect hypothesis.
6. Run a sensitivity analysis with at least one alternative prior.

Sample sentence: “Using the default Cauchy prior (width 0.707), the Bayes factor BF₀₁ = 4.2 indicated moderate evidence for no difference; the result was stable across alternative prior widths.”

Next step: take one “non-significant” result from your own research, rerun the equivalent test in JASP in Bayesian mode and read the Bayes factor. You will learn whether your data actually support “no effect”, or are simply not informative enough to say anything.

Câu hỏi thường gặp