Statistical Significance

Statistical significance describes a result that would be unlikely to appear if there were truly no effect or no difference in the population being studied.
555
minutes read

What Is Statistical Significance?

Statistical significance describes a result that would be unlikely to appear if there were truly no effect or no difference in the population being studied. It is the standard decision rule researchers use to judge whether an observed pattern likely reflects more than chance.

The concept sits at the heart of hypothesis testing. A researcher starts with a null hypothesis, usually stating that there is no effect, and then asks how compatible the collected data are with that assumption. When the data would be very surprising under the null hypothesis, the result is called significant.

Importantly, significance is a statement about the data relative to a model. It is not a measure of how large, important, or true an effect is.

‍

The Role of P-Values and Significance Levels

The p-value is the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is true. Smaller values indicate that the data are less compatible with the null hypothesis.

The significance level, written as alpha, is the threshold chosen before the analysis. In many fields it is set at 0.05. If the computed probability falls below alpha, the null hypothesis is rejected. Some disciplines, such as particle physics and genomics, use far stricter thresholds to limit false positives across many tests.

Setting alpha in advance matters. Choosing it after seeing the data undermines the logic of the test.

‍

Statistical vs. Clinical Significance

A statistically significant finding is not automatically meaningful in practice. Clinical significance asks whether an effect is large enough to matter to patients, practitioners, or decision-makers.

For example, a very large trial might show that a drug lowers blood pressure by a tiny amount, with a very small probability value. The result is real in a statistical sense, yet too small to change treatment. Reporting effect sizes alongside test results helps readers judge practical importance.

‍

Common Misreadings of Significant Results

Significance testing is widely used and widely misunderstood. Frequent errors include:

  • Treating a small probability value as the probability that the null hypothesis is true
  • Assuming a significant result proves the research hypothesis or a causal claim
  • Reading a non-significant result as evidence that no effect exists
  • Comparing two studies by whether each crossed the threshold, rather than by their estimates
  • Running many tests and reporting only those that reached significance, a practice often called p-hacking

Each of these misreadings can lead to overstated conclusions, and reviewers are increasingly alert to them.

‍

Reporting Significance in a Manuscript

Clear reporting lets readers evaluate results on their own terms. A strong results section states the test used, the exact probability value rather than only "p < 0.05", the effect size, and a confidence interval.

It should also explain how the sample size was determined and whether any corrections for multiple comparisons were applied. The same details are what readers look for when they critique a research paper.

‍

FAQ

Does sample size affect statistical significance?

Yes. Larger samples reduce standard error, so even very small effects can reach significance in big studies. Small samples have less statistical power and may miss real effects. This is why researchers plan sample size in advance using a power analysis and report effect sizes alongside their test results.

Can a non-significant result still be useful?

Yes. A non-significant result can narrow the range of plausible effects, especially when the confidence interval is tight. It can also feed into future meta-analyses and help reduce publication bias. Researchers should describe what the data do and do not rule out, rather than presenting the finding as proof of no effect.

Which statistical tests are used to assess significance?

The choice depends on the data and the research question. Common options include t-tests for comparing two means, ANOVA for several groups, chi-square tests for categorical data, correlation and regression for relationships, and nonparametric tests such as the Mann-Whitney U test when distributional assumptions are not met.

What does “marginally significant” mean?

Authors use the phrase for results that fall just above the chosen threshold, such as 0.06 when alpha is 0.05. Many statisticians and journals discourage it, because it bends a pre-set decision rule after the fact. A better approach is to report the exact value and effect size, then interpret them

Free access for academic researchers

Create your free QED account to validate your research, strengthen grant proposals, and uncover scientific insights.
We've sent you an access link.
Please check your inbox.

Didn't get your email? Check your spam folder or reach out to info@qedscience.com

Oops! Something went wrong while submitting the form.
Looking for QED for pharma, biotech or life science organizations?