Only 6 of 53 landmark preclinical cancer findings held up when an industry team tried to confirm them.

The authors traced the difference to choices about controls, reagents, blinding, and which data made it into the paper. Some of those choices show up in the methods. Others, like presenting only the experiments that supported the hypothesis, never reach the page at all.

Journal name, citation count, and a tidy abstract say little about whether a study's design can support its claim.

Careful readers and authors check validity one type at a time. They ask what each design choice lets the study conclude, and where the evidence falls short of the claim.

Each of the main types of validity in research comes with a life-science example, the threats that undermine it, and questions you can apply to your next paper or study design.

‍

Why Validity Is the Standard Every Research Design Must Meet

Validity is the degree to which a study's conclusions are justified by how the study was done. It covers the whole chain, from what was measured, to how groups were formed, to whom the result can apply.

Reliability is a narrower property. A reliable method gives consistent results when repeated, and a method can do that while measuring the wrong thing.

Begley and Ellis, reporting on the Amgen effort, described what set the reproducible studies apart. Those studies had paid close attention to controls, reagents, investigator bias, and reporting the complete data set, and the authors called for investigators blinded to treatment arms, rigorously validated reagents, and positive and negative controls in every experiment.

Each of those recommendations protects a form of validity: 

• Blinding guards against biased assessment. 
• Validated reagents guard against measuring the wrong biology. 
• Controls let a reader attribute an effect to the intervention.

The cost of a validity failure also travels. Later studies that build on a weak result without testing it again inherit that same weakness.

‍

‍

The Main Types of Validity in Research and What Each One Measures

Each of the main types of validity in research answers a different question about a study. This table summarizes them, and the sections that follow explain each one with an example.

‍

‍

Internal Validity

Internal validity is the confidence that a change in the outcome was caused by the intervention and by nothing else. It depends on how groups are formed, how they are treated, and who knows which group is which.

Consider a hypothetical mouse study in which treated animals sit on one rack and controls sit on another, next to a noisy door. Any difference in stress, light, or handling now travels with the treatment. The drug may work, but the design can no longer show it.

External Validity

External validity asks whether a result applies beyond the sample, setting, and time in which it was produced. In clinical research, it is whether trial results can reasonably be applied to a definable group of patients in routine practice, and the same review argues that researchers, funders, journals, and regulators tend to overlook it.

Picture a hypothetical trial that excludes everyone over 75 and anyone with kidney disease. It may be rigorous inside its own walls. Its findings still say little about the older, sicker patients who receive the drug once it is approved.

Construct Validity

Construct validity asks whether a study's measures and manipulations capture the concepts its conclusions name. A study of breast cancer biology run on cells from a different cancer has a construct problem, no matter how clean the rest of the design.

The problem is widespread. Horbach and Halffman identified 32,755 articles reporting on research with misidentified cells, cited by an estimated half a million other papers.

Endpoints raise the same issue. Surrogate endpoints frequently fail to predict clinical outcomes, and the Cardiac Arrhythmia Suppression Trial showed how far they can mislead. Drugs that suppressed arrhythmias increased mortality.

Content Validity

Content validity asks whether a measure covers every important part of the concept it targets. It matters most for questionnaires, scales, and composite scores, where one missing item quietly narrows what the instrument measures.

FDA treats coverage as a core expectation. Its 2025 guidance on clinical outcome assessments says all important parts of the concept of interest should be covered by the chosen assessment, and notes that this property was previously called content validity. When the concept closely reflects patients' daily experience with their disease, the guidance calls patient input, for example from qualitative studies, important.

A hypothetical disease where fatigue is the main complaint shows the risk. If the symptom questionnaire omits fatigue, a drug could score well while leaving patients' main problem untouched.

‍

The Most Common Threats to Validity and How They Enter a Study

Threats to validity enter through specific decisions, and each one weakens a particular kind of conclusion. Shadish, Cook, and Campbell's typology for experiments covers internal, external, and construct validity plus a fourth type, statistical conclusion validity, which asks whether the statistics support the inference. Content validity comes from measurement theory instead. 

The table maps common threats to the type of validity they damage.

‍

‍

Randomization shows how much a single design choice can move a result. An overview of 31 systematic reviews of animal studies found that failure to randomize significantly increased effect sizes. Without random allocation, part of a reported effect can come from how the animals were assigned.

Sampling choices limit external validity. A 2009 review of mammalian research in 10 biological fields found male bias in 8 of them, and in neuroscience, male-only studies outnumbered female-only studies 5.5 to 1. A result established only in males carries an untested assumption about everyone else.

‍

How to Evaluate Validity When Reading a Research Paper

Readers can judge much of a paper's validity from its methods section, provided the methods report what was done. That reporting is often thin. Of 2,671 in vivo publications across eight disease models, 24.8% reported randomization, 29.5% reported blinded outcome assessment, and 0.7% reported a sample size calculation. A missing report leaves the reader unable to check whether the step happened.

These seven questions test the design choices behind each type of validity:

• Was allocation random and was outcome assessment blind? Look for how subjects were assigned to groups and who knew the assignment. Without both, differences between groups may reflect bias.
• Was the sample size justified before the study began? A power calculation shows the effect size the authors planned to detect. Check whether that effect is realistic. Without one, a positive result is harder to trust and a null result is hard to interpret.
• Do the model and sample match the population in the conclusion? Check species, strain, sex, age, and exclusion criteria against the claim in the abstract.
• Were cell lines and key reagents authenticated? Look for identity testing of cell lines and validation data for antibodies.
• Was any questionnaire or scale developed with patient or expert input? Look for how the instrument's items were chosen and whether it was tested with the population in the study.
• Does the endpoint measure the outcome that matters? A biomarker change supports a claim about the biomarker. A claim about patients needs evidence that the marker predicts patient outcomes.
• Do the conclusions stay inside the data? Compare the title and discussion against the experiments. Claims about mechanism, generality, or clinical relevance can reach further than the evidence.

For animal work, ARRIVE 2.0's Essential 10 sets the minimum reporting without which readers cannot confidently assess the reliability of the findings. The EQUATOR Network collects reporting guidelines for other designs, including CONSORT for randomized trials.

‍

How to Strengthen Validity at the Research Design Stage

Validity is easiest to protect before you collect the first sample. Once the data exist, many design flaws can only be disclosed.

• Run a power calculation before recruiting: Base it on a realistic effect size drawn from prior work or pilot data. The median statistical power of neuroscience studies across 49 meta-analyses was about 21%, within the authors' estimated range of roughly 8% to 31%, and a reanalysis of the same meta-analyses found that power varies widely by subfield and is lowest in candidate gene association studies.
• Randomize and blind wherever the design allows: Randomize allocation, conceal it from the people enrolling subjects, and blind outcome assessment, especially for subjective measures.
• Include both sexes, or justify one: For applications submitted for the January 25, 2016 due date and later, NIH expects sex to be factored into research design, analysis, and reporting in vertebrate animal and human studies.
• Authenticate cell lines and validate key reagents: Test cell line identity when a line enters the lab and again before publication, and document antibody validation for each application.
• Choose endpoints with an established link to the outcome: When a surrogate is the only practical option, state the evidence that it predicts the clinical outcome and limit the conclusion to what the surrogate shows.
• Preregister the primary outcome and analysis plan: A plan fixed in advance separates confirmatory tests from exploratory ones and protects the validity of statistical conclusions.
• Use the reporting guideline as a design checklist: Items such as sample size justification and blinding are easier to report when they were planned from the start.

‍

Check Every Claim Against Its Evidence Before Submission

Authors have the most control over validity before submission, when they can still narrow a claim to fit the evidence or add an experiment to support it. Reviewers raise the same gaps later, and by then each fix costs a revision cycle.

QED Science was built for that pre-submission check. Paper Review breaks a manuscript into its core claims, tests the evidence behind each one against the existing literature, and returns structured, actionable findings. Every manuscript is anonymized first, and the same claim-level approach underpins QED Score, a quality metric for originality and validity.

QED is a first-pass assessment designed to work alongside expert judgment, and it is free for academic researchers. 

‍

Upload a manuscript to Paper Review and see where your evidence supports your claims and where it falls short.

‍

FAQs