Study Design and Evidence Strength: A Plain-Language Guide

Maybe a headline said a food or supplement is linked to lower risk, and you wanted to know whether to believe it. Maybe your clinician mentioned an evidence level and you nodded without knowing what it meant. Either way, you are asking the right question: how much should this research change what I do?

The honest short answer is that a single number in a headline tells you very little on its own. You judge a study by its design, its size, what it was compared with, and what it measured. This guide walks you through those checks in the order you would use them, with links to the site’s plain-language guides for each one.

Step 1: Find the study behind the claim

You are at the headline stage. You have a claim, and you want to know where it came from.

Start by naming who made the claim. A headline, an ad or a seller’s page is that party’s statement, even when it cites research. Then look for the study itself. PubMed, the database run by the U.S. National Library of Medicine, labels each study’s type. Its record for a 2018 fibromyalgia review in Advances in Rheumatology identifies it as a systematic review, which tells you what kind of evidence it is.

Read the abstract before you trust the headline’s version of it. The health study abstract guide shows where the key results sit, and study result versus author interpretation explains why the authors’ conclusion can go further than their data.

Once you have the study in hand, the next question is what kind of study it is.

Step 2: Work out what kind of study it was

Design decides what a study can show. Some designs assign people to groups by chance. Others only watch what people already do.

A randomized controlled trial (a study where researchers randomly assign people to a treatment or a comparison group) is the design most people mean when they ask about proof. A registered running-injury trial, NCT01900262, plans three groups of 40 people: ankle strengthening, balance training, and an activity-matched control. Its PubMed record describes that plan and the injury tracking over 6 months, not the final results.

An observational study can find links, but it cannot assign cause. The Observational Study or Clinical Trial? guide explains how it differs from a clinical trial.

A systematic review gathers many studies on one question. The fibromyalgia review included 22 studies of strength training. A review is only as strong as the studies it includes, which is why the systematic review guide matters.

An in vitro study (a test run outside a living body) is another design. A 2022 systematic review of 17 in vitro studies of dental veneers noted a lack of in vivo studies, meaning studies in living people. A lab result describes a material or a test setting, not what happens in a person.

Evidence-based design is a term to read carefully. The running-trial authors based their ankle-focused approach on earlier findings that hip-strength gains did not change running mechanics. The phrase is used loosely, so check how the paper itself uses it.

Design tells you what a study could show. Size and comparison tell you how much it actually shows.

Step 3: Check size, comparison and what was measured

Once you know the design, look at who was in it, what they were compared with, and what was measured.

Strength looks like a few specific things:

• Registration. The fibromyalgia review reports the registration number CRD42016048480.
• Independent checks. The 2022 COVID-19 policy review had three reviewers assess each study separately before reaching consensus.
• Agreement across studies. The fibromyalgia review’s authors report that its 22 studies pointed the same way on pain, fatigue and sleep. They also note that no specific training protocol was defined, which is a limit you should keep in mind.

Size matters because small studies swing. The 2017 supplement review covered 26 studies, with between 9 and 206 people each. The study size guide explains what study size can and cannot tell you.

Consider a worked example. A small study reports that 2 of 10 people on a drug had a side effect, versus 1 of 10 on a placebo (a dummy treatment with no active ingredient). That is 20% against 10%. A headline might say the drug doubled the risk, yet the actual gap is one person in ten. The relative versus absolute risk guide shows how to read both numbers.

Finally, check what was measured. A lab marker or a score is not the same as an outcome you feel or a clinical event. The outcome measure guide explains that difference, and the p-value guide explains the chance-related number that often appears next to results.

With the study’s size and comparison clear, the last question is how much weight the whole body of research can carry.

Step 4: Make sense of evidence levels, ratings and numbers

You have seen "level 1", "level 3" or "low certainty" and want to know what they mean.

Evidence can be rated down. The GRADE questions guide offers a plain-language walkthrough of the questions behind evidence ratings.

Published does not mean well designed. In a 2022 BMJ Open systematic review of COVID-19 policy impact studies, reviewers assessed 27 full articles. Only 4 of the 36 published evaluations passed a set of key design checks for identifying causal impact. The review’s authors concluded that the literature largely failed to meet key design criteria for policy-actionable evidence.

The four designs that come up most in this guide are randomized controlled trials, observational studies, systematic reviews and in vitro lab studies. Pre/post and cross-sectional designs also appear. That same COVID-19 review set aside 8 pre/post studies and 1 cross-sectional study as unsuitable for its question.

If you want a clear overview of these design types, Understanding Clinical Study Design: RCTs, Observational Studies, and More covers the main categories.

Once you know how the evidence was rated, you can see how a product’s own claims line up with it.

Step 5: Hold supplement and product claims up to the design

You may be here because a product page cites studies. The seller’s statements are the seller’s, so check whether the cited study matches the product: same ingredient, similar people, similar dose.

A 2017 systematic review in the Journal of Dietary Supplements examined 26 published supplement studies of heart rhythm (the QT interval). Its designs were 15 crossover, 6 parallel and 5 observational studies. Eight found a significant change in QT intervals, and more than 30% showed some ECG change. The authors say the studies are typically well performed but could be strengthened by some design changes.

Regulation is a separate question. A 2023 narrative review in Innovations in Pharmacy argues that FDA oversight of dietary supplements falls short. It counts 79,071 adverse event reports to the FDA’s food safety reporting system between 2004 and 2021. That is the review’s own argument, drawn from case studies, and it sits beside whatever the seller claims.

For how marketing claims differ from ingredient evidence, Ingredient Evidence vs. Marketing Claims sets out the difference.

Whatever you decide about a product, the decision is better made with a clinician who knows your history.

Step 6: Take your questions to a clinician

At this point you know more than the headline did. Bring the study, its design and your question to a clinician. They can judge whether a finding applies to you, and that call belongs to them.

Questions worth taking with you:

• Was the study done in people like me, and did it measure outcomes I care about?
• Which grading system gave this finding its level, and how sure is the evidence?
• Is the comparison group a fair one, and how big was the difference for a real person?
• What would change your advice if a newer, larger study came out?

Your next step

1. Write the claim in one sentence and name who made it.
2. Find the original study and note its design, size, comparison and follow-up.
3. Check which grading system produced any level, grade or certainty rating.
4. Take the questions above to your clinician before you change anything.

Every guide on this topic

Reading a headline or a claim

• Trace Health Claim to Original Source: following a headline back to its study.
• Study Result Versus Author Interpretation: what the data show compared with what authors conclude.
• How to Read a Health Study Abstract: finding the key results quickly.
• Relative vs Absolute Risk Explained: why percentages can mislead.
• How to Calculate Relative Risk: working through the arithmetic.
• Number Needed to Harm: Why Time Period and Baseline Risk Matter: reading harm figures.
• Questions to Ask When Health Information May Not Apply to You: checking fit with your situation.
• Health Test Accuracy Claims Guide: test accuracy in plain terms.

Study design and who was studied

• Observational Study or Clinical Trial?: what the design changes.
• How to Read the Methods Section of a Health Study: five questions to ask.
• Why Study Population Matters: who was in the study.
• Why a Study Protocol and the Published Report May Not Match: comparing plan and result.
• How to Tell a Study, Guideline, and Patient Fact Sheet Apart: knowing which document you hold.
• Preprint or Peer-Reviewed Study? How to Tell What You Are Reading: checking publication status.
• How to Read a ClinicalTrials.gov Record: finding a registered plan.
• The Four Phases of Clinical Trials: how trials are staged.
• Which Clinical Trials Pay the Most: how trial payments are described.
• Health Study Corrections, Expressions of Concern and Retractions: checking whether a study was changed.

Size, chance and outcomes

• What Study Size Can—and Cannot—Tell You: reliability of small and large studies.
• What a P-Value Can—and Cannot—Tell You: the chance-related number.
• What a Confidence Interval Can Tell You: the range around a result.
• Statistical Significance and Practical Importance: when a result matters in daily life.
• Why the Outcome Measured Matters in a Health Study: what was actually tracked.
• Surrogate Outcomes and Patient Outcomes in Health Claims: markers versus outcomes you feel.
• Composite Endpoints: Which Events Actually Drove the Health Claim: combined outcomes.
• Correlation and Causation in Health Research: why linked is not caused.
• Finding Funding and Conflict Disclosures in Health Research: who paid for the work.

Grading evidence and certainty

• Why Evidence Gets Downgraded: Five GRADE Questions in Plain Language: the reasons ratings fall.
• Recommendation Strength vs. Evidence Certainty Explained: two separate guideline ratings.
• What a Systematic Review Can—and Cannot—Answer: what pooling studies can and cannot fix.
• What Insufficient Evidence Means in Plain Language: what the phrase tells you.

Study skills and reading tools

• Top 5 Study Techniques: A Practical Guide: learning to read research well.
• Best Study Methods for Medical Students Doing Research: study methods for research learners.

This guide is for education and does not replace care from a qualified professional.

By Health Evidence Literacy Guide Editorial Team