The Research Evidence Hierarchy (Evidence Pyramid) - A Complete Guide

Part 1: The Evidence Pyramid - Overview

         ┌─────────────────────┐
         │  Systematic Review  │  ← Highest evidence
         │    & Meta-Analysis  │
         ├─────────────────────┤
         │         RCT         │
         │  (Randomized        │
         │  Controlled Trial)  │
         ├─────────────────────┤
         │    Cohort Studies   │
         ├─────────────────────┤
         │  Case-Control       │
         │  Studies            │
         ├─────────────────────┤
         │  Cross-Sectional    │
         │  Studies            │
         ├─────────────────────┤
         │  Case Series /      │
         │  Case Reports       │
         ├─────────────────────┤
         │  Expert Opinion /   │  ← Lowest evidence
         │  Animal/Lab Studies │
         └─────────────────────┘

1. Filtered (secondary) evidence - at the top: systematic reviews, meta-analyses, critically appraised topics. These synthesize and appraise existing literature.
2. Unfiltered (primary) evidence - below: RCTs, cohort studies, case-control, case series. These are original studies.

Oxford CEBM Levels of Evidence (2011)

Why Does Hierarchy Matter?

• Bias control: Lower-level studies are more vulnerable to selection bias, confounding, recall bias, and observer bias.
• Clinical decision-making: When searching for best evidence, you aim to use the highest available level.
• Different questions need different designs: As the Canberra Library guide notes, the optimal study design depends on the question type:

Part 2: Study Designs - Detailed Methodology

1. RANDOMIZED CONTROLLED TRIAL (RCT)

Core Steps of RCT Methodology

• P = Population (who are the participants?)
• I = Intervention (what is being tested?)
• C = Comparator/Control (what is it being compared to?)
• O = Outcome (what is being measured?)

• Define inclusion criteria (who qualifies?) and exclusion criteria (who is excluded?)
• The goal is to define a target population and then access a representative study sample from it
• Per Cummings Otolaryngology: "The degree to which conclusions drawn from a study are valid for the target population (beyond the subjects in the study) results from representative sampling and appropriate selection criteria"

• Intervention group (receives the treatment/drug/procedure)
• Control group (receives placebo, standard treatment, or nothing)

• Simple randomization - like a coin flip
• Block randomization - ensures equal group sizes over time
• Stratified randomization - ensures balanced distribution of key variables (e.g., age, sex) across groups
• Cluster randomization - entire groups (e.g., hospitals, villages) are randomized together

• Single-blind: The participant doesn't know which group they're in
• Double-blind: Neither the participant nor the researcher/assessor knows the group assignment - this is the gold standard
• Triple-blind: Participant, researcher, AND statistician are all blinded
• Open-label: No blinding (sometimes necessary for surgical or behavioral interventions)

• Pre-specified primary outcomes (the main endpoint, e.g., mortality, tumor response)
• Secondary outcomes (additional endpoints)
• Standardized, objective measurement tools are used

• Type I error (alpha, α) - false positive rate (typically set at 0.05)
• Type II error (beta, β) - false negative rate (typically set at 0.20, giving 80% power)
• Expected effect size between groups
• Standard deviation of measurements

• Intention-to-treat (ITT) analysis: All randomized participants are analyzed in their original groups regardless of whether they completed the treatment - this preserves the integrity of randomization
• Per-protocol analysis: Only participants who completed the study protocol are analyzed
• Statistical tests compare outcomes between groups (t-test, chi-square, survival analysis, etc.)

RCT Variants

Strengths

• Best design to establish causation
• Randomization controls for confounding
• Blinding reduces measurement bias

Weaknesses

• Expensive and time-consuming
• Ethical constraints (cannot randomize harmful exposures)
• Results may not generalize to real-world patients (artificial setting)
• Short follow-up may miss long-term effects

2. COHORT STUDY

Core Methodology

• A group of people is assembled, none of whom have experienced the outcome of interest, but all of whom could experience it
• On entry, participants are classified by their exposure status (exposed vs. unexposed to the factor of interest, e.g., smokers vs. non-smokers)

• Prospective cohort: The cohort is assembled now and followed forward in time. Data on determinants and outcomes are collected as they occur. More accurate but takes longer.
• Retrospective cohort: Uses historical records. The cohort existed in the past; the researcher looks backward using existing records. Faster but dependent on record quality.
• Ambidirectional cohort: A mix of both

• Exposure = the risk factor or intervention being studied (e.g., passive smoking, a medication, an occupational hazard)
• Outcome = the disease or event being tracked (e.g., cancer, MI, death)

• Participants are followed over a defined time period
• Loss to follow-up is a major concern - if those who drop out differ systematically from those who remain, bias is introduced

• Incidence rate = new cases per unit person-time
• Relative Risk (RR) = incidence in exposed ÷ incidence in unexposed
• Attributable Risk = incidence in exposed - incidence in unexposed

Strengths

• Best for establishing temporal sequence (exposure precedes outcome)
• Can assess multiple outcomes from a single exposure
• Best method for measuring disease incidence and natural history
• Especially valuable for studying fatal or short-duration conditions

Weaknesses

• Expensive and time-consuming for rare outcomes
• Loss to follow-up can bias results
• Prospective studies may take decades (e.g., studying passive smoking and head/neck cancer - over 10 years)
• Cannot control for all confounders (unlike RCTs)

3. CASE-CONTROL STUDY

Core Methodology

• Select individuals who already have the disease or outcome of interest (e.g., patients with perioperative MI)
• Cases are typically identified from hospital records, disease registries, or clinical databases

• Select a comparable group of individuals without the disease
• Controls must be drawn from the same population that gave rise to the cases
• Matching is often used - controls are matched to cases by age, sex, or other characteristics to control for known confounders

• Both groups are questioned or records examined for past exposure to the risk factor
• This is the retrospective element - looking backward from outcome to exposure

• In case-control studies, the key measure is the Odds Ratio (OR) - not relative risk (because the total population denominators are not known)
• OR = (odds of exposure in cases) ÷ (odds of exposure in controls)
• OR > 1 = exposure associated with increased odds of disease
• OR < 1 = exposure associated with reduced odds (protective)

Confounding Control in Analytical Studies

• Restriction: Only include subjects with certain characteristics (e.g., only non-smokers) - reduces recruitment
• Matching: Match cases and controls on confounders

• Stratification: Compare groups within strata of the confounding variable, then pool
• Multivariate adjustment: Use regression models (logistic, Cox) to adjust for multiple confounders simultaneously

Strengths

• Efficient for rare diseases (no need to follow thousands over years)
• Relatively fast and inexpensive
• Can study multiple exposures for one outcome

Weaknesses

• Cannot calculate true incidence or RR (only OR)
• Susceptible to recall bias (cases remember exposures differently than controls)
• Susceptible to selection bias (if controls are not representative)
• Cannot establish temporal sequence as clearly as cohort studies
• Difficult to establish causation

4. CROSS-SECTIONAL STUDY

Core Methodology

• Define the target population
• Use a random or representative sampling method
• Define the time point or time window for data collection

• Both the exposure (risk factor) and the outcome (disease/condition) are measured at the same visit/time point
• No follow-up is conducted

• Primary measure is prevalence (proportion of the population with the condition at that time)
• Can calculate a prevalence ratio or prevalence odds ratio

• The test result and reference standard (gold standard) are obtained at the same time
• The pathologist examining the specimen should be blinded to the test result
• Results expressed as sensitivity, specificity, positive/negative predictive values

Strengths

• Fast, relatively cheap
• No loss to follow-up (snapshot in time)
• Good for measuring disease prevalence and distribution patterns
• Useful for generating hypotheses

Weaknesses

• Provides only a snapshot - cannot establish temporal sequence
• Cannot determine causality (was the exposure present before or after the disease?)
• Not suitable for studying prognosis or natural history
• Intermittent conditions (e.g., episodic middle ear disease) may be missed at the single time point

5. CASE SERIES AND CASE REPORTS

Core Methodology

• Detailed description of a single patient's presentation, diagnostic workup, treatment, and outcome
• Includes clinical history, examination findings, investigations, management, and follow-up

• Collect cases with a specific condition, procedure, or exposure from a clinical setting

• Who (demographics, characteristics)
• What (disease/exposure/outcome)
• Why (possible etiology - though cautiously)
• When (timing, duration)
• Where (geographic distribution)

• No control group is assembled
• Data are presented descriptively (frequencies, means, ranges)
• Trends may be noted (e.g., increasing hospital admissions over years)

• Cannot quantify risk from a specific determinant
• Cannot establish causation

Strengths

• Generate hypotheses ("signal detection")
• Describe rare or novel conditions
• Inexpensive and use existing clinical data
• Few ethical limitations
• Useful for hypothesis generation before designing larger studies

Weaknesses

• No comparison group - cannot quantify risk attributable to a specific cause
• No control for confounders
• High risk of overinterpretation and publication bias (dramatic cases get published)
• Cannot determine causality
• Results are not generalizable

6. SYSTEMATIC REVIEW AND META-ANALYSIS

• A systematic review focuses on a clinical topic and answers a specific question. An extensive literature search is conducted to identify studies with sound methodology. The studies are reviewed, assessed for quality, and results are summarized according to predetermined criteria.
• A meta-analysis is a systematic review that uses quantitative statistical methods to synthesize and combine the results of multiple studies as if they were one large study.

Systematic Review Methodology (Step-by-Step)

• The question must be clear, answerable, and specific
• e.g., "In adults with hypertension (P), does ACE inhibitor therapy (I) compared to placebo (C) reduce cardiovascular mortality (O)?"

• The protocol is pre-registered (e.g., PROSPERO database) to prevent outcome-reporting bias
• Defines inclusion/exclusion criteria, databases, search terms, and analysis plan

• Multiple databases searched: MEDLINE, EMBASE, Cochrane, CINAHL, etc.
• Additional sources: grey literature, trial registries, conference proceedings, reference lists
• Two or more independent reviewers conduct the search to minimize missed studies

• All search results are screened in two stages:
  1. Title/abstract screening (broad filter)
  2. Full-text review (detailed assessment)
• PRISMA flow diagram documents how many studies were included/excluded at each stage

• Each included study is assessed for methodological quality using validated tools:
  • For RCTs: Cochrane Risk of Bias Tool (RoB 2)
  • For observational studies: Newcastle-Ottawa Scale
  • For diagnostic studies: QUADAS-2
• Studies with high risk of bias may be excluded or analyzed separately

• Standardized forms extract: study design, population, intervention, outcomes, results, sample size
• Two reviewers extract independently; discrepancies resolved by consensus

• Even without meta-analysis, a narrative summary of findings is produced
• Organized by outcome type, population, or intervention

Meta-Analysis Methodology (additional statistical steps)

• I² statistic: measures the percentage of variability attributable to true heterogeneity (not chance)
  • I² < 25% = low heterogeneity (safe to pool)
  • I² 25-75% = moderate heterogeneity
  • I² > 75% = high heterogeneity (pooling may be inappropriate)
• Q test (Cochran's Q): chi-square test for heterogeneity

• Fixed-effects model: Assumes all studies estimate the same true effect; appropriate when I² is low
• Random-effects model: Assumes studies estimate different but related true effects; appropriate when I² is moderate to high; gives more conservative (wider) confidence intervals

• Continuous data: Weighted Mean Difference (WMD) or Standardized Mean Difference (SMD/Cohen's d)
• Dichotomous data: Pooled Risk Ratio (RR), Odds Ratio (OR), or Risk Difference (RD)
• Results presented as a forest plot - each study is a horizontal line; the pooled estimate is the diamond at the bottom

• Funnel plot: A scatter plot of effect size vs. study precision; asymmetry suggests publication bias (small negative studies were not published)
• Egger's test or Begg's test: Statistical tests for funnel plot asymmetry

• Subgroup analysis: Does the effect differ by population subgroups?
• Sensitivity analysis: Does the result change if high-bias studies are removed?

Strengths

• Highest level of evidence - synthesizes all available data
• Larger effective sample size - increases statistical power
• Can detect effects too small for any single RCT to detect
• Quantifies heterogeneity between studies

Weaknesses

• Only as good as the included studies ("garbage in, garbage out")
• Time-consuming (rigorous reviews can take years)
• Subject to publication bias (positive results more likely published)
• High heterogeneity can make pooling misleading
• A single large, well-conducted RCT may be more convincing than a meta-analysis of several small, poor-quality RCTs

Summary Comparison Table

Key Concepts That Cut Across All Designs

Validity

• Internal validity: Results are valid for the study sample - achieved through proper design, unbiased measurements, and sound statistics
• External validity (generalizability): Results are valid beyond the study - achieved through representative sampling

Types of Bias

• Selection bias: Systematic difference between study sample and target population
• Information/measurement bias: Systematic error in measuring exposure or outcome
• Recall bias: Cases remember exposures differently than controls (affects case-control studies)
• Confounding: A third variable related to both exposure and outcome distorts the apparent association

Confounding Control Strategies (Scott-Brown's Otorhinolaryngology)

• Design phase: Restriction, matching, randomization
• Analysis phase: Stratification, multivariate regression adjustment