Dr. Amanda Foster, Director of Medical Education at Leeds Teaching Hospitals NHS Trust, has noticed that traditional lecture-based continuing professional development (CPD) programmes have poor engagement rates (35%) among healthcare professionals. Her team has developed an innovative digital learning platform combining interactive simulations, micro-learning modules, and peer collaboration tools. The platform includes personalised learning pathways delivered via mobile app, weekly virtual case discussions with expert facilitators, and real-time assessment of competency progression. Initial pilot work with 20 healthcare professionals showed promising results – 85% completion rates and improved clinical knowledge scores. However, Dr Foster knows that to change education policy and secure NHS Education funding, she needs robust evidence from a well-designed randomised controlled trial. The hospital research committee has challenged her to design a study that will provide definitive evidence while addressing the practical and ethical challenges of testing educational interventions with busy healthcare professionals. The stakes are high: if effective, this intervention could revolutionise professional development delivery and improve competencies for thousands of healthcare workers. If the study is poorly designed, valuable educational resources will be wasted and learning may not improve. 1. I'd like the group to critically analyse what distinguishes experimental research from the observational studies we've covered in previous weeks. Drawing on Dr Foster's scenario, what fundamental principles must she incorporate to establish whether her digital platform actually causes improved learning outcomes, rather than just being associated with them? Please bring evidence from your professional experience to support your discussion about why this distinction matters for educational practice. 2. Please consider the various threats to internal validity that could undermine Dr Foster's ability to draw causal conclusions. I would like the group to identify at least three specific threats in this educational context and propose how randomisation and other design features could address these challenges. What might happen if these threats aren't properly controlled? 3. I'd like you to debate the tension between internal and external validity in Dr. Foster's study design. How might design choices that strengthen her ability to prove the intervention works (internal validity) potentially limit how generalisable the results are to real-world educational practice (external validity)? Please discuss specific design decisions she'll need to make and their trade-offs. 4. Drawing on your professional experience, please analyse what factors Dr. Foster should consider when choosing between an efficacy trial (explanatory) versus an effectiveness trial (pragmatic) approach. I would like the group to discuss how this fundamental choice will influence every aspect of her study design, from participant selection to outcome measurement. The following articles could help you: Cook, D.A. and Reed, D.A. (2015) 'Appraising the quality of medical education research methods: the Medical Education Research Study Quality Instrument and the Newcastle-Ottawa Scale-Education', Academic Medicine, 90(8), pp. 1067-1076. Available at: https://roecsg.org/wp-content/uploads/2021/01/mersqi-and-nos-ecook-and-reed-acad-med-2015.pdf Friedman, L.M., Furberg, C.D., DeMets, D.L. and Reboussin, D.M. (2015) Fundamentals of Clinical Trials. 5th ed. Cham: Springer International Publishing.

Reading File

Searching PubMed

randomised controlled trial medical education digital learning

Searching the Web

internal external validity medical education RCT design efficacy effectiveness trial CPD

Reading Article

Searching PubMed

pragmatic trial efficacy effectiveness healthcare education continuing professional development

Searching PubMed

threats internal validity educational research randomisation bias

Reading Web Page

MERSQI medical education research quality internal validity randomisation

Searching PubMed

online learning continuing professional development healthcare workers knowledge outcomes

Searching PubMed

cluster randomised trial educational intervention Hawthorne effect contamination healthcare

Reading Web Page

pragmatic vs explanatory clinical trial PRECIS design framework healthcare

Reading Article

Searching PubMed

simulation based medical education randomised controlled trial competency assessment

I now have enough evidence to write a thorough, well-sourced response to all four questions. Let me compose it.

RCT Design for Digital CPD Interventions: A Group Discussion Guide

Dr Amanda Foster's Digital Learning Platform - Leeds Teaching Hospitals NHS Trust

Question 1: Experimental vs. Observational Research - Establishing Causation

The Core Distinction

The fundamental difference between experimental and observational research lies in investigator control over the exposure. In observational designs - cohort studies, cross-sectional surveys, before-and-after studies - the researcher measures what naturally occurs. In experimental research, the investigator actively allocates the intervention and controls conditions to isolate its effect.

Dr Foster's pilot data (n=20, 85% completion, improved knowledge scores) is descriptive and observational. It establishes an association between using the platform and better outcomes, but cannot tell her whether the platform caused the improvement. Several alternative explanations are plausible: these 20 volunteers may be unusually motivated, they may have sought other learning opportunities simultaneously, or knowledge may have improved simply because time passed (maturation). To move from association to causation, she needs a design that can rule out these rival explanations systematically.

The Three Causal Criteria (Bradford Hill Adapted)

For Dr Foster to claim her platform causes improved learning, she must demonstrate:

Temporality - platform exposure must precede knowledge gain (easily satisfied by a pre-post design)
Association strength - a statistically significant effect must exist (requires adequate sample size and a control group)
Ruling out alternative explanations - this is where observational designs fail, and only proper experimental control achieves it

Why Randomisation is the Pivotal Tool

Randomisation is not merely a procedural step - it is the mechanism by which unknown confounders are distributed equally across groups. As Friedman et al. (2015) emphasise, randomisation is the only method that simultaneously controls for both known and unknown confounders, which is why the RCT holds a privileged position in the hierarchy of evidence. In educational research specifically, confounders are numerous and often invisible: prior digital literacy, intrinsic motivation, departmental learning culture, senior mentorship availability, and individual cognitive load all influence learning outcomes independently of the intervention.

Cook and Reed (2015) specifically address the weakness of non-randomised educational studies using the MERSQI and Newcastle-Ottawa Scale-Education (NOS-E). Their work shows that most medical education research scores poorly on study design and internal validity domains - the median MERSQI score across reviewed studies was 11.3 out of 18, with randomisation and comparison group selection being consistently weak areas. This means the field has a systemic problem with conflating association and causation, making Dr Foster's well-designed RCT all the more impactful.

Why This Distinction Matters for Educational Policy

The stakes of this distinction are not merely academic. If Dr Foster secures NHS Education funding based on flawed evidence, one of two costly errors follows. She may implement an ineffective platform widely (wasting resources and displacing other CPD time), or she may abandon a genuinely effective one because a confounded study failed to detect its benefit. From a professional practice standpoint, healthcare professionals - already under time pressure - will only engage sustainably with CPD that demonstrably improves their clinical competence. Evidence of genuine causal benefit, rather than correlational association, is what justifies mandatory adoption, redesign of rotation schedules, and reallocation of training budgets at NHS scale.

A systematic review and meta-analysis of spaced digital education (Martinengo et al., 2024, PMID 39388234) found that spaced online education was superior to massed online delivery for knowledge (SMD 0.32, 95% CI 0.13-0.51) and clinical behaviour change (SMD 0.67, 95% CI 0.43-0.91) - but critically, 83% of included studies had unclear or high risk of bias, largely because they lacked proper randomisation and comparison group design. This underscores exactly why Dr Foster's RCT matters: it would be among the minority of studies in this space that can actually justify causal claims.

Question 2: Threats to Internal Validity and How to Address Them

Internal validity asks: did the intervention, and nothing else, cause the observed difference? In educational settings, this question is harder to answer than in pharmacological trials. At least three major threats apply directly to Dr Foster's scenario.

Threat 1: Selection Bias

The problem: If participants self-select into the digital platform group (or if allocation is based on convenience - e.g., one ward gets the intervention, another does not), the groups may differ systematically at baseline. Healthcare professionals who volunteer for a digital CPD pilot may be younger, more tech-confident, or more intrinsically motivated than those who do not. Any outcome difference could then reflect who the participants are, not what the platform does.

The solution: Individual or cluster randomisation with allocation concealment. Dr Foster should use a centralised randomisation service (e.g., a sealed envelope system or computer-generated sequence held by an independent statistician) so that neither the recruiter nor the participant can predict or influence allocation. This distributes motivation, digital literacy, specialty, and seniority equally between arms by the operation of chance. Stratified randomisation (by grade, specialty, or trust site) can further ensure balance on the most important known confounders.

If unaddressed: Without randomisation, Dr Foster cannot distinguish platform effects from self-selection effects. She may report a 30% knowledge gain, but sceptics on the NHS funding committee will correctly argue that the intervention group was simply more motivated to learn, rendering her findings uninterpretable.

Threat 2: Contamination (Diffusion of Treatment)

The problem: Healthcare professionals work in teams, share offices, and talk to each other. If control group participants learn about, discuss, or informally access elements of the digital platform - or if intervention participants share micro-learning content with colleagues - the contrast between groups erodes. This is called contamination, and it biases the study toward the null, potentially causing a real effect to appear absent.

The solution: The most effective structural protection is cluster randomisation - randomising entire units (e.g., wards, clinical teams, or hospital sites) rather than individuals. If one team receives the platform and the adjacent team does not, informal cross-contamination between trial arms is substantially reduced. Dr Foster should also pre-specify in her protocol that sharing of intervention materials with control participants constitutes a protocol deviation and track it prospectively. A minimum physical or organisational separation between clusters provides additional protection.

If unaddressed: The trial could produce a false negative - an apparently ineffective intervention - because control participants partially received the intervention through informal channels. NHS Education would then decline to fund a platform that actually works.

Threat 3: The Hawthorne Effect and Performance Bias

The problem: Participants who know they are in a study - and particularly those who know they are in the intervention arm - may change their behaviour for reasons unrelated to the platform itself. They may study harder, seek supplementary resources, or engage more attentively with patients as a result of feeling observed and valued. This is the Hawthorne effect. Similarly, knowledge that colleagues in the control group are receiving "standard" CPD may demoralise control participants, artificially depressing their performance (resentful demoralisation).

The solution: Complete blinding of participants is not feasible (they will know whether they have a mobile app or not), but active control comparisons substantially mitigate this threat. Rather than a no-treatment comparison, control participants should receive the current standard CPD delivery (e.g., traditional lectures and mandatory e-learning modules) packaged with equivalent contact time and facilitator attention. This ensures that any observed difference reflects the platform's specific features rather than the mere fact of receiving structured educational attention. The use of blinded outcome assessors - who do not know which arm participants came from when scoring knowledge assessments - is also essential.

If unaddressed: Even if Dr Foster finds a significant effect, reviewers will argue that the improvement was caused by the enthusiasm and attention that accompanied the new platform, not by the platform's specific mechanisms. This is particularly damaging for policy change because it suggests the effect will vanish once the novelty wears off.

Additional Threat: Attrition Bias

Healthcare professionals are among the busiest potential research participants. If drop-out is not random - if, for instance, the highest-burden clinicians in the intervention arm disengage from the platform (and therefore drop out), while lower-burden clinicians complete it - then the final analysis will compare a selected subset of the intervention group against the full control group. Dr Foster should pre-specify an intention-to-treat (ITT) analysis as the primary analysis, which includes all randomised participants regardless of whether they completed the intervention. This preserves the protection against selection bias that randomisation provided at the outset, and produces a real-world estimate of effect that reflects the practical reality of incomplete engagement.

Question 3: The Internal vs. External Validity Tension

This tension is one of the most intellectually honest debates in trial design, and it is particularly acute in educational research where the "intervention" is a complex, multi-component experience rather than a discrete molecule.

The Tension Defined

Internal validity (can we trust that the intervention caused the effect, within this study?) is maximised by tight experimental control: strict eligibility criteria, standardised delivery, controlled settings, frequent monitoring, and protocol adherence checks. External validity (do the findings apply to other healthcare professionals, settings, and contexts?) is maximised by the opposite: broad eligibility, flexible delivery, real-world settings, and minimal protocol constraints.

These two goals pull in opposite directions, and every major design decision Dr Foster makes will involve a trade-off between them.

Key Design Decisions and Their Trade-offs

Participant selection criteria

For internal validity: Dr Foster might restrict recruitment to a single specialty (e.g., emergency medicine), a single grade (e.g., Band 6-7 nurses), and a single site (Leeds General Infirmary). This produces a homogeneous, comparable sample and minimises unexplained variance in outcomes.
Cost to external validity: Results will only confidently apply to Band 6-7 emergency nurses at a large teaching hospital in West Yorkshire. NHS Education may legitimately ask whether the effect holds for GPs in rural practices, for Band 3 healthcare assistants, or for doctors in specialty training.
The trade-off: A broad multi-specialty, multi-site, multi-grade design increases generalisability but introduces heterogeneity that complicates analysis and may dilute detectable effects if the platform works better for some groups than others.

Intervention standardisation vs. flexibility

For internal validity: The protocol should standardise exactly which modules are delivered, in what sequence, over what timeframe, and with what facilitator scripts. This ensures all intervention participants receive the same "dose" and allows confident attribution of effects to the platform.
Cost to external validity: In real-world deployment, NHS trusts would adapt the platform - adjusting module content to local guidelines, varying session frequency, deploying different facilitators. A rigidly standardised protocol therefore tests a version of the platform that may never exist outside the trial.
The trade-off: Allowing some adaptation (e.g., specifying a minimum dose but permitting local facilitator latitude) preserves ecological validity but makes it harder to identify which elements are driving outcomes.

Comparison condition

For internal validity: A no-treatment control is the cleanest comparator - any difference from zero must be due to the platform.
Cost to external validity: No NHS trust operates without any CPD. A pragmatic comparison against current practice (the platform vs. whatever CPD already happens) is more policy-relevant but introduces a weaker contrast, requiring larger sample sizes.
The trade-off: An active control comparison against "usual CPD" (lectures, mandatory modules) produces results directly relevant to NHS commissioning decisions, even if the detectable effect size is smaller and the required sample larger.

Outcome measures

For internal validity: Multiple-choice knowledge tests administered under controlled conditions are reliable, objective, and easy to blind. They produce clean data with low measurement error.
Cost to external validity: MCQ scores do not capture what ultimately matters - whether healthcare professionals practise differently and whether patient outcomes improve. Moore's Outcomes Framework distinguishes seven levels, from participation (Level 1) through to health outcomes (Level 7). Williams et al. (2023, PMID 37316431) found that e-learning CPD interventions largely demonstrated improvements only up to Level 4 (competence in educational settings), with no studies demonstrating improvements in real-world workplace performance or patient health. This is the external validity gap Dr Foster must try to close.
The trade-off: Including workplace-based assessments, case audit data, or patient outcome metrics as secondary outcomes would dramatically strengthen external validity but adds cost, complexity, and the risk that these measures are confounded by factors outside Dr Foster's control.

Setting and fidelity

Conducting the trial entirely within Leeds Teaching Hospitals improves operational control (internal validity) but limits generalisability across the diversity of NHS settings (a single tertiary teaching hospital is atypical of the broader NHS workforce landscape).
A multi-site cluster RCT across, for example, Leeds, Sheffield, and a district general hospital would meaningfully extend external validity, though it adds logistical complexity, increases risk of site-level confounding, and requires a larger sample.

The Practical Recommendation

Rather than treating this as a binary choice, Dr Foster should consider a phased design strategy: a tightly controlled efficacy trial first (optimising internal validity to establish whether the platform works under ideal conditions), followed by a pragmatic effectiveness trial (testing whether it works in routine NHS deployment). This is analogous to Phase II/III drug development and is explicitly supported by the effectiveness-implementation hybrid design literature (Curran et al., PMC3731143). This staged approach avoids wasting resources on wide-scale implementation before efficacy is confirmed, while building the evidence base for policy change in a structured and credible way.

Question 4: Efficacy (Explanatory) vs. Effectiveness (Pragmatic) Trial

Defining the Choice

An efficacy trial (explanatory) asks: can the intervention work under optimal, controlled conditions? It maximises fidelity to the protocol, selects participants likely to comply and respond, and controls the environment tightly to isolate the intervention's mechanism. Its answer is relevant to the question of whether the platform is worth developing further.

An effectiveness trial (pragmatic) asks: does the intervention work in real-world NHS practice, delivered by ordinary facilitators to ordinary healthcare professionals? It relaxes eligibility criteria, allows adaptive delivery, and measures outcomes that matter to patients and system commissioners. Its answer is relevant to the question of whether the platform should be funded and rolled out at scale.

The PRECIS-2 framework (Pragmatic-Explanatory Continuum Indicator Summary) provides a useful tool for characterising where a trial sits on this spectrum across nine domains: eligibility, recruitment, setting, organisation, flexibility of delivery, flexibility of adherence, follow-up, primary outcome, and primary analysis.

Factors Dr Foster Should Consider

The funding objective

NHS Education funding decisions require evidence that an intervention is effective in NHS conditions - not just that it works in a tightly controlled academic trial. This argues for a pragmatic approach. However, if the pilot data (n=20) is the only prior evidence, it is premature to run a fully pragmatic trial without first knowing whether the intervention works at all under reasonably controlled conditions. The question of whether must precede the question of how well in the real world.

Participant selection

Explanatory: Include only participants who have adequate digital literacy, stable clinical rotas, and no competing mandatory CPD. This produces a "best case" estimate of the platform's effect in willing, able participants.
Pragmatic: Include the full range of NHS healthcare professionals - variable digital skills, chaotic shift patterns, mandatory competing training - and allow the platform to be tested under the conditions it will actually face. If it fails here, it was never going to succeed at scale.
The pragmatic approach is more policy-relevant but requires a larger sample to detect effects diluted by non-engagement from the least tech-confident participants.

Intervention delivery

Explanatory: Training facilitators to a high standard, providing technical support, monitoring completion weekly, and intervening when participants fall behind. This ensures the intervention is delivered as intended (high fidelity), but it also means the trial is testing a version of the platform that requires resources not available in routine NHS deployment.
Pragmatic: Allow facilitators to deliver the platform with standard orientation only, mirroring what would actually happen if the platform were commissioned. If fidelity naturally deteriorates, this is part of the real-world effect.
Dr Foster must decide whether she is evaluating a platform or evaluating a platform plus a delivery infrastructure. These are different interventions with different scalability.

Outcome measurement

Explanatory: Pre-specified MCQ knowledge assessments, OSCE-style competency assessments, and blinded marking. These produce clean, objective data but require logistical effort that would not happen in routine deployment.
Pragmatic: Routine clinical competency records already collected by the trust (e.g., clinical supervision logs, workplace-based assessment scores, mandatory training completion records) as the primary outcome. These have ecological validity but are noisy, incomplete, and confounded by other factors.
A pragmatic approach might also incorporate patient-reported outcomes or clinical audit data (e.g., compliance with care bundle metrics), though attributing changes in these outcomes to CPD is methodologically challenging given the multiple other influences on clinical performance.

Analysis approach

Explanatory trials typically use per-protocol analysis as the primary analysis (to see what happens when the intervention is properly delivered) alongside ITT as a sensitivity analysis.
Pragmatic trials use ITT as the primary analysis, accepting that real-world partial engagement is part of the intervention's real-world effect.
For NHS funding purposes, ITT analysis is more credible because it reflects the effect of offering the platform, not the effect of completing it perfectly.

How this choice cascades through the entire design

The explanatory/pragmatic choice is genuinely fundamental because it determines:

Design element	Explanatory approach	Pragmatic approach
Eligibility criteria	Narrow, homogeneous	Broad, inclusive
Recruitment setting	Single specialist centre	Multiple NHS sites
Intervention delivery	Standardised, monitored	Flexible, locally adapted
Comparator	No treatment or waiting list	Usual CPD practice
Outcomes	MCQ tests, OSCE	Workplace performance, audit
Follow-up	Fixed protocol schedule	Routine record review
Primary analysis	Per-protocol	Intention-to-treat
Sample size	Smaller (homogeneous sample)	Larger (heterogeneous sample)

A Practical Recommendation for Dr Foster

Given that:

The pilot was only 20 participants with no control group
NHS Education requires evidence for policy change (implying a pragmatic answer is ultimately needed)
The research committee wants "definitive evidence" (suggesting internal validity is a priority)

...the most defensible approach is a hybrid efficacy-effectiveness design: an individually or cluster-randomised trial conducted across 2-3 NHS sites, with reasonably inclusive eligibility criteria (not just enthusiastic digital natives), an active comparator of usual CPD, ITT as the primary analysis, and a mixed outcome battery that includes both knowledge assessments (for interpretability) and routine workplace-based assessment data (for real-world relevance). This gives Dr Foster findings she can defend to both methodological reviewers and NHS commissioning bodies, and it mirrors the "Phase IIb" approach increasingly recommended in health services and education research.

Synthesis: Key Evidence Sources

Source	Relevance
Martinengo et al. (2024) [PMID 39388234] - J Med Internet Res, SR/meta-analysis	Spaced digital education produces moderate-certainty gains in knowledge (SMD 0.32) and clinical behaviour change (SMD 0.67); 83% of included studies had high/unclear bias risk, underscoring the need for Dr Foster's well-designed RCT
Williams et al. (2023) [PMID 37316431] - J Surg Educ, SR	E-learning CPD associated with satisfaction and knowledge gains up to Level 4 (Moore's Framework); no evidence yet for workplace performance or patient outcomes (Levels 5-7) - sets the challenge for Dr Foster's outcome selection
Cook & Reed (2015) - Academic Medicine	MERSQI/NOS-E tools show medical education research systematically weak on randomisation and comparison group design; provides the methodological benchmarks Dr Foster must meet
Friedman et al. (2015) - Fundamentals of Clinical Trials	Definitive reference for RCT principles: randomisation as control for unknown confounders, ITT analysis, allocation concealment, and the efficacy/effectiveness continuum
Curran et al. (PMC3731143) - Implement Sci	Effectiveness-implementation hybrid designs offer a staged framework bridging efficacy trial findings to real-world deployment - directly relevant to Dr Foster's phased strategy

This is a shared conversation. Sign in to Orris to start your own chat.