The differences and overlaps between ‘explanatory’ and ‘pragmatic’ controlled trials: a historical perspective

Abstract

In designing a controlled trial, a potential tension may exist between (1) providing insights into therapeutic mechanisms and (2) guiding decisions about ‘what works’ in everyday practice. In their 1967 paper in the Journal of Chronic Diseases, Schwartz and Lellouch¹ designated these aims as ‘explanatory’ and ‘pragmatic’, illustrating the potential tension with the different trial designs needed to assess the effect of an agent hypothesised to increase responsiveness to radiotherapy (a ‘radiosensitiser’) with immediate initiation of radiotherapy in the ‘pragmatic’ design versus a 30-day delay in initiating radiotherapy for the ‘explanatory’ design. This dilemma is manifest in real clinical situations, such as chemotherapy or radiotherapy prior to cancer surgery (‘neoadjuvant therapy’), ‘pre-habilitation’ to prepare patients for joint surgery and whether to use a placebo when comparing two versus four times daily drug dosing.

Most study design choices are not so clearly and unidimensionally ‘pragmatic’ or ‘explanatory’. The PRagmatic-Explanatory Continuum Indicator Summary (PRECIS) tool² was developed to provide a graphical assessment of this spectrum for 10 ‘domains’ that include eligibility criteria, flexibility of interventions, practitioner expertise, intensity of follow-up, compliance with treatments and scope of the analysis.

The ‘explanatory’ versus ‘pragmatic’ tensions can sometimes be resolved. We suggest three categories: (1) sometimes both ‘explanatory’ and ‘pragmatic’ objectives of a trial can be achieved by aligning the design to address both; (2) sometimes there are possible compromises, for example, using an ‘explanatory’ sub-study within the framework of a ‘pragmatic’ main trial; and (3) rarely, there are incompatible objectives as illustrated by Schwartz and Lellouch.¹

Both the ‘explanatory’ and ‘pragmatic’ aims of trials are important. In general, as research moves from an understanding of the mode of therapeutic action towards clinical application, studies move increasingly from the ‘explanatory’ mode to the ‘pragmatic’ mode. Recognising when and how to focus on ‘explanatory’ versus ‘pragmatic’ reasons for research will remain an important element in that judgement.

Introduction

Controlled trials are the principal means of investigating the effects of therapeutic and prophylactic interventions. They can be designed to avoid biases, and so provide clear and reliable insights into ‘what works’ under ideal circumstances. Controlled trials can, however, also generate evidence to inform decisions on the choice of methods to control biases in everyday circumstances. This distinction between different types of trials came to be characterised by using the terms ‘explanatory’ and ‘pragmatic’ to distinguish different types of trials.^1,2

The need for these or similar terms (Table 1) began to be appreciated during the 1950s when it was realised that different terms were needed to distinguish between trials being done with differing primary objectives. In 1959, under the aegis of WHO and UNICEF, the Council for International Organizations of Medical Sciences sponsored a landmark 5-day conference in Vienna on the role of controlled clinical trials.⁹

Table 1.

Characterisations of some different types of controlled trials, 1960–2009.

Article	Similar terms for ‘explanatory’	Similar terms for ‘pragmatic’
Bywaters³	Analysis based only on patients remaining in a trial, and on specified treatments at each assessment time point.	Analysis based on the group as a whole, excluding only those who could not be assessed due to death or non-attendance, irrespective of whether they have remained on the specified therapy.
Schwartz et al.⁴	Trials that are tightly controlled and have studied reactions to treatment in clearly delineated types of patients.	Trials with wide entry criteria, a consequence of which many patients will have to be excluded after random allocation.
Schwartz and Lellouch,¹ pp. 47	‘Explanatory’‘aimed at understanding …whether a difference exists between two treatments which are specified by strict and usually simple definition’.	‘Pragmatic’…aimed at decision-making …seeking to answer the question, which of two (flexible and complex) treatments should be preferred’.
Cochrane,⁵ pp. 2–3	Effectiveness‘… the effect of a particular medical action in altering the natural history of a particular disease for the better’ (therapy).	Efficiency… board and lodging and tender, loving, care’.
Office of Technology Assessment (1978).⁶ Assessing the Efficacy and Safety of Medical Technologies	EfficacyThe probability of benefit to individuals in a defined population from a medical technology applied for a given medical problem under ideal conditions of use.	EffectivenessA judgement of the acceptability of relative risk in a specified situation.
Sackett 20066.4 Explanatory vs. Management Trials. In Haynes et al.⁷ p 183	‘Explanatory’/efficacyYields biologically answered questions about the effects of drugs given in ideal circumstances.	Management/‘pragmatic’/effectivenessWill give clinically/community-relevant answers to questions when offered to a wide range of patients
Thorpe et al.²	‘Explanatory’‘Can this intervention work under ideal conditions?’ Trials primarily designed to determine the effects of an intervention under ideal circumstances.	‘Pragmatic’‘Does this intervention work under usual conditions?’ Trials primarily designed to determine the effects of an intervention under ideal circumstances
Karanicolas et al.⁸	MechanisticA trial is mechanistic to the extent that it addresses a biological relationship.	PracticalA trial is practical to the extent that it provides comprehensive information that bears directly on specific healthcare decisions.

Among the 23 papers presented at the conference was one by the British rheumatologist Eric Bywaters, who reported his experience dealing with the challenges presented by loss of trial participants over a three-year follow-up period after initial randomisation.³ Non-random dropouts and other losses typical of real-life trials raised concerns that bias would result and undermine the reliability of the trial outcome.

The challenges presented to clinical decision-making by this practical issue were raised in a ‘rapport interprétatif’ (in French) of the conference prepared by Daniel Schwartz, a senior statistician at Inserm in Paris, and colleagues.⁴ The Chapter V was dedicated to ‘Portée des conclusions’ (in French) and stressed that:

(…) il y a entre la définition de l’essai et la portée des conclusions un conditionnement étroit. Ce conditionnement est mutuel : si la portée des conclusions dépend de la définition de l’essai, celui-ci inversement doit être organisé de façon à répondre au problème posé.C’est sous cet angle qu’on devra considérer les divers choix qui constituent l’organisation de l’essai. Ainsi en est-il, par exemple, du choix toujours offert, au moment de définir la maladie ou les malades, entre l’homogénéité et l’hétérogénéité (…) les deux méthodes ont chacune un avantage et un inconvénient : à nombre égal de malades, l’examen d’un groupe homogène permet d’y voir plus clair et donne un résultat plus précis : mais ce résultat est moins général (..). C’est en fonction du problème posé – ou de conditions d’opportunité – que le choix devra être effectué.

[Translation: There is close conditioning between the definition of the trial and the scope of the conclusions. This conditioning is mutual: if the scope of the conclusions depends on the definition of the trial, the latter must conversely be organised in such a way as to respond to the problem posed.

It is from this angle that the various choices that constitute the organisation of the trial should be considered. For example, when defining the disease or the patients, there is always a choice between homogeneity and heterogeneity (…) each of the two methods has an advantage and a disadvantage: with the same number of patients, the examination of a homogeneous group makes it possible to see things more clearly and gives a more accurate result: but this result is less general (…). The choice will have to be made according to the problem posed – or the conditions of opportunity.]

In 1966, Marvin Schneiderman, a statistician who had been involved in US trials of cancer therapy in the early 1950s, more specifically suggested dual purposes of randomised controlled trials in a working paper prepared for a WHO Expert Committee on Cancer Treatment:

In [cancer] chemotherapy at least, there appear to be two different kinds of trials, conducted for distinctly different purposes. There are the patient-orientated trial, and the drug-orientated trial. Patient-orientated trials are designed to give answers to the question ‘How shall I treat the next patient with cancer who comes into my care?’ The drug-orientated trials attempt to answer the questions ‘Has this drug enough promise that I can bring it into patient-orientated trials?’ and ‘If I were to bring it into a patient orientated trial, how is it best to give it?’ (Schneiderman 1966,¹⁰ p. 5)

Though Schneiderman’s distinction between ‘drug-oriented trials’ and ‘patient-oriented trials’ uses outdated terminology, we might now see these are precursors to the distinction between phase 2 and phase 3 trials. Indeed, under Schneiderman’s later guidance, as Chief Statistician at the US National Cancer Institute, this division of the two types of trials into early and later stage trials led to the three phases of drug trial with which we are now familiar (see Zwarenstein: https://www.jameslindlibrary.org/articles/pragmatic-and-explanatory-attitudes-to-randomized-trials/).¹¹

Seven years after the Vienna meeting, Schwartz and his colleague Joseph Lellouch explored the issue of the different purposes of trials further, and coined the terms now widely used in clinical trial design: ‘explanatory’ trials were designed to provide understanding of the mechanisms through which treatments might have their effects, and ‘pragmatic’ trials were designed to inform real-life decision-making.¹ While this distinction has some overlaps with Schneiderman’s distinction between ‘drug-oriented trials’ and ‘patient-oriented trials’, as the ‘explanatory’ versus ‘pragmatic’ distinction is more encompassing, both in terms of the types of intervention considered (not only drugs), and in the implications for design and analysis of trials.

In what follows, we describe the evolution of this distinction and the recognition that it does not reflect a simple dichotomy but forms a continuum. Its importance lies in how decisions made at the design stage affect where a trial lies on the several dimensions of the ‘pragmatic’–‘explanatory’ continuum.² This in turn has a direct bearing on whether the trial can give reliable and relevant answers to the questions being asked of it.

As such, the distinction has an important if largely unrecognised role in the debate about waste in clinical research.¹² In recent years, systematic methods for determining whether a specific trial design is appropriate for its intended purpose have been proposed.^2,13 These have served to underline the importance and subtlety of the distinction made by Schwartz and Lellouch over half a century ago, the implications of which remain a focus of active research, such as surveys of how trials labelled as ‘pragmatic’ differ from others.^14,15

The original statement of the distinction

In their 1967 paper in the Journal of Chronic Diseases, Schwartz and Lellouch¹ first state (p. 638) their chosen terms for what they held to be two contrasting trial types. As noted by Armitage,¹⁶ trialists had already noted the potential tension in designing a single trial that could provide reliable insights into therapeutic mechanisms while also guiding decisions about ‘what works’ in everyday circumstances. Schwartz and Lellouch characterised the two types of trials as follows:

The first type corresponds to an ‘ explanatory’ approach, aimed at understanding . It seeks to discover whether a difference exists between two treatments which are specified by strict and usually simple definitions. Their effects are assessed by biologically meaningful criteria, and they are applied to a class of patients which is rather arbitrarily defined, but which is as likely as possible to reveal any difference that may exist.

The second type corresponds to a ‘ pragmatic’ approach, aimed at decision . It seeks to answer the question – which of the two treatments should we prefer? The definition of the treatments is flexible and usually complex: it takes account of auxiliary treatments and of the possibility of withdrawals. The criteria by which the effects are assessed take into account the interests of patients and the costs in the widest sense. The class of patients is predetermined as that to which the results of the trial are to be extrapolated.

To illustrate how this distinction impacts trial design, Schwartz and Lellouch gave as an example a comparison of immediate initiation of radiotherapy for treating a newly diagnosed cancer versus delayed initiation of radiotherapy for 30 days during which the patient would take a ‘radiosensitiser’ – a drug believed to increase tumour-sensitivity and responsiveness to radiotherapy. Figure 1 (modified from the original) shows how the design of the trial thus depends on its aims. This difference in design prevents the two trials being equally reliable in resolving scientific and practical questions, as the confounding effect of the delay in giving the radiotherapy is unknown and could affect the outcome in different ways. The two-arm trials in Figure 1 cannot answer both the ‘explanatory’ and ‘pragmatic’ questions, though a three-arm trial (incorporating both ‘standard’ arms) could address both questions.

Figure 1.

Does a radiosensitiser help? The ‘pragmatic’ design (left) would compare immediate radiotherapy to radiosensitiser then (delayed) radiotherapy; the ‘explanatory’ design (right) would also delay radiotherapy in the standard group to match the radiotherapy + radiosensitiser group, removing ‘delay’ as an ‘explanatory’ factor for differences in effect (but the ethics of any such delay will need to be considered – see section on examples).

Schwartz and Lellouch summarised the distinction as follows:

We may say in general that the ‘explanatory’ approach will always give an answer to the scientific problem but only sometimes to the problem of immediate practicality (depending on the result of the trial); while the reverse is true for the ‘pragmatic’ approach.

Three years after their 1967 article, Schwartz and his colleagues expanded on the distinction between these two types of trials in a book entitled ‘L’essai thérapeutique chez l’homme’,⁴ subsequently translated into English by the British statistician Michael Healy (1980).¹⁷

Examples of the ‘explanatory’ vs ‘pragmatic’ dilemma

Schwartz and Lellouch used a hypothetical example to illustrate a dilemma in addressing ‘explanatory’ and ‘pragmatic’ trial designs concurrently (Figure 1). However, the dilemma is manifested in multiple real clinical situations. Two actual clinical problems with similar timing, structure and dilemma are: (1) whether to give chemotherapy or radiotherapy or neither prior to cancer surgery;¹⁸ and (2) whether or not to use ‘pre-habilitation’ to prepare patients for joint surgery.¹⁹

In the first example, chemotherapy can be given for several weeks prior to surgery to reduce ‘tumour load’ (known as neoadjuvant therapy, which can also be radiotherapy), but this will obviously delay surgery. Should trials of chemotherapy versus no chemotherapy prior to surgery use the ‘explanatory’ approach by arranging equal delay in the two comparison arms, or the ‘pragmatic’ approach by using immediate surgery for those not allocated to chemotherapy? In a tabulation of studies of short-course versus long-course radiotherapy (Koukourakis¹⁸ and Supplementary Figure 2b), both designs occurred: some studies used equal delay in both arms, and others began surgery immediately upon completion of short duration radiotherapy.

A similar dilemma arises with the second example – whether or not to use ‘pre-habilitation’ to prepare the patient physically and psychologically for joint surgery or replacement.¹⁹ Analogous to the preoperative chemotherapy example, the dilemma for those designing trials is whether to arrange for an equal delay in both arms, or to have surgery as soon as possible (Supplementary Figure 2c). Often, this dilemma is avoided by treating patients on long surgical waitlists, so waiting time is equalised, which, in effect, is the equivalent of the ‘explanatory’ design.

Delaying treatment is just one of several examples illustrating the challenge of designing studies addressing ‘explanatory’ and ‘pragmatic’ trial analyses concurrently. For example, a common problem in drug dosing is whether to prescribe medication once, twice or thrice daily. Less frequent medication may result in better adherence to assigned treatment; more frequent medication would tend to result in more stable drug levels.

Figure 2 shows options for comparing twice daily with four times daily antibiotics with two options for the twice daily design.²⁰ Option A does not use a placebo but only simple twice daily administration. With this comparison, the clinical impact will reflect both the biological effect and the effects of differences in adherence, but we cannot know the relative contributions of these two effects. Option B replaces every second dose of the drug with a placebo. This option addresses the ‘explanatory’ effect, namely the value of steadier versus more intermittent drug levels by using placebo to remove the adherence effect (Supplementary Figure 2c). We cannot answer both questions with a two-arm trial, but we can do this by using all three arms, a potential solution to the challenges of applying ‘explanatory’ and ‘pragmatic’ designs concurrently.

Figure 2.

Twice daily dosing options: ‘pragmatic’ (option A) or ‘explanatory’ (option B).

The emergence of alternative terms

Most study design choices are not so clearly ‘pragmatic’ or ‘explanatory’ as the example given by Schwartz and Lellouch. Moreover – and as so often in the terminology of trial design – their chosen nomenclature can be criticised as failing to capture the nuances of the concepts involved. Over the decades, this has led to the emergence of alternative terms for the same basic idea (Table 1), among them those proposed by Cochrane,⁵ the US Congress Office of Technology Assessment (1978) and Sackett (2006). Nevertheless, the original choice of ‘explanatory’ and ‘pragmatic’ appears to have become the de facto nomenclature, and we will use these terms as the default during the rest of this article.

From simple dichotomy to continuum

In their seminal 1967 article, Schwartz and Lellouch begin by stating their belief that ‘most therapeutic trials are inadequately formulated… in that the trials may be aimed at the solution of one or other of two radically different kinds of problem’. [emphasis added]. In their summary section, the authors qualified this apparent dichotomy, stating that, with the comparison of two treatments, ‘the “explanatory” approach will always give an answer to the scientific problem but only sometimes to the problem of immediate practicability (depending on the result of the trial); while the reverse is true for the “pragmatic” approach’. By the close of the article, they had moved away from so simple a classification and towards a continuum, stating that ‘Most real problems contain both “explanatory” and “pragmatic” elements, for ethical reasons’.

Armitage elaborated on this in a 1998 article¹⁶ stating:

It may be more realistic to suggest that the two attitudes are likely to co-exist, and to compete for ascendency, in any one trial. The investigators planning a trial may well wish to ‘kill two birds with one stone’ to gain information about biological mechanisms, and to assess strategies of treatment.

Perhaps, surprisingly, given its obvious importance, practical guidance on ensuring that a trial design was ‘fit for purpose’ only emerged half a century after the Vienna conference. It took the form of a means of assessing a given design according to criteria which collectively capture the degree to which a trial is ‘pragmatic’ or ‘explanatory’ in nature. Devised and reported by Thorpe et al.,² the PRECIS tool leads to a graphical assessment based on 10 ‘domains’:

The eligibility criteria of trial participants.

The flexibility with which the experimental intervention is applied.

The degree of practitioner expertise in applying and monitoring the experimental intervention.

The flexibility with which the comparison intervention is applied.

The degree of practitioner expertise in applying and monitoring the comparison intervention.

The intensity of follow-up of trial participants.

The nature of the trial’s primary outcome.

The intensity of measuring participants’ compliance with the prescribed intervention, and whether compliance-improving strategies have been used.

The intensity of the measurement of practitioners’ adherence to the study protocol, and whether adherence-improving strategies are used.

The specification and scope of the analysis of the primary outcome.

Squaring the circle of ‘pragmatic’ and ‘explanatory’ designs

The domains suggested by Thorpe et al.² open up the possibility of trial designs that can achieve both ‘explanatory’ and ‘pragmatic’ objectives.

Aligned

Sometimes both ‘explanatory’ and ‘pragmatic’ objectives of a trial can be achieved by aligning the design to address both. For example, large sample sizes and strict adherence to randomly assigned treatment are desirable for both ‘explanatory’ and ‘pragmatic’ aims. An ‘explanatory study’ may accept a somewhat smaller sample size aimed at improving adherence, for example, by using a ‘run in period’ to check for tolerance or adherence to treatment (the FIELD trial of 10,000 randomised participants²¹ did this, so it is not just for small trials). But such run-ins would also be acceptable for analysing the pragmatic component of the study.

Compromise needed and possible

Sometimes there are possible compromises. For example, the ‘pragmatic’ trial might be large and simple, but use a sub-study with greater measurement options for achieving some ‘explanatory’ components (e.g. the LIPID trial with 9000 randomised²² had a carotid substudy of 400 patients to look at the effect of statins on carotid thickening). Similarly, the large coronavirus disease 2019 vaccine trials had a broad population with a simple intervention,²³ but most of these also included some ‘explanatory’ elements, such as identifying asymptomatic cases or taking blood for assessing antigenic response in a subsample of people.

Incompatible objectives

Sometimes pursuit of ‘explanatory’ and ‘pragmatic’ trial objectives concurrently within the same trial is impossible. The radiosensitiser example (see above) illustrates this problem. As noted earlier, for that example, no two-arm trials (Figure 1) can address and answer both the ‘explanatory’ and ‘pragmatic’ questions, though a three-arm trial (incorporating both ‘standard’ arms) could. Irrespective of size, inclusion criteria, etc. a single two-arm trial design cannot address both the ‘explanatory’ (biological) and the ‘pragmatic’ (decisional) conceptualisation (for more details, see Zwarenstein: https://www.jameslindlibrary.org/articles/pragmatic-and-explanatory-attitudes-to-randomized-trials/).¹¹

Relationship to intention-to-treat analysis

‘Pragmatic’ and ‘explanatory’ are different facets of the design and analysis of trials. They do not necessarily compete but can co-exist in the same trial. The ‘pragmatic’ and ‘explanatory’ aims are about decisions to be taken at or before the start of a trial about the questions and the purposes of the investigation. The purpose of ‘pragmatic’ trials is about the decision, often for a broad group of patients, and who might benefit by that intervention. By contrast, the ‘explanatory’ route focuses on understanding mechanisms, and often requires highly selected patients whose condition is well understood and might benefit or become clearer by a treatment comparison. Both types of trials might require an intention-to-treat analysis, although this would be more likely to be needed in a pragmatic trial than in an ‘explanatory’ trial because of the heterogeneity of the patients in the former group.

In their highly cited paper, Thorpe et al.² note that, whether a trial is characterised as ‘explanatory’ or ‘pragmatic’, the primary, unbiased analyses should be by ‘intention to treat’ (see article by Chalmers et al.²⁴).

Thus, for analyses of ‘ pragmatic’ trials , ‘the primary outcome should include all patients, regardless of eligibility and compliance, etc. (“intention-to-treat” analysis). In other words, the analysis attempts to see if the treatment works under usual conditions, with all the “noise” inherent therein.’

For analyses of ‘ explanatory’ trials , ‘an “intention-to-treat” analysis is usually performed. However, this may be supplemented by secondary analyses, for example, an analysis restricted to “compliers” or other subgroups in order to estimate maximum achievable treatment effect. Analyses are conducted that attempt to answer the narrowest, “mechanistic” question (whether biological, educational or organizational’ (Thorpe et al.,² p. E48). If estimates of treatment effects from non-ITT and ITT analyses are indistinguishable it provides some reassurance that residual biases are likely to be small.

Additional analyses, such as adjustments for non-compliance, may help to explain trial results, but the confusing term ‘per-protocol’ should be avoided.

In a 1998 paper, Armitage¹⁶ suggested that the ‘explanatory’ effect, which, if present, should occur with 100% compliance, might be best done by one of several modelling approaches which adjust for non-compliance rather than dismissing the data of the non-compliant. Since then, there have been many other such approaches developed, which use the Intention-To-Treat (ITT) analysis but make an adjustment for non-compliance. Unlike a so-called ‘per-protocol’ analysis, these compliance-adjusted ITT estimates will not create an effect when there is none.

Conclusions

As randomised trials gained acceptance in the 1950s, some recognition emerged that trials may have different purposes and hence different design requirements. For example, Schneiderman identified the different requirements for examining the biological activity of a drug versus the assessment of its value in treating patients. This recognition resulted in the different aims and designs of phase 2 and phase 3 clinical trials. The 1967 paper by Schwartz and Lellouch drew the key distinction between the ‘explanatory’ and ‘pragmatic’ aims of a trial.

Both the ‘explanatory’ and ‘pragmatic’ aims are important. In general, as research moves from an initial understanding of a potential therapy towards its clinical application with patients, studies move increasingly from the ‘explanatory’ mode to the ‘pragmatic’ mode. That is, research moves progressively from the small ‘explanatory’ trial, which measures multiple surrogate outcomes to examine biological activity, to the larger scale simple ‘pragmatic’ trial, which focuses much more on patient-relevant outcomes, both beneficial and harmful. As we have set out in the later sections of this article, the ‘explanatory’ and ‘pragmatic’ tensions can sometimes be resolved, but as Schwartz and Lellouch illustrated with their radiotherapy example, occasionally the two aims are irreconcilable.

Since this watershed recognition, there have been ongoing developments. For example, Thorpe et al.² describe multiple domains, and how within each domain there is a spectrum of ‘explanatory’ to ‘pragmatic’ characteristics. As we adopt trial designs beyond the simple parallel two-group trial – such as factorial, platform and adaptive trials – the ‘explanatory’ versus ‘pragmatic’, distinction will undoubtably evolve further and be recognised as a key feature in design. That recognition would be an important step to achieving Doug Altman’s famous wish for ‘… less research, better research and research done for the right reasons’.²⁵ Recognising when and how to focus on ‘explanatory’ versus ‘pragmatic’ reasons for research will be an important element in that endeavour.

Footnotes

Declarations

References

Schwartz

Lellouch

Explanatory and pragmatic attitudes in therapeutic trials. J Chronic Dis 1967; 20: 637–648.

Thorpe

Zwarenstein

Oxman

Treweek

Furberg

Altman

, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ 2009; 180: E47–E57.

Bywaters

Rheumatoid arthritis. Treatment and illustrative answers. In: Hill

, ed. Controlled Clinical Trials. Oxford: Blackwell, 1960:75–83.

Schwartz

Flamant

Lellouch

Rouquette

Les essais thérapeutiques cliniques. Paris: Masson, 1960.

Cochrane

AL.

Effectiveness and efficiency: random reflections on health services. Nuffield Provincial Hospital Trust, 1972. See https://www.nuffieldtrust.org.uk/research/effectiveness-and-efficiency-random-reflectionson-health-services (last checked 4 October 2023).

US Congress Office of Technology Assessment. Assessing the efficacy and safety of medical technologies. Washington, DC: US Government Printing Office; 1978.

Haynes

Sackett

Guyatt

, et al. Clinical epidemiology: how to do clinical practice research. 3rd edition. Philadelphia, PA: Lippincott Williams & Wilkins, 2006.

Karanicolas

Montori

Devereaux

Schünemann

Guyatt

GH.

A new 'mechanistic-practical’ framework for designing and interpreting randomized trials. J Clin Epidemiol 2009; 62: 479–484.

Bird

SM.

The 1959 meeting in Vienna on controlled clinical trials – a methodological landmark. 2014. JLL Bulletin: commentaries on the history of treatment evaluation. See www.jameslindlibrary.org/articles/the-1959-meeting-in-vienna-on-controlled-clinical-trials-a-methodological-landmark/ (last checked 4 October 2023).

10.

Schneiderman

. Therapeutic trials in cancer. Working paper prepared for WHO Expert Committee on Cancer Treatment, Geneva, Switzerland, 9–15 March 1965. WHO/CANC/66.66, 1966.

11.

Zwarenstein

‘ Pragmatic’ and ‘explanatory’ attitudes to randomized trials. JLL Bulletin: commentaries on the history of treatment evaluation. 2016. See www.jameslindlibrary.org/articles/pragmatic-and-explanatory-attitudes-to-randomized-trials/ (last checked 4 October 2023). Also as Zwarenstein M. ‘Pragmatic’ and ‘explanatory’ attitudes to randomised trials. J R Soc Med 2017; 110: 208–218.

12.

Chalmers

Glasziou

Avoidable waste in the production and reporting of research evidence. Lancet 2009; 374: 86–89.

13.

Loudon

Treweek

Sullivan

Donnan

Thorpe

Zwarenstein

The PRECIS-2 tool: designing trials that are fit for purpose. BMJ 2015; 350: h2147.

14.

Palakshappa

Gibbs

Lannan

Cranford

Taylor

SP.

Systematic review of the “pragmatism” of pragmatic critical care trials. Crit Care Explor 2022; 4: e0738.

15.

Taljaard

Nicholls

Howie

Nix

Carroll

Moon

, et al. An analysis of published trials found that current use of pragmatic trial labels is uninformative. J Clin Epidemiol 2022; 151: 113–121.

16.

Armitage

Attitudes in clinical trials. Stat Med 1998; 17: 2675–2683.

17.

Schwartz

Flamant

Lellouch

Clinical Trials. Translated by MJR Healy. London: Academic Press, 1980.

18.

Koukourakis

Which is the best neoadjuvant (pre-surgery) chemoradiation regimen for locally advanced rectal carcinoma? Short or long course of radiation therapy? Do we have new data?

J BUON 2020; 25: 51–61.

19.

McIsaac

Gill

Boland

Hutton

Branje

Shaw

, et al.; Prehabilitation Knowledge Network. Prehabilitation in adult patients undergoing surgery: an umbrella review of systematic reviews. Br J Anaesth 2022; 128: 244–257.

20.

Raz

Elchanan

Colodner

Reiss

Schvartzman

Tabenkin

Leshem

. Penicillin V twice daily vs. four times daily in the treatment of streptococcal pharyngitis. Infect Dis Clin Practice 1995; 4: 50–54.

21.

FIELD Study Investigators. The need for a large-scale trial of fibrate therapy in diabetes: the rationale and design of the Fenofibrate Intervention and Event Lowering in Diabetes (FIELD) study [ISRCTN64783481]. Cardiovasc Diabetol 2004; 3: 9.

22.

MacMahon

Sharpe

Gamble

Hart

Scott

Simes

, et al. Effects of lowering average of below-average cholesterol levels on the progression of carotid atherosclerosis: results of the LIPID Atherosclerosis Substudy. LIPID Trial Research Group. Circulation 1998; 97: 1784–1790.

23.

Voysey

Clemens

SAC

Madhi

Weckx

Folegatti

Aley

, et al.; Oxford COVID Vaccine Trial Group. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK. Lancet 2021; 397: 99–111.

24.

Chalmers

Matthews

Glasziou

Boutron

Armitage

Analysis of clinical trial by treatment allocated or by treatment received? Applying ‘the intention-to-treat principle’. 2023. JLL Bulletin: commentaries on the history of treatment evaluation. See www.jameslindlibrary.org/articles/analysis-of-clinical-trial-by-treatment-allocated-or-by-treatment-received-applying-the-intention-to-treat-principle/ (last checked 4 October 2023).

25.

Altman

DG.

The scandal of poor medical research. BMJ 1994; 308: 283–284.