Clinical Trials

Biomarkers have many distinct purposes, and depending on their intended use, the validation process varies substantially.

The goal of this article is to provide an introduction to the topic of biomarkers, and then to discuss three specific types of biomarkers, namely, prognostic, predictive, and surrogate.

A principle challenge for biomarker validation from a statistical perspective is the issue of multiplicity. In general, the solution to this multiplicity challenge is well known to statisticians: pre-specification and replication. Critical requirements for prognostic marker validation include uniform treatment, complete follow-up, unbiased case selection, and complete ascertainment of the many possible confounders that exist in the context of an observational sample. In the case of predictive biomarker validation, observational data are clearly inadequate and randomized controlled trials are mandatory. Within the context of randomization, strategies for predictive marker validation can be grouped into two categories: retrospective versus prospective validation. The critical validation criteria for a surrogate endpoint is to ensure that if a trial uses a surrogate endpoint, the trial will result in the same inferences as if the trial had observed the true endpoint. The field of surrogate endpoint validation has now moved to the multi-trial or meta-analytic setting as the preferred method.

Biomarkers are a highly active research area. For all biomarker developmental and validation studies, the importance of fundamental statistical concepts remains the following: pre-specification of hypotheses, randomization, and replication. Further statistical methodology research in this area is clearly needed as we move forward.

Many articles have been published in biomedical journals reporting on the development of prognostic and therapy-guiding biomarkers or predictors developed from high-dimensional data generated by omics technologies. Few of these tests have advanced to routine clinical use.

We discuss statistical issues in the development and evaluation of prognostic and therapy-guiding biomarkers and omics-based tests.

Concepts relevant to the development and evaluation of prognostic and therapy-guiding clinical tests are illustrated through discussion and examples. Some differences between statistical approaches for test evaluation and therapy evaluation are explained. The additional complexities introduced in the evaluation of omics-based tests are highlighted.

Distinctions are made between clinical validity of a test and clinical utility. To establish clinical utility for prognostic tests, it is explained why absolute risk should be evaluated in addition to relative risk measures. The critical role of an appropriate control group is emphasized for evaluation of therapy-guiding tests. Common pitfalls in the development and evaluation of tests generated from high-dimensional omics data such as model overfitting and inappropriate methods for test performance evaluation are explained, and proper approaches are suggested.

The cited references do not comprise an exhaustive list of useful references on this topic, and a systematic review of the literature was not performed. Instead, a few key points were highlighted and illustrated with examples drawn from the oncology literature.

Approaches for the development and statistical evaluation of clinical tests useful for predicting prognosis and selecting therapy differ from standard approaches for therapy evaluation. Proper evaluation requires an understanding of the clinical setting and what information is likely to influence clinical decisions. Specialized expertise relevant to building mathematical predictor models from high-dimensional data is helpful to avoid common pitfalls in the development and evaluation of omics-based tests.

Biomarker assays can be evaluated for analytical performance (ability of assay to measure the biomarker quantity) and clinical performance (ability of assay result to inform of the clinical condition of interest). Additionally, a biomarker assay is said to have clinical utility if it ultimately improves patient outcomes when used as intended.

This article reviews analytical and clinical performance studies of biomarker assay tests and also some designs of clinical utility studies.

Appropriate design and statistical analysis of analytical and clinical evaluation studies depend on the intended clinical use of the test. Some key aspects to valid performance studies include using subjects who are independent of those used to develop the test, masking users of the test to any other available test or reference results, and including in the primary statistical analysis subjects with unavailable results in an intention-to-diagnose analysis. Ingenuity in study design and analysis may be required for efficient and unbiased estimation of performance.

Performance studies need to be carefully planned as they can be prone to many sources of bias. Analytical inaccuracy can hamper the clinical performance of biomarkers.

As biomedical research and technology advance, challenges in study design and statistical analysis will continue to emerge for analytical and clinical performance studies of biomarker assays. Although not emphasized in some circles, the analytical performance of a biomarker assay is important to characterize. Analytical performance studies have many study design and statistical analysis challenges that deserve further attention.

Accurate risk prediction plays a key role in disease prevention and disease management; emergence of new biomarkers may lead to an important question about how much improvement in prediction accuracy it would achieve by adding the new markers into the existing risk prediction tools.

In large prospective cohort studies, the standard full-cohort design, requiring marker measurement on the entire cohort, may be infeasible due to cost and low rate of the clinical condition of interest. To overcome such difficulties, nested case–control (NCC) studies provide cost-effective alternatives but bring about challenges in statistical analyses due to complex data sets generated.

To evaluate prognostic accuracy of a risk model, Cai and Zheng proposed a class of nonparametric inverse probability weighting (IPW) estimators for accuracy measures in the time-dependent receiver operating characteristic curve analysis. To accommodate a three-phase NCC design in Nurses’ Health Study, we extend the double IPW estimators of Cai and Zheng to develop risk prediction models under time-dependent generalized linear models and evaluate the incremental values of new biomarkers and genetic markers.

Our results suggest that aggregating the information from both the genetic markers and biomarkers substantially improves the accuracy for predicting 5-year and 10-year risks of rheumatoid arthritis.

Our method provided robust procedures to evaluate the incremental value of new biomarkers allowing for complex sampling designs.

It has become commonplace to use receiver operating curve (ROC) methodology to evaluate the incremental predictive accuracy of new markers in the presence of existing predictors. However, concerns have been raised about the validity of this practice. We have evaluated this issue in detail.

Simulations have been used that show clearly that use of risk predictors from nested models as data in subsequent tests comparing areas under the ROC curves of the models leads to grossly invalid inferences. Careful examination of the issue reveals two major problems: (1) the data elements are strongly correlated from case to case and (2) the model that includes the additional marker has a tendency to interpret predictive contributions as positive information regardless of whether observed effect of the marker is negative or positive. Both of these phenomena lead to profound bias in the test.

We recommend strongly against the use of ROC methods derived from risk predictors from nested regression models to test the incremental information of a new marker.

Serum creatinine has been used as the diagnostic test for acute kidney injury (AKI) for decades despite having imperfect sensitivity and specificity. Novel tubular injury biomarkers may revolutionize the diagnosis of AKI; however, even if a novel tubular injury biomarker is 100% sensitive and 100% specific, it may appear inaccurate when using serum creatinine as the gold standard.

In general, the apparent diagnostic performance of a biomarker depends not only on its ability to detect injury but also on disease prevalence and the sensitivity and specificity of the imperfect gold standard. Apparent errors in diagnosis using a new biomarker may be a reflection of errors in the imperfect gold standard itself rather than poor performance of the biomarker.

The development of personalised (stratified) medicine is intrinsically dependent on an understanding of treatment-effect mechanisms (effects on therapeutic targets that mediate the effect of the treatment on clinical outcomes). There is a need for clinical trial data for the joint evaluation of treatment efficacy, the utility of predictive markers as indicators of treatment efficacy, and the mediational mechanisms proposed as the explanation of these effects.

(1) To review the problem of confounding (common causes) for the drawing of valid inferences concerning treatment-effect mechanisms, even when the data have been generated using a randomised controlled trial, and (2) to suggest and illustrate solutions to this problem of confounding.

We illustrate the potential of the predictive biomarker stratified design, together with baseline measurement of all known prognostic markers, to enable us to evaluate both the utility of the predictive biomarker in such a stratification and, perhaps more importantly, to estimate how much of the treatment’s effect is actually explained by changes in the putative mediator. The analysis strategy involves the use of instrumental variable (IV) regression, using the treatment by predictive biomarker interaction as an IV – a refined, much more powerful, and (in the present context) subtle use of Mendelian randomisation.

Personalised (stratified) medicine and treatment-effect mechanisms evaluation are inextricably linked. Stratification without corresponding mechanisms evaluation lacks credibility. In the presence of mediator-outcome confounding, mechanisms evaluation is dependent on stratification for its validity. Both stratification and treatment-effect mediation can be evaluated using a biomarker stratified trial design together with detailed baseline measurement of all known prognostic biomarkers and other prognostic covariates. Direct and indirect (mediated) effects should be estimated through the use of IV methods (the IV being the predictive marker by treatment interaction) together with adjustments for all known prognostic markers (confounders) – the latter adjustments contributing to increased precision (as in a conventional analysis of treatment effects) rather than bias reduction.

In oncology, the treatment paradigm is shifting toward personalized medicine, where the goal is to match patients to the treatments most likely to deliver benefit. Treatment effects in various subpopulations may provide some information about treatment effects in other subpopulations.

We compare different approaches to Phase II trial design where a new treatment is being investigated in several groups of patients. We compare considering each group in an independent trial to a single trial with hierarchical modeling of the patient groups.

We assume four patient groups with different background response rates and simulate operating characteristics of three trial designs, Simon’s Optimal Two-Stage design, a Bayesian adaptive design with frequent interim analyses, and a Bayesian adaptive design with frequent interim analyses and hierarchical modeling across patient groups.

Simon’s designs are based on 10% Type I and Type II error rates. The independent Bayesian designs are tuned to have similar error rates, but may have a slightly smaller mean sample size due to more frequent interim analyses. Under the null, the mean sample size is 2–4 patients smaller. A hierarchical model across patient groups can provide additional power and a further reduction in mean sample size. Under the null, the addition of the hierarchical model decreases the mean sample size an additional 4–7 patients in each group. Under the alternative hypothesis, power is increased to at least 98% in all groups.

Hierarchical borrowing can make finding a single group in which the treatment is promising, if there is only one, more difficult. In a scenario where the treatment is uninteresting in all but one group, power for that one group is reduced to 65%. When the drug appears promising in some groups and not in others, there is potential for borrowing to inflate the Type I error rate.

The Bayesian hierarchical design is more likely to correctly conclude efficacy or futility than the other two designs in many scenarios. The Bayesian hierarchical design is a strong design for addressing possibly differential effects in different groups.

Well-conducted, investigator-led randomized controlled trials (RCTs) are the gold standard for evaluating the efficacy of new treatments and are a key component of evidence-based medicine. It is unclear whether participating in an RCT is beneficial to the individual before the results of RCTs are known.

In a matched historical cohort study, we examined whether participation in RCTs was associated with improved health outcomes.

Participants in the Action in Diabetes and Vascular Disease: Preterax and Diamicron MR Controlled Evaluation (ADVANCE), Ongoing Telmisartan Alone and in Combination with Ramipril Global Endpoint Trial (ONTARGET), or Telmisartan Randomized Assessment Study in ACE-intolerant Subjects with Cardiovascular Disease (TRANSCEND) studies and non-participant controls were selected from patients attending outpatient clinics at Middlemore Hospital between 2001 and 2003.

A total of 251 RCT participants and 502 randomly selected patients not enrolled in a trial but who met study entry criteria were matched for age, gender, and ethnicity. There was a significant difference in all-cause mortality for trial participants versus non-participants over the study period (unadjusted relative risk reduction (RRR) = 63%; 95% confidence interval (CI) = 28%–81%) and a significant reduction in cardiovascular mortality (unadjusted RRR = 81%; 95% CI = 17%–95%) favouring RCT participants. Allowing for co-morbidity, the adjusted RRR of all-cause mortality associated with trial participation was 55% (95% CI = 10%–77%). Active treatment in an RCT was found to be less explanatory than trial participation. The adjusted RRR for cardiovascular mortality associated with active treatment in a trial was 86% (95% CI = −2% to 98%), with trial participation found to be less explanatory than active treatment.

The main limitations of this trial relate to its design as a retrospective study with a historical cohort comparison group. Limitations include lack of complete data for some patients, bias in selection of the comparison group, and the effects of confounding variables. However, the study design and analysis were planned so as to minimize these as much as possible.

This study revealed significantly lower all-cause mortality among participants in industry-sponsored RCTs compared with non-participants who received routine hospital outpatient care. This effect was independent of study drug.

With blinded data, several authors have concluded that there is a negligible chance of inferring a non-null treatment effect. The recent Food and Drug Administration (FDA) draft guidance document on adaptive trials, by encouraging blinded sample size reestimation, implies the same.

We derive methods to investigate whether the probability of inferring a treatment effect is much larger than previously thought, and whether that is of concern.

A statistic is developed that contributes to improving signal detection. Additionally, trials that are overpowered, for reasons external to powering the primary objective, further strengthen the chance of finding a signal.

An example of data from a clinical trial shows how revealing a blinded analysis can be. The ability to infer a non-null effect while a blinded trial is ongoing is a serious matter.

The methods apply to superiority trials and are of limited use for non-inferiority or equivalence trials.

It is important, therefore, that guidance documents include clear language to limit or prevent inference from blinded data to maintain trial integrity. Simple steps are proposed to make inference difficult.

Measurement error in time-to-event end points complicates interpretation of treatment effects in clinical trials. Non-differential measurement error is unlikely to produce large bias [1]. When error depends on treatment arm, bias is of greater concern. Blinded-independent central review (BICR) of all images from a trial is commonly undertaken to mitigate differential measurement-error bias that may be present in hazard ratios (HRs) based on local evaluations. Similar BICR and local evaluation HRs may provide reassurance about the treatment effect, but BICR adds considerable time and expense to trials.

We describe a BICR audit strategy [2] and apply it to five randomized controlled trials to evaluate its use and to provide practical guidelines. The strategy requires BICR on a subset of study subjects, rather than a complete-case BICR, and makes use of an auxiliary-variable estimator.

When the effect size is relatively large, the method provides a substantial reduction in the size of the BICRs. In a trial with 722 participants and a HR of 0.48, an average audit of 28% of the data was needed and always confirmed the treatment effect as assessed by local evaluations. More moderate effect sizes and/or smaller trial sizes required larger proportions of audited images, ranging from 57% to 100% for HRs ranging from 0.55 to 0.77 and sample sizes between 209 and 737.

The method is developed for a simple random sample of study subjects. In studies with low event rates, more efficient estimation may result from sampling individuals with events at a higher rate.

The proposed strategy can greatly decrease the costs and time associated with BICR, by reducing the number of images undergoing review. The savings will depend on the underlying treatment effect and trial size, with larger treatment effects and larger trials requiring smaller proportions of audited data.

Institutional Review Boards (IRBs) have been increasingly criticized for how they review protocols, but how IRBs perceive, and make decisions about, the quality of the science of protocols has not been examined.

To explore how and when IRBs view and make decisions about the quality of the science of studies they review.

I contacted the leadership of 60 IRBs (every fourth one in the list of the top 240 institutions by National Institutes of Health (NIH) funding) and interviewed IRB chairs, co-chairs, administrators, and a director from 34 IRBs (response rate = 55%), and an additional 7 members.

Interviewees faced several ambiguities and questions concerning the quality of the science of protocols. IRBs are often not sure how and to what extent to evaluate the science of protocols, whether the science should be ‘good enough’ (and if so, what that means) versus as good as possible. Federal regulations state that IRBs should ensure that risks are minimized, and commensurate with benefits. Thus, at times IRBs feel that changing the science is ethically necessary. But IRBs also then struggle with whether to adopt a higher threshold (1) that social and thus scientific benefits be maximized and (2) that scientific efforts and resources should not be wasted. Committees face dilemmas – for example, if a ‘perfect’ study is not feasible. For protocols already approved elsewhere (e.g., by the NIH), IRBs vary in how much they feel they can request alterations, and sometimes make changes nonetheless. Larger institutional contexts and biases can shape these issues, and IRBs differ in how much they are ‘pro-research’, and have sufficient expertise. IRBs at times also approve studies despite reservations about the science.

This study includes interviews with IRBs, but not observations of IRB meetings.

IRBs often face ambiguities and conflicting goals in assessing scientific quality. Many IRBs try to improve the science beyond what the regulations mandate. These data have important implications for improving practice, education, research, and policy.

Hispanic cancer patients are underrepresented in clinical trials; research suggests lack of knowledge and language barriers contribute to low accrual. Multimedia materials offer advantages to Hispanic populations because they have high acceptability, are easy to disseminate, and can be viewed with family.

Hispanic cancer patients and caregivers participated in focus groups to aid in developing a Spanish-language multimedia intervention to educate Hispanic cancer patients about clinical trials. We explored the feasibility of delivering the intervention in medical oncology clinics.

A total of 35 patients were randomized to either the multimedia intervention group (n = 18) or a control group (n = 17) who were asked to read the National Cancer Institute’s Spanish-language clinical trials brochure. Self-reported data on knowledge about and attitudes toward clinical trials, self-efficacy for participating in a clinical trial, intention to participate in a clinical trial if asked, and receptivity to information about a clinical trial were collected at baseline and 10 days later.

Delivery of the multimedia presentation in oncology clinics was feasible. The intervention group had more knowledge about clinical trials at follow-up than the control group; scores for intention to participate in a clinical trial by participants in the intervention group increased from 3.8 to 4.0 of a possible 5, but declined in the control group from 4.5 to 4.1. No statistically significant difference was detected between groups in scores for attitudes or self-efficacy for making a decision to participate in a clinical trial.

Our sample size was inadequate to identify differences between the informational methods. Although all patients were asked about their willingness to participate in a clinical trial, this decision was hypothetical. In addition, the study was conducted with a sample of Spanish-speaking Hispanic cancer patients at a comprehensive cancer center in Florida. Thus, the results may not generalize to other Hispanic populations.

In the pilot project, we demonstrated the feasibility of delivering multimedia information to patients in medical oncology clinics. Because delivery in a clinical setting was found to be feasible, a larger study should be conducted to evaluate the efficacy of the multimedia intervention with respect to promoting accrual of Hispanic patients to clinical trials.

Retention of study participants in randomized controlled trials (RCTs) is crucial to study validity.

We analyzed the results of four retention strategies used to reconnect with urban teens enrolled in a school-based RCT and overdue for a 12-month follow-up survey.

Traditional retention strategies used to reconnect with teens categorized as ‘unable to contact’ were weekly redials of nonworking telephone numbers and mailings to the student’s home. Nontraditional retention strategies were obtaining assistance from school administration and performing outreach on Facebook.

Of the 422 students enrolled, 125 (29.5%) were overdue for a 12-month follow-up survey, but had no working telephone number (unable to contact). We made 196 attempts to contact these 125 students, of which 82 attempts (41.8%) were successful in ‘reconnecting’ with the student. Using ‘mailed reminder letters’ as the referent category, odds ratios (95% confidence intervals) for the association between the strategy used and reconnecting were 4.60 (1.8–11.8), 1.94 (1.01–3.73), and 2.91 (0.58–14.50), respectively, for telephone number redials, Facebook outreach, and school administration assistance. Of the 422 students, 380 (90%) ultimately completed the 12-month follow-up survey.

Retention strategies were not applied hierarchically or systematically. We were unable to determine student preference for a particular strategy. Findings are likely only applicable to similar study populations.

A mix of traditional retention strategies and more contemporary methods was effective in reconnecting with urban teenagers enrolled in a school-based RCT and in controlling attrition during the 12-month follow-up survey period.

On-site source data verification is a common and expensive activity, with little evidence that it is worthwhile. Central statistical monitoring (CSM) is a cheaper alternative, where data checks are performed by the coordinating centre, avoiding the need to visit all sites. Several publications have suggested methods for CSM; however, few have described their use in real trials.

R-programs were created to check data at either the subject level (7 tests within 3 programs) or site level (9 tests within 8 programs) using previously described methods or new ones we developed. These aimed to find possible data errors such as outliers, incorrect dates, or anomalous data patterns; digit preference, values too close or too far from the means, unusual correlation structures, extreme variances which may indicate fraud or procedural errors and under-reporting of adverse events. The methods were applied to three trials, one of which had closed and has been published, one in follow-up, and a third to which fabricated data were added. We examined how well the methods work, discussing their strengths and limitations.

The R-programs produced simple tables or easy-to-read figures. Few data errors were found in the first two trials, and those added to the third were easily detected. The programs were able to identify patients with outliers based on single or multiple variables. They also detected (1) fabricated patients, generated to have values too close to the multivariate mean, or with too low variances in repeated measurements, and (2) sites which had unusual correlation structures or too few adverse events. Some methods were unreliable if applied to centres with few patients or if data were fabricated in a way which did not fit the assumptions used to create the programs. Outputs from the R-programs are interpreted using examples.

Detecting data errors is relatively straightforward; however, there are several limitations in the detection of fraud: some programs cannot be applied to small trials or to centres with few patients (<10) and data falsified in a manner which does not fit the program’s assumptions may not be detected. In addition, many tests require a visual assessment of the output (showing flagged participants or sites), before data queries are made or on-site visits performed.

CSM is a worthwhile alternative to on-site data checking and may be used to limit the number of site visits by targeting only sites which are picked up by the programs. We summarise the methods, show how they are implemented and that they can be easy to interpret. The methods can identify incorrect or unusual data for a trial subject, or centres where the data considered together are too different to other centres and therefore should be reviewed, possibly through an on-site visit.

Randomized clinical trials, particularly for comparative effectiveness research (CER), are frequently criticized for being overly restrictive or untimely for health-care decision making.

Our prospectively designed REsearch in ADAptive methods for Pragmatic Trials (RE-ADAPT) study is a ‘proof of concept’ to stimulate investment in Bayesian adaptive designs for future CER trials.

We will assess whether Bayesian adaptive designs offer potential efficiencies in CER by simulating a re-execution of the Antihypertensive and Lipid Lowering Treatment to Prevent Heart Attack Trial (ALLHAT) study using actual data from ALLHAT.

We prospectively define seven alternate designs consisting of various combinations of arm dropping, adaptive randomization, and early stopping and describe how these designs will be compared to the original ALLHAT design. We identify the one particular design that would have been executed, which incorporates early stopping and information-based adaptive randomization.

While the simulation realistically emulates patient enrollment, interim analyses, and adaptive changes to design, it cannot incorporate key features like the involvement of data monitoring committee in making decisions about adaptive changes.

This article describes our analytic approach for RE-ADAPT. The next stage of the project is to conduct the re-execution analyses using the seven prespecified designs and the original ALLHAT data.