Abstract
Clinical research management (CRM) is a critical resource for the management of clinical trials and it requires proper evaluation. This article advances a model of evaluation that has three local levels, plus one global level, for evaluating the value of CRM. The primary level for evaluation is that of the study or processes level. The managerial or aggregate level concerns management of the portfolio of trials under the control of the CRM office. The third, often overlooked level of evaluation, is the strategic level, whose goal is encapsulated in the phrase, “doing the right trials, while doing trials right.” The global (“plus one”) evaluation level concerns the need to evaluate the ever-increasing number of multi-institutional and multinational studies. As there are host of evaluation metrics, this article provides representative examples of metrics at each level and provides methods that can aid in the selecting appropriate metrics for an organization.
Introduction to the Model
Most medical organizations and academic health centers (AHCs) that complete clinical trials have, in one form or another, an office responsible for the conduct of such studies. Called by a variety of titles, such as the clinical trials office (Paller, Hostetler, & Dykhuis, 2002), clinical trial management, or the clinical research management (CRM) office (Johnson et al., 2010), this group is responsible for the opening, running, monitoring, reporting, and closing of clinical trials. The major areas of focus for CRM offices, as identified by the National Institutes of Health’s (NIH) Clinical & Translational Science Awards (CTSA) program, are listed in Table 1 (2013a) 1 . This NIH-supported program is particularly appropriate for understanding these areas as it is comprised of about 60 academic medical institutions and a coordinating center all working to work together to transform the way biomedical research is conducted.
CTSA Clinical Research Management Key Functions.
Note. CTSA = Clinical & Translational Science Awards; IRB = institutional review board.
Source. https://www.ctsacentral.org/committee/clinical-research-management.
These areas of focus are excellent from a tactical perspective, but what is missing from this list are the more managerial and strategic foci that are needed to fully integrate CRM with the overall strategy of the organization. For example, are the right studies being managed to achieve the strategic portfolio of the organization? CRM is a primary infrastructure component of all clinical research and thus its proper evaluation is vital to the efficient and effective completion of clinical research.
The critical performance criteria for such an evaluation of the CRM organizational function can be difficult to ascertain. For example, one emphasis of CRM evaluation has been on the variability of time required for institutional review board (IRB) review approval both in the United States (Conforti, Hess, Ross, Lynn, & Holmboe, 2012; McWilliams et al., 2003) and internationally (Metzger-Filho et al., 2013). While this is a key trial performance indicator, if the resulting study accrues no patients because the organization does not see the appropriate patients, additional evaluation criteria are clearly required.
This article proposes a CRM evaluation model at three local, plus one global, levels (referred to as a “3+1 model”). The first level of evaluation is at the individual study level, where the primary emphases are on time and the elimination of waste and inefficiencies in the CRM processes. The second level of evaluation is at the managerial level of CRM, where the emphasis concerns the portfolio of trials under the management of the CRM. The third local level concerns the strategic “fit” of individual trials and the total clinical trial portfolio with the strategic intent of the organization. For example, if the primary strategy for a cancer center is on personalized, or precision medicine, how many trials underway can be considered to fall into that category?
The three local levels must be extended to include a more global, network, or multicenter level (the “+1” aspect). As medicine becomes more knowledgeable about the underlying genetics of disease states, then additional inclusion and exclusion criteria will be added to clinical trial protocols. From a trial perspective, this means that it will be increasingly difficult for any single site to accrue all the patients necessary for a trial, as the existing local patient population is divided into smaller and smaller possible accrual pools. Hence, there will be a need for evaluation methods for an organization’s CRM to understand which other sites to partner with, what the effects are of joining different trial networks, and the importance of global, i.e. network, effectiveness measures. As such a network relationship is key to the NIH’s CTSA program, it is important that metrics be developed that allow for the measurement of how such networks are accelerating discoveries toward better health.
Such a multi-level structure has been discussed as general framework for health systems (Berwick, 2002); hence, it may also be appropriate for use in clinical research. This model is shown in Figure 1, which shows that the lower levels of the CRM evaluation model have a wide variety of possible evaluation metrics. However, these metrics have limited reach, that is, they impact primarily individual trials or the local portfolio of trials. At the higher levels, the reach of the evaluation metrics is greater: they impact all trials, but the number of possible metrics is limited. For example, there are few benchmark metrics to evaluate the efficiency or effectiveness of clinical trial networks, even though such networks span the nation and, in some cases, the globe.
The 3+1 CRM evaluation model. CRM = clinical research management.
All the 3+1 levels will be discussed in turn. For each level, its primary focus will be identified. This will be followed by a discussion of potential evaluation metrics for that level. Next will be suggestions for possible tools and techniques for improving performance at that level. It is important to note that this article only scratches the surface of potential CRM evaluation metrics and solution tools. There is a host of other metrics and tools that can be utilized for improving CRM performance. For example, George et al. (George, Rowlands, Price, & Maxey, 2005) list almost 100 tools that can be used just at the first two levels, namely at the process and managerial levels.
The current article does not intend to present a comprehensive review of all possible metrics or improvement tools. Rather, the purpose is to highlight that it is important that the CRM function be evaluated at multiple levels simultaneously and that focusing on either a single metric or a single level of performance will lead to local optimization at the expense of more important global optimization. To help structure the discussion, Table 2 shows the focus of each level, along with possible measurements.
Potential Metrics for Evaluation in Clinical Research Management.
The “Three-Plus-One” Evaluation Model for CRM
Level 1: Study and Process Level
The primary focus of this level is the individual study or function. Key evaluations that normally take place at this level include the scientific merit of the proposed study, the time and effort required to open the study, and the processes, steps, and number of people involved in opening and running an individual study. It is at this level where analysis of the inefficiencies of IRBs should be taken into account. Three primary questions normally addressed here are “what is the scientific merit of the study?,” “are study participants sufficiently informed about the risks and benefits of the study?,” and “how much time will it take to open the study?”
While these are all important questions, there is an additional set of evaluation metrics that should be analyzed, which are encapsulated in the question, “What is the operational feasibility of this study?” In other words, given the demands of the proposed trial will patients be likely to agree to participate, are there sufficient resources to properly execute the study, and has such a study been successful in accruing patients in the past? For example, a scientifically strong study in one geographic location might be extremely difficult to accrue to in a different location, say if a unique or rare nonmobile patient population is required. One method to successfully address these issues is the implementation of a feasibility checklist that, using historical data, is used by CRM to verify operational feasibility of the proposed study.
At this level of CRM evaluation, there are numerous solution methods for addressing the process flow questions. The simplest is the creation of a process map that lists all the steps and participants required to open and run a study. This technique has been widely used in oncology (Dilts & Sandler, 2006; Dilts et al., 2006, 2008, 2009). This research has highlighted that there are different types of process barriers to opening of studies, namely procedural, structural, and infrastructural, and that there can be a high percentage of nonvalue-added steps in the CRM processes. A more advanced technique that has also been successful in streamlining CRM processes is that of value stream mapping (McJoynt et al., 2009), which not only lists processes but also includes timing information. It is at this level where most of the more popular efficiency concepts such as Lean and Six-Sigma are applied (George et al., 2005).
Level 2: Managerial Level
Supporting the portfolio of trials is the focus of this level of CRM evaluation. It is at this stage where resource utilization of CRM staff is evaluated, as is the financial viability of the clinical trials office. Indeed, in oncology clinical research, because of the mixture of payment methods that vary by type of trials (e.g., investigator-initiated trials, National Cancer Institute–sponsored cooperative group trials, and industry/pharmacy company–sponsored trials), it is possible for the work of the entire staff of the CRM office to be completely committed but for the CRM office to be losing money (e.g., expenses higher than revenue). Thus, it is important for the proper evaluation of the CRM function that it be evaluated at a level higher than that of individual studies.
Management of the CRM needs to be concerned with critical factors such as the number and cost of low-enrolling studies (Kitterman, Cheng, Dilts, & Orwoll, 2011), number of patients accrued to trials compared to total available patient population (Massett et al., 2011), and the concerns of investigators who use the services of the CRM (Miller, Sevastita, Chaitt, Tavel, & Pierson, 2013). In the subsequent section is a closer look at each of these evaluation metrics.
First, it has been shown that a significant percentage of trials results in zero or one accrual at time of study closure (Dilts, Cheng, Crites, Sandler, & Doroshow, 2010). Even at a mid-size AHC, such dormant trials can cost nearly a million dollars per year of wasted resources (Kitterman et al., 2011). Even more meaningful than money is that such studies, if the organization does not allow competing trials for the same patient population, can prevent other important studies from opening. The individual study might be scientifically valid, but if there is a low likelihood of accruing to the trial or if the trial requires excess resources compared to its potential scientific value, it is important that CRM function have the ability to limit such studies. One of the most simple but difficult solutions to this issue is to “just say ‘no’,” that is, to not open studies that have limited likelihood of achieving accrual goals.
A standard metric used by many AHCs and NIH institutes, is the percentage of available patients who enroll on clinical trials. This has been a continuing problem in oncology and has resulted in numerous strategies, tools, and solution methods summarized in the website AccrualNet.org supported by the National Cancer Institute (Massett et al., 2011). This portal represents a central location where questions and answers concerning the development, selection, and recruitment to a trial can be viewed. Most importantly for management, it includes a stage of evaluation and reporting of lessons learned so that future trials can be enhanced.
Finally, CRM needs to understand what the principal investigators (PIs) think of their services. A popular evaluation method to determine that is a voice of the customer survey (Miller et al., 2013). Made popular by the Baldrige Performance Excellence Program (NIST, 2013), also known as the Baldrige Award, such a survey provides managerial information that allows a CRM to understand what their customers (i.e., PIs) feel is the service level being provided. If done correctly, it can also explore why some PIs are noncustomers, that is, why they create their own “mini-CRM.” The results of such a survey can highlight areas where additional mentoring, credentialing, and training of PIs and research staff should be completed.
Evaluation metrics at the study and managerial levels are interconnected, as poor performance at one level can significantly impact performance at the other level. For example, it has been shown that the time to open an oncology clinical trial is significantly and negatively related to eventual total trial accrual (Cheng, Dietrich, & Dilts, 2010). Or, to put it in other words: the longer it takes to open a trial, the less likely the trial will achieve its accrual goals.
Each organization must choose for itself the most important managerial metrics and measurements. In this regard, it is critical for the management of the CRM, and the AHC, to understand what might be appropriate for one organization might not be appropriate for a different organization facing different local demands. For example, uniqueness of available patient populations or of organizational resources such as a proton beam can limit potential trials that can be open and managed at a particular site. Thus, care should be taken when selecting managerial CRM performance metrics.
Level 3: Strategic Level
Are the right trials being done? This is the fundamental question to address at the strategic level of CRM evaluation. It is important that not only are trials done right but also the right trials are being done (Dilts & Cheng, 2012). To put it differently, efficiency—the primary focus of the previous two levels—is a necessary but not sufficient focus for the strategic success of CRM (Porter, 1996).
It is at this level that the most fundamental evaluation questions of the entire CRM function should be addressed. For example, there are two conceptually different ways of approaching the portfolio of trials. One way is to identify existing research and management strengths and concentrate the trial portfolio on those strengths or core competencies (Prahalad & Hamel, 1990). In this inside-out approach, existing skill sets and organizational competencies can foster and enhance the completion of clinical research as the CRM function can utilize the learning-curve effect. The opposing way of approaching the portfolio of trials is from outside-in (Porter, 2008). Here the concentration of the trial portfolio is on trials that best match existing patient mix. The advantage to this market-facing, or bed-to-bench approach is that achievement of trial accrual goals can be easier to accomplish.
It is important to note that neither of these approaches is “best.” Rather, it is vital for CRM to understand which strategy it is following when determining which trials to open and resource. Different strategies should lead to different critical evaluation metrics. While neither the inside-out nor the outside-in approach can be considered universally the best strategy, there is a worse choice: attempting to blend the two strategic approaches together. This results in the well-known management strategy of “stuck-in-the-middle” (Porter, 2008), where PIs, patients, and CRM members are confused and dissatisfied as they do not know what is most important to the success of the research and to CRM, and thus the purpose and effectiveness of the CRM can be called into question.
A common approach to determining metrics at this level is via strategic planning. While common, such planning often leads to the creation of reports that are primarily used only for budgeting exercises and therefore have limited relationship to actual decisions and operations (Mintzberg, 1994). A different method is to ask each of the key stakeholders in CRM what they believe the mission and strategy of CRM is and should be (Dilts, Peters, Cheng, & Stadum, 2012). This approach allows for mapping beliefs by various stakeholders of CRM and it can clearly show mismatches in expectations and strategies. By using such a grounded approach to strategic CRM evaluation, strategic gaps and strategic disconnects can be highlighted and then solved.
As with the trial and the managerial levels, strategic level decisions interact with the other levels. For example, it is possible to analyze how effective the portfolio of trials have been, given the levels of resources required. This can be done using either return on investment (ROI) or data envelopment analysis (DEA; Dilts, 2012). This analysis can then show how future resources of CRM should be expended to get the highest return on resource use, scientific knowledge, or both.
Such strategic metrics of effectiveness are increasingly being used by pharmaceutical and device organizations when determining strategic partners. For example, clinical research organizations (CROs) are expanding beyond the traditional factors of key opinion leaders and trial cost at a center, to a more global evaluation of the total AHC performance with respect to average overall time to open a study and global accrual performance on all studies opened by the CRO at that AHC (Dilts, 2013, personal communications).
“Plus-One” Level: Expanding beyond the Bounds of a Single Organization
The existing clinical trial enterprise in America is under stress and must be reevaluated (Dilts, 2010; Nass, Moses, & Mendelsohn, 2010). Looking at only two aspects will highlight the issues. First, there is a rapid and extensive growth of clinical research completed internationally, with only limited involvement by domestic researchers (Dilts, 2010). As has happened in many industries, globalization has changed the fundamental success model of clinical research. Second, clinical trials are using an ever-expanding number of “–omics,” genetics, and biomarkers to drive inclusion and exclusion criteria. While such divisions are sound from a research perspective, they reduce the total population of possible patients to accrue to a trial at a single site. Such market segmentation means that it will be increasingly difficult for any individual site to enroll all patients required to achieve trial accrual goals. This implies that multi-institutional trials will become the norm, rather than the exception. This is already well known in the pharmaceutical industry where Phase III studies are rarely completed at a single institution. Such multicenter trials have much the same issues and solution strategies as those that occur in a single site, for example, with IRB approval times (Greene & Geiger, 2006; McWilliams et al., 2003).
However, this level also has unique evaluation and performance metrics. For example, at the +1 level, it is possible to understand the trial demands of different diseases (Califf et al., 2012) and of the demands of different histologies within the same disease (Hirsch et al., 2012). It is also important to understand that there are different research networks, with very different network characteristics (Cheng et al., 2012). As at the strategic level, there is no single “best” selection criteria for determining which network to join or which institutions with which to partner, but, again, it is well known that the worst possible alternative is to join multiple, noncomplementary networks.
At this level, there are few metrics of performance within CRM. However, other industries have addressed the issues of network structure and metrics of performance. For example, the manufacturing and service sectors have developed standards and a reference model for how such cooperation should occur. Called the Supply Chain Operations Reference (SCOR) model, this model has been widely used and contains over 150 performance metrics (Supply Chain Council, 2010).
A completely different approach is the one used by open-source networks where there is no financial involvement by the parties in the network (Benkler, 2006, 2011). These networks are created to foster both collaborative solutions in an environment that highlights accomplishments, health competition, and shared goals. Easy to understand, such environments are extremely difficult to achieve and there are few metrics to evaluate their performance.
A third approach to the +1 level issues is that being used by the NIH CTSA program. Here, the NIH is fostering the ability of consortium members to work nationally to increase the efficiency and speed of clinical and translational research across the country (CTSA, 2013c). By creating an infrastructure that allows for virtual integration of efforts and the ease in developing partnerships for research across sites (Dilts, Rosenblum, & Trochim, 2012), this should go far in overcoming the performance barriers inherent in multi-institutional studies.
Limitations and Conclusions
This article has only scratched the surface of possible evaluation metrics for CRMs. For example, it did not address how a CRM should be evaluated as it simultaneously “competes” with a neighboring organization while it also cooperates to complete multi-institutional studies. Nor did the article address how multiple, competing CRMs, common in large AHCs, could have both complementary and competing strategic evaluation metrics.
The intention of this artcle was not to be an exhaustive treatment of all possible CRM metrics. Rather the 3+1 model was created to show that the efficient and effective performance of a CRM cannot be evaluated at a single level. The most effective CRM will be appraised on a dashboard of possible metrics, evaluated simultaneously at multiple levels, with each level using different but complementary metrics.
In order to best implement the 3-plus-one evaluation model, four factors are important to consider:
What metrics of performance are globally recognized as important at each level? For example, oncology has long had a metric with respect to percentage of patients who should be placed on clinical trials. Such generally used metrics allows for cross-center comparisons.
What metrics of performance are vital to the local institution? While well-recognized metrics are helpful, local metrics, driven by local conditions are also important.
What measurements should be made for the betterment of the entire clinical trial process? For example, there were no measurements or metrics for the time to open oncology clinical trials until the lengthy delays in this process was highlighted and published. While general “wisdom” acknowledged that the time was extensive, until the exact time was measured, only limited improvements were made. However, once measured, performance could, and was, significantly improved.
Finally, it is important to understand how measurements at one level impact measures at other levels. Generically, this is the issue of “fit,” where it is important, at the basic level, that metrics at one level do not conflict with metrics at a different level, and at a higher level, how metrics at one level can lead to better overall performance at all levels. For example, reducing the number of trials that do not strategically fit the mission of the organization should lower the number of low-accruing trials, which should free up time in the CRM such that the time to open a trial is reduced.
Identification of proper metrics, particularly those that are reinforcing, do not come “naturally” or easily. They require time commitments by key leaders in both the CRM and the organization to develop the best mix of metrics for local conditions. Then they need to be continuously monitored for both performance on the measurements and on evaluating if the set of metrics are still proper. Thus, the process of using and developing metrics should be part of a CRM learning organization, which learns from experience what is, and is not, important. Therefore, it is critical for the success of CRM, and for every CTSA, to continue on this quest to develop and use better performance measures.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The contributions of David M. Dilts are supported by The National Cancer Institute (NCI) of the National Institutes of Health (NIH), Contract SAIC-F Agreement 13XS141 and Oregon Clinical & Translational Research Institute (OCTRI), Grant Number (UL1TR000128 from the National Institutes of Health (NIH) National Center for Advancing Translational Sciences (NCATS). The content of the article is the sole responsibility of the author and does not necessarily represent the official views of the NIH.