Evaluation and the NIH Clinical and Translational Science Awards

Abstract

Since 2006, a total of 61 Clinical and Translational Science Institutes (CTSAs) have been funded by the National Institutes of Health (NIH), with the aim of reducing translation time from a bench discovery to when it impacts patients. This special issue of Evaluation & the Health Professions focuses on evaluation within and across the large, complex system of the CTSA Program of NIH. Through insights gained by reading the articles in this special edition and the experience of the authors, a “top ten” list of lessons learned and insights gained is presented. The list outlines issues that face those who evaluate the influence of the CTSA Program, as they work to anticipate what will be needed for continuing success. Themes include (1) considering the needs of stakeholders, (2) the perspective of the evaluators, (3) the importance of service improvement, (4) the importance of teams and people, (5) costs and return on investments, (6) methodology considerations to evaluate the CTSA enterprise, (7) innovation in evaluation, (8) defining the transformation of research, (9) evaluating the long-term impact of the CTSAs on public health, and (10) contributing to science policy formulation and implementation. The establishment of the CTSA Program, with its mandated evaluation component, has not only influenced the infrastructure and nature of translational research but will continue to impact policy and management in science.

Keywords

evaluation CTSA science policy translational research transformation quality improvement team science evaluation methods

Introduction

Since funding the first cohort in 2006, the National Institutes of Health (NIH) has funded a total of 61 Clinical and Translational Science Awards (CTSAs) (https://www.ctsacentral.org/). The program was intended to build a broad array of intra- and inter-institutional resources aimed at reducing translation time from a bench discovery to when it impacts patients (Butler, 2008). Importantly, NIH required, as one of the key functions of CTSAs, a formal evaluation program. Moreover, at the level of the overall consortium of CTSAs, NIH established a Key Function Committee on Evaluation to enable the sharing of best practices and the development of collaborative efforts. This special edition of the Evaluation & the Health Professions focuses on evaluation within and across the large, complex system of the CTSA Program of NIH. We cover a variety of themes in this edition, including clinical research administration and finance, validating measurement tools, assessing investigator networks and productivity, program evaluation, investigator training, and the evaluation workforce, directions and resources. Our intention with this editorial is to synthesize these articles through the presentation of a top ten “lessons learned” list. This list outlines issues that face CTSA evaluators as they work to anticipate what they need to succeed for the future.

Consider Market Segment Needs

Who needs to know what in order to answer which questions?

Evaluation of the CTSA-supported academic institutions is being conducted within a very complex scientific, political, and social ecology. Evaluators need to negotiate different interests, understand multiple power relationships, and identify key priorities. Multiple “market segments” need to be delineated; each has different roles with regard to the research/CTSA enterprise and each faces different challenges. To meet these challenges, key questions need answers and evaluation is expected to provide the answers. Evaluators thus need to frame their tasks around a fundamental question: “who needs to know what in order to answer which critical management questions?” and, ultimately, weigh the needs and set priorities across these multiple stakeholders. At the institutional level, stakeholders range from trainees and researchers to department chairs, CTSA PIs, and Deans. These are matched at the national level by stakeholders that include National Center for Advancing Translational Sciences (NCATS) which is the formal NIH center that manages the CTSA program, the NIH Director’s Office, Institutes and Centers, the United States Congress, and the public (Figure 1).

Figure 1.

CTSA Market Segments.

For example, in the recent Institute of Medicine (IOM) evaluation of the CTSA Program, the Consensus Committee, reflecting a national stakeholder point of view, expressed a very broad agenda for CTSAs that includes tackling:

…challenges associated with first-in-human studies; limited recruitment and retention in clinical trials; the identification and measurement of health outcomes to assess intervention effectiveness; barriers to increasing awareness about research resources and potential research partnerships at the investigator and community levels; lack of incentives for team-based science; policy and regulatory challenges in developing full and substantive collaborations with industry and other partners; and ethical concerns (including related regulatory requirements) associated with the interplay between clinical research and practice. (p. 45)

Starting at a more “molecular” approach, Dilts (2013), in his article, explores the multiple evaluative needs through his “three-Plus-One” evaluation model focused on clinical research management. He identifies three levels of evaluation: (1) The study and process level where the focus is on an individual study or trial; (2) The managerial level which related to the management of the trial portfolio; and the resource utilization of the clinical research system; and (3) The strategic level asks whether the right trials are being conducted. His “plus-one” level moves beyond a single organization—to multicenter trials and the needs of a network, such as the CTSAs.

Clarify the Role(s) of Evaluation

The day-to-day life of evaluators

Numerous and complex demands of multiple stakeholders and limited resources available for evaluation (see Kane, Alexander, Hogle, Parsons, & Phelps, 2013) present enormous challenges to the evaluator. Such challenges are further compounded by a frequent lack of understanding of the role, methods and utility of evaluation. Evaluators have the responsibility to frame often amorphously posed questions in answerable ways and distinguish what data are truly needed from those that would be “nice to know.”

As a first step in communication with these stakeholders, it may be helpful to organize the various evaluation activities into the following buckets:

Service Improvement

Service improvement is all about meeting the needs of consumers (e.g., clinical and translational investigators and trainees) and other stakeholders (e.g., the departments, schools at the CTSA institution), making sure that the evaluators understand what consumers needs are and making sure that the “products” of the CTSA (e.g., training programs, bio statistical and design consultations, informatics supports, etc.) are meeting those needs. Of course, this must be a continually evolving effort inextricably linked to quality improvement strategies derived from industrial engineering (e.g., Lean, Six-Sigma, and Plan-Do-Study-Act cycles, etc.).

National Evaluation Activities

In general, the Evaluation Key Function Committee of the CTSA consortium serves at the epicenter of national evaluation activities, providing a venue and an organizing force for collaborative projects among CTSAs including this special section of the journal Evaluation and the Health Professions. While many evaluation projects are aimed at the needs of national stakeholders, most also provide pragmatic utility for evaluators with application at a local level. Rubio et al. (2013) describe the common metrics project that was responsive to expressed needs of national stakeholders and was explicitly acknowledged and encouraged in the IOM report (2013) and discussed in more detail below.

Grant Renewal

For the principal investigators and the institution, the highest priority is grant renewal. Evaluators are critical for this task, since the responsibility of most evaluators is to amass data documenting results of each CTSA’s activities. More importantly, renewal, in principle and reality, serves as an endorsement that a CTSA is succeeding in its mission of service and accomplishment.

Evaluation Science

The presence of evaluation as a key CTSA function provides a remarkable opportunity and platform for projects that illustrate the contributions of evaluation as a science and also that expand the boundaries of evaluation methods and approaches. The evaluation section of the IOM report (pp 81–89) concluded with a call for CTSAs to become innovators in evaluation methodologies and metrics.

Transformation

What is meant by transformation, and how should it be measured? This cuts through all market segments and stakeholders, yet remains largely undefined as discussed below. In some way, the initial CTSA mandate for evaluation and recent support by IOM for integration into the CTSA mission may be transformative itself.

Apply Quality Measurement and Improvement

It’s all about service

A CTSA is expected to facilitate research by establishing infrastructure and providing services to investigators and trainees. While one might take a narrow view of the role of evaluation in this context as being one of assessing the satisfaction of these consumers with the services they have been receiving, the much larger and more important challenge is shifting the culture of the organization toward continuous learning and improvement. Recently, a report issued by the IOM on the “learning health care system” is an ideal, designed to (1) generate and apply the best evidence for collaborative health care choices of each patient and provider, (2) drive the process of discovery as a natural outgrowth of patient care, and (3) ensure innovation, quality, safety, and value in health care. The learning health care system is founded on the value of continuous improvement and the imperative to translate health care knowledge into health care action (National Research Council [NRC], 2013). In a sense, one can think of the “ideal CTSA” as a “learning health science system,” linked to (or a component of) a learning health care system. Several articles describe elements and examples of projects along a developmental pathway toward achieving this ideal.

Hogle et al. (2013) discuss a case-study based approach to service improvement. Given the complexity of large NIH grants such as the CTSAs, Hogle et al. argue that creating cases that demonstrate how successful researchers have utilized and capitalized on the CTSA infrastructures allow evaluators to understand what factors lead to success in order to identify patterns. Allen, Ripley, Coe, & Clore (2013) describe formal quality improvement efforts involved in transforming a general clinical research center operation into a more effective and responsive Clinical Research Services entity. Mazmanian, Coe, Evans, Longo, & Wright (2013) identify heavy investment of the CTSA Program into researcher development and the weak evidence to support causal linkages of education and training to improved researcher behavior. Scott et al. (2013) describe using a combination of the Kellogg logic model and the World Health Organization’s (WHO) Health Services Program Evaluation Model to assess the impact of their individual CTSA and to facilitate decision making for service improvement.

Understand and Assess Team Science

It’s all about teams/people

All CTSA supported institutions are expected to provide robust postgraduate training (NIH, 2012). Many maintain extensive education programs that include undergraduate and predoctoral student training as well as training for junior faculty, research staff, and community collaborators. For example, while Tillman et al. (2013) document the policies and activities among 53 CTSA KL2 programs, the evidence to support the effectiveness of the programs is weak, and evaluators are challenged not only to produce better quality evaluative research but also to develop new tools for detecting causal linkages among and between education, training, and other research support services of CTSA-supported academic institutions and effective translational research (Mazmanian et al., 2013; Mills, Caetano, & Rhea, 2013; Rubio, 2013; Wooten et al., 2013; Tillman et al., 2013). The success of an integrated team-based and interdisciplinary approach to science, health care, and education requires relationship building among and between professional schools and with community partners such as patients, families, and health care providers. Emphasis on experiential and team-based learning and incorporating topics outside the traditional realm of basic and clinical science provides a solid foundation needed to spur clinical and translational research (NRC, 2013). CTSAs have the potential both to create learning cultures that embrace innovative teaching methods and content (Wooten et al., 2013), both to create valid and reliable tools for measuring researcher progress, to disseminate those innovations rapidly and effectively (Dembe et al., 2013; Mills et al., 2013).

While the CTSA Program can support this potential for learning, there is a need to “…understand how teams collaborate to achieve scientific breakthroughs that would not be attainable through either individual efforts or a sequence of additive contributions” (Falk-Krzesinski et al., 2011, p. 146). The work by Falk-Krzesinski, using concept mapping and a sample of over 800 stakeholders created a taxonomy of issues related to the science of team science (SciTS) which can help frame a starting point for CTSA collaborative projects to learn, among other things, the “dynamics of knowledge integration” within teams. While the discussion of team science has grown, a good deal of that literature is still formative and has focused on how teams are created and sustained. While team performance and the study of teams is typical in many fields, there is little evidence as to whether team science versus individually focused science will lead to improved health outcomes, and there is no evidence yet of efforts to create such evaluations (Falk-Krzesinski et al., 2011).

In efforts to foster interdisciplinary research, the National Research Council of the National Academies (NRC) examined research and doctoral programs of major universities and associated medical centers (NRC, 2004) for their place in the United States system of higher education, innovation, and industrial enterprises (NRC, 2012a). The organizational, financial, and intellectual capacity of public and private research universities was assessed. NRC recommended that the federal government should adopt stable and effective policies, practices, and funding for university performed research and development and for graduate education, strengthening the role of business in the research partnership and facilitating the transfer of knowledge, ideas, and technology to society for accelerating “time to innovation” in order to achieve national goals (NRC, 2012a).

Create Efficiency and Value

Money is an object

An elusive goal of the CTSA endeavor is to get more things done with fewer resources, that is, make things “better, cheaper, and faster.” So the questions raised are as follows: How do you evaluate whether it is cheaper and what is the “it” that is supposed to be cheaper? For whom are things being made cheaper and whose return on investment (ROI) should be of highest concern (universities, NIH, pharmaceutical companies, society)? The issue is also temporal. Costs can shift over the short and long term. Costs can vary across categories including research and health care. There are methodological issues to consider. There is also the reality that a majority of translational projects do not find their way into clinical practice as they fail to reach their goal of improving human health through technology and innovations. In fact, we know that only a small fraction of those studies that claimed they would have human applicability actually resulted in the improvement of human health or had any impact (Contopoulos-Ioannidis, Ntzani, & Ioannidis, 2003; Crowley, 2003).

But there are things that can be done that will have monetary implications (e.g., move things more quickly through the regulatory processes, efficiently recruit participants, etc.). The Evaluation Key Function Committee’s common metrics project (Rubio et al., 2013) suggests ways to standardize these measurements. Grazier, Trochim, Dilts, & Kirk (2013) discuss approaches to assessing ROI in one specific component of the CTSA: The clinical research unit. They consider the lens of the investigator, the program, and the institution. They distinguish between the classic indicators of economic ROI (such as additional grants, publications, patents, etc.) but also to incorporate the concept of a social ROI, the “value” added that is not just economic. Their approach includes “…identifying types of costs, impacts and values (external, internal, financial, social); creating alternative conceptual frameworks to estimate the impacts and value of translational research on individual researchers, the research enterprise, consumers of research and clinical care, and the public; surveying CTSAs on available sources and formats of economic and social impact data; determining costs of collecting and analyzing financial data; testing usability of each framework and efficacy of resulting metrics; and creating protocols for use by CTSAs.”

Apply Appropriate Evaluation Strategies

It’s all about Methods

CTSAs were established as the centerpiece for a refocus and redesign of the NIH toward promoting and fostering translational research. They were also uniquely given the explicit requirement to engage in evaluation activities. This requirement along with this broad set of goals presents each CTSA evaluator with having to make critical trade-offs to move from broad CTSA goals to specific evaluative questions.

Concerns over methodological and conceptual trade-offs have been part of the history of evaluation almost since its inception (Cronbach, 1980, 1982) and continues today in evaluation (Glasgow, 2007; Mercer, DeVinney, Fine, Green, & Dougherty, 2007; Shadish & Cook, 2009) and in the health/ medical research literature (Gaglio, Shoup, & Glasgow, 2013; Olsen, Aisner, & McGinnis, 2007; Rawlins, 2008). These ongoing debates over trade-offs in large part hinge on ascertaining which design provides robust and credible evidence for making causal inferences about the effects of interventions. The CTSA is, after all, a major political, social, and institutional intervention.

This question of causal attribution will be difficult to resolve given the complexity of a CTSA (IOM, 2013) but any effort to do so will most likely require a judicious mix of qualitative and quantitative approaches. Wooten et al. (2013) offer one approach of this broad mix of approaches by highlighting the difference among outcomes, process, and developmental evaluation, and the attendant questions that these evaluation approaches can address. There are many ways of conducting an evaluation, for example, utilization evaluation (Patton, 2008), realistic evaluation (Pawson & Tilley, 1998), empowerment evaluation (Fetterman, 2001), and many different ways of collecting and analyzing data, as presented by Kane et al. (2013) and reflected in the evaluation literature, for example, archival data, primary data collection, quantitative analysis, and qualitative analysis (Denzin & Lincoln, 1998; Greene, 2007; Shadish, Cook, & Campbell, 2002; Stake, 2006; Yin, 2009). Is there a best way for a CTSA evaluator to select an evaluation approach, or a best method to use? A better question may be: what methods work best for what questions?

It is clear from the articles presented in this supplement that no single approach dominates but rather a variety of evaluation approaches are used to collect data, for example, surveys (Kane et al., 2013), case studies (Hogle, 2013), measures (Mills et al., 2013; Dembe et al., 2013; Rubio, 2013), face-to-face interviews (Hogle, 2013; Wooten, 2013), and mixed methods involving original data and archives (Feeney, 2013). In addition, methods and analytic approaches used in other disciplines are starting to be incorporated into the evaluation CTSA efforts, for example, economics and ROI (Grazier et al., 2013), and quality improvement such as plan-do-study-act cycles (Allen et al., 2013). For example, process studies that include the time it takes to apply for a grant, accomplish an Institutional Review Board (IRB) review, and accrue subjects into approved protocols (Dilts, Cheng, Crites, Sandler, & Doroshow, 2010) are becoming increasingly visible topics of discussion during face-to-face meetings of evaluators intending to measure the influence of CTSA supported services (October 2011 and 2012 Evaluation Face-to-face meetings).

This discipline borrowing may be important for CTSA evaluations. While articles are by design limited in what they can present, the ones in this supplement suggest that only using singular methods (e.g., only case studies, only surveys, only process mapping) may be inadequate to address and evaluate the complexity of a CTSA. This point is clearly raised in Hogle’s attempt to grapple with one particular but critical component of a CTSA, how a CTSA helps advance the career of translational researchers (Hogle et al., 2013). Cook and Campbell (1979) were careful to note the biases that can result from mono-method and mono-operation in that we are not able to separate the result from the method used. There is a need to avoid the typical biases from mono-method and mono-operation evaluations. Obtaining comparable results with varied methods and metrics will help create a more robust and credible evaluation.

One additional concern should be noted. Most of the articles in the clinical and translational science issue of Evaluation & the Health Professions present descriptive evaluations—reporting what has been put in place by a CTSA, the development of metrics to be used prospectively, or frameworks for assessing ROI or team science. While these are essential components of a CTSA evaluation, it is worth noting that there are no designed prospective evaluation research studies, no studies that employ randomization, and few that use comparison groups—either within or among CTSAs or between CTSA and non-CTSA universities and institutions.

These types of concerns are not presented as critiques but are noted as opening gambits in an effort to deepen the evaluations of CTSA initiatives and the CTSA Program, and the contributions to evaluation science. This is the opportunity to contribute to the emerging literature on evaluating complex interventions (Medical Research Council, 2000)—an intervention with many component parts—and to better understand and document how CTSAs are trying to transform research and the research enterprise. Not all of this can or will be captured through the use of metrics or in today’s evaluation methods (IOM, 2013).

Develop Innovations in Evaluation

The Vagaries of Choice: Definitions Matter

This vagueness regarding the initial CTSA goals carries over to a more fundamental problem for evaluators: what is translational research? There is no clear, commonly accepted, operationally defined metric. This leaves the definition and assessment of a CTSA's success in increasing translational research open to the perhaps idiosyncratic choices of individual CTSAs. Feeney et al.’s article underscores this concern by noting the modest correlation among differing definitions of translational research (2013, this issue). How does an evaluation proceed in this broad context of ambiguity?

The task of moving from the nonspecific mandate and terminology of the CTSA’s to evaluative questions or studies surfaces at least three critical decision points: Who and how will the evaluative question(s) be determined; what will the unit of analysis be—for example, the researcher, the research team, the research project, the clinical unit, the CTSA core, the larger institution or the entire CTSA national program; and how can these efforts, or at least some of them, be assessed and or measured? These decisions will not only bind what is documented by the evaluation, they will limit what can be learned from any CTSA evaluation. This concern about boundaries, inclusion or exclusion criteria in the clinical trial literature, carries over to the measures used or to be used for they will determine who or what “counts”. For example, who is a translational researcher? In a context in which counting more may be seen as “success” and the pressure to show more “success” intensifies, having clear, standardized definitions of numerators and denominators used across CTSAs matter. From a national perspective, evaluations that are too narrowly focused within a single CTSA will fail to address the broader national questions as to the overall benefits of having invested in the CTSAs—benefits both in terms of the success (e.g., training more translational researchers) and of the value of the CTSAs (e.g., Improving patient care as well as community/public health). Simply put: definitions matter. Thus, the vagaries of individual CTSA definitions of critical terms must be reduced.

Reducing the singularity of individual CTSA evaluations requires conducting evaluations that use a common set of questions, standardized metrics, and methods. Such efforts have started and are reflected in the articles by Rubio, Dembe et al., and Mills et al. (2013). But Scott et al. (2013) point to what may be an important limitation to some of the metrics being developed and reported in incorporating the use of the WHO framework with the Kaiser Foundation approach to logic models.

By adding the WHO questions to more traditionally used metrics, Scott et al. note: “The WHO approach…focuses more on eight evaluative focal points including: relevance, efficiency, adequacy, effectiveness, process, impact, equity and sustainability. These were interesting because they could generate user judgments related to how well the efforts of the Institute were progressing.”

This innovative set of questions may move closer to obtaining data that are critical for decision makers. Similarly, Dozier’s (2013) use of Social Network Analysis is one way, when supplemented by the type of case study reported by Hogle and traditional surveys, to better document and understand the development and growth of both translational researchers and team science. Each of these articles and others such as those by Grazier et al. (2013) on the ROI and the social return on investment (SROI) as well as Rubio, Dembe, and Mills et al. suggest ways to innovatively conduct cross CTSA evaluations.

Other innovations are also possible. For example, no evaluation presented here uses or mentions the use of one strong quasi-experimental design, the use of time-series designs (Shadish & Cook, 2009). This design could be used both within a CTSA to monitor change and with a common standardized metric, across CTSAs, to see how change varies among the CTSAs. At this point in time, no single CTSA has access to any data other than their own. However, NCATS has access to all the CTSA data and is uniquely positioned, with input and participation from CTSA evaluators, to take a lead in looking at data over time—picking up and augmenting the ideas suggested in the Feeny article for defining and evaluating translational research, for example.

Beware of Overpromising

What exactly do we mean by ‘transformation’?

In seeking to alter the character of research, the CTSAs were given a challenge that would require change, a transformation, at multiple levels. These levels extend from transformation of the individual researcher to transformation of the institution to transformation of the science and health care. This implied a cultural transformation not unlike that described above with regard to the “learning health care system.” As recognized in the original CTSA RFA (Request for Application) the initial impetus was to build the infrastructure to support, or enable, a researcher to engage in translational research. This was to be accomplished by providing training and mentoring, biostatistical support, and informatics infrastructures for example. Documenting that this foundation has been built and is being used by researchers has been and remains a critical component of any CTSA evaluation (IOM, 2013). However, as important as this descriptive effort is and was, it is an incomplete evaluation of the transformative expectations of the CTSAs.

The incompleteness of this effort can be seen by using the example of team science. In the discussion under Item 4 above, a key requirement for all CTSAs from the very beginning was the development of research training and career development programs that emphasize interdisciplinary and team science. While the evaluation of the process and outcomes of these programs is commonplace among the CTSAs, there are multiple levels and dimensions that could be tracked and assessed.

There are critical strategies that affect outcomes for transforming the scientific workforce that relate to institutional and science policies that may need to be altered. Team science requires an infrastructure that values individuals who are not the principal investigators, who are not the first author, and who can work with and as part of a multidisciplinary team. If we want individual scientists to move from being independent practitioners to working as partners, they need incentives to do so and these incentives must be structured and provided at the institutional and federal levels. For example, medical schools may need to modify criteria for promotion so that being a first author or the recipient of an R01 is valued and incorporated into promotion criteria (Mann, 2013). Being a second author, writing parts of the R01 should count. In addition, we may want more formal acknowledgments of investigators who work with multiple disciplines and departments. Changes at NIH may also be necessary, and a review panel may also need to learn that by being a team scientist an investigator should not be penalized for not having R01s, or having none or limited first author articles. These changes are hard to define and institute and will take time to assess. But arguing for these changes will require institutional leadership by the CTSA principal investigators, as it will require a bit of institutional entrepreneurship—negotiating, bargaining, and cajoling (Thornton, Ocasio, & Lounsbury, 2012)—to address existing organizational priorities, hierarchies, and power structures.

This depiction of what a CTSA may need to accomplish in transforming how research is conducted also underscores two cautions in the IOM report: that it may not be feasible for a CTSA evaluation to answer immediately many of the complex questions raised by the creation of a CTSA and that some of the evaluation questions raised by the formation of a CTSA may not be measurable. For the CTSA evaluator, these cautions are reminders to not over promise on what can be delivered. But they are also a reminder that evaluators need to take these complex entities and conduct evaluations of them that can contribute new insights and innovative methods for evaluating complex interventions, that is, transformations that address the individual, interpersonal, and organizational dimensions of translational research.

Address the Challenge of the Long-Term Impact of CTSAs

In the long run, we are all dead

The recent IOM evaluation of the CTSA program strikes both a note of caution and a challenge for evaluators. It describes the CTSA program as a bold initiative with the lofty and challenging goals of facilitating and accelerating translational research, ultimately to improve human health. But IOM cautions:

Although it would be ideal to evaluate the CTSA Program’s impact on clinical care and public health, currently this is neither feasible nor realistic given the numerous driving forces that shape the research enterprise….and the multitude of factors that affect health outcomes In addition, there are multiple direct and indirect ways in which the CTSA Program contributes to research infrastructure and resources, collaborations, cultural changes, training, and community engagement that influence clinical and translational research but cannot be easily identified or measured. (p. 89)

Later, and on the same page the report indicates that:

Despite these and other such challenges, the CTSA Program can be a leader in developing evaluation methodologies and metrics that could provide more real-time assessments of progress in advancing clinical and translational research, overcoming research barriers, fulfilling the program’s mission and strategic goals, and, whenever possible, changing clinical care and improving public health. (p. 89)

The challenges of determining causality should not diminish the responsibility of evaluators to discover new methods to organize and test a body of knowledge that leads us toward more rigorous measures and technologies for assessing progress toward achievement of CTSA goals. The role of science in health policy is to (1) identify problems such as infant mortality, obesity, or research translation, (2) measure the magnitude and seriousness, (3) review alternative policy interventions, (4) systematically assess the likely consequences of particular policy actions—intended and unintended, desired and unwanted, and (5) evaluate what, in fact, results from policy. Across all these tasks, there are political and value considerations that are outside the scope of science but a systems perspective can help decision makers and researchers think broadly about the many effects a policy may produce and the ways in which a planned social intervention interacts with other existing interventions and institutional practices (NRC, 2012b).

It may be argued that evaluating the impact of the CTSA Program on clinical and public health is not only feasible and realistic, it is practically required in order to assure reasonable progress in achieving CTSA goals. Prior studies show how various system inputs lead to certain intermediate outcomes that result in ultimate population health outcomes. They illustrate the complexity of causal factors and the interactions among them. With innovative methods and metrics, evaluators may document efficiencies and inefficiencies of the translational research enterprise and over time contribute to improving the health of patients and communities. Modeling is a suggested method to inform decision makers about interventions that are likely to have the greatest impact (Diez Roux, 2011; IOM, 2013; NRC, 2013).

While there have been larger scale assessments of the CTSA Program (Frechtling, Raue, Michie, Miyaoka, & Spiegelman, 2012), there is currently no plan to systematically repeat these efforts. Importantly, as the IOM report noted, such an evaluation plan must go substantially beyond the annual progress reporting and the renewal cycles and make a true contribution to science policy.

Contribute to Science Policy Formulation and Implementation

If you don’t know where you are going, it’s hard to get there

CTSA evaluation activities should be conceptualized as part of a broader effort to build the science of science policy and management. By establishing translational research as a major policy and investing a significant component of NIH resources, the strong scientific leadership at NIH has made a choice among alternate policy options. The strong evaluation activities both within and across CTSAs are ultimately addressing the results from policy choices made at both the highest level of policy and decision making within the CTSAs. The irony is that most policy and management decisions affecting science (at both national and institutional levels) are based on processes that almost devoid of a scientific basis. For example, Balas & Elkin (2013) considers the evidence base for the 17-year translational cycle that was at the basis of the development of the CTSA program. Challenging the methodology of the 17-year “fact,” Balas demonstrates that what we have labeled as facts are not really facts. He suggests that new and better methods and models are needed to assess the translational process in a more valid and useful way (Trochim, Kane, Graham, & Pincus, 2011). Every day choices are made about which science programs will be initiated, which grants will be funded, which research training strategies will be implemented, and so on, with little or no systematic data to guide those decisions. The science of science management aims to instill empirical and methodological rigor in the decision making behind research investments. CTSA evaluators have to engage in a larger conversation around measuring the impact of investments in science and innovation by developing tools to answer questions about the value of the nation’s investment, ways to predict discovery, and methods to determine the effectiveness of investments (Office of Science and Technology Policy, National Science and Technology Council, 2008).

Building an infrastructure for evidence-based policy making is not a simple task. Several efforts across public and private organizations are being undertaken (cite NSF [National Science Foundation], OSTP [Office of Science and Technology Policy], AAMC [Association of American Medical Colleges], AAAS [American Association for the Advancement of Science], RAND). And certainly, placing an evaluation function into CTSAs was a step forward toward putting “science” into science policy. But much more needs to be done. For example, every time the NIH (or other science agency) funds a grant, it is essentially making a hypothesis that there will be certain outcomes emanating from that decision. Unfortunately, rarely is that hypothesis made explicit, nor is there a strategy in place to test that hypothesis or even to obtain systematic data about those outcomes in a meaningful way. Developing a culture, capacity, and infrastructure to do this would go a long way toward enhancing our ability to assess the impact from NIH programming and funding mechanisms

CTSA evaluators have been given an important opportunity to engage in a larger conversation around measuring the impact of investments in science and innovation by developing tools to answer questions about the value of the investment, ways to predict discovery, and the methods to determine the effectiveness of investments. The common metrics project is an example of this kind of opportunity. This project will look across all CTSAs to assess the impact of CTSAs on the most salient metrics that will shed light on the impact of individual CTSAs on their institutions and also the impact of CTSAs overall on the science infrastructure (Rubio, 2013).

Conclusion

The establishment of CTSAs has not only created a major shift in how the infrastructure for translational science is being implemented but also, as demonstrated by the contributions in this special issue, provides an important substrate for the further development of an evidence-based science policy. The mandate that CTSAs formally incorporate evaluation as a key function provides a stepping-stone toward enhancing the use of evaluation in making decisions throughout the scientific establishment. It also places an enormous responsibility on evaluation: to make sure the needs of various stakeholders are considered, measuring and improving service quality, understanding and assessing the importance of teams and individuals, creating efficiency and value with the resources available, using the right methods, developing innovative evaluations for large, complex grants, such as the CTSA, being aware of the challenges of measuring transformation of the research enterprise and the long-term impact of CTSAs, and keeping in mind the science policy and decisions being made by NIH leadership in funding academic institutions receiving CTSAs.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The contributions of Harold Pincus and Zainab Abedin were supported by the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH), through grant number UL1 TR000040. The contributions of Arthur Blank were supported by NCATS/NIH grant number 8UL1 TR000086. The contributions of Paul Mazmanian were supported by NCATS/NIH award number UL1 TR000058. The content of the article is the sole responsibility of the authors and does not necessarily represent the official views of the NIH.

References

Allen

Ripley

Coe

Clore

(2013). Reorganizing the General Clinical Research Center to Improve the Clinical and Translational Research Enterprise. Evaluation and the Health Professions, 36, 492–504.

Balas

E. A.

Elkin

P. L.

(2013). Technology Transfer From Biomedical Research to Clinical Practice: Measuring Innovation Performance. Evaluation and the Health Professions, 6, 505–517.

Butler

(2008). Translational research: Crossing the valley of death. Nature, 453, 840–842.

Contopoulos-Ioannidis

D. G.

Ntzani

Ioannidis

J. P.

(2003). Translation of highly promising basic science research into clinical applications. American Journal of Medicine, 114, 477–484.

Cook

T. D.

Campbell

D. T.

(1979). Quasi Experimentation: Design & Analysis Issues for Field Settings. Chicago, IL: Rand McNally.

Cronbach

L. J.

(1980). Toward the reform of program evaluation. San Francisco, CA: Jossey-Bass.

Cronbach

L. J.

(1982). Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.

Crowley

W. F.

Jr. (2003). Translation of basic research into useful treatments: How often does it occur? American Journal of Medicine, 114, 503–505.

Dembe

A. E.

Lynch

M. S.

Gugiu

P. C.

Jackson

R. D.

(2013). The translational research impact scale: development, construct validity, and reliability testing. Evaluation and the Health Professions. Advance online publication. doi:10.1177/0163278713506112

10.

Denzin

N. K.

Lincoln

Y. S.

(Eds.). (1998). The landscape of qualitative research. Thousand Oaks, CA: Sage.

11.

Dilts

D. M.

(2013). A “Three-Plus-One” Evaluation Model for Clinical Research Management. Evaluation and the Health Professions, 36, 464–477.

12.

Dilts

D. M.

Cheng

Crites

J. S.

Sander

A. B.

Doroshow

(2010). Phase III clinical trial development: A process of chutes & ladders. Clinical Cancer Research. 16: 5381–89. PMID: 21062928.

13.

Diex Roux

A. V.

(2011). Complex systems thinking and current impasses in health disparities research. American Journal of Public Health, 101, 1627–1634.

14.

Dozier

A. M.

Martina

C. A.

O'Dell

N. L.

Fogg

T. T.

Lurie

S. J.

Rubinstein

E. P.

Pearson

T. A.

(2013). Identifying Emerging Research Collaborations and Networks: Method Development. Evaluation and the Health Professions. Advance online publication. doi:10.1177/0163278713501693

15.

Falk-Krzesinski

H. J.

Contractor

Fiore

S. M.

Hall

K. L.

Kane

Keyton

… Trochim

. (2011). Mapping a research agenda for team science. Research Evaluation, 20, 145–158.

16.

Feeney

M. K.

Johnson

Welch

E. W.

(2013). Methods for identifying translational researchers. Evaluation and the Health Professions. Advance online publication. doi:10.1177/0163278713504583

17.

Fetterman

D. M.

(2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage.

18.

Frechtling

Raue

Michie

Miyaoka

Spiegelman

. (2012). The CTSA national evaluation phase 1 final report. Rockville, MD: Westat. Retrieved August 28, 2013 from https://www.ctsacentral.org/sites/default/files/files/CTSANationalEval_FinalReport_20120416.pdf

19.

Gaglio

Shoup

J. A.

Glasgow

R. E.

(2013). The RE-AIM framework: A systematic review of use over time. American Journal of Public Health 103, 38–46.

20.

Glasgow

(2007). A focus on external validity. Evaluation and the Health Professions, 30, 115–117.

21.

Grazier

K. L.

Trochim

W. M.

Dilts

D. M.

Kirk

(2013). Estimating Return on Investment in Translational Research: Methods and Protocols. Evaluation and the Health Professions, 36, 478–491.

22.

Greene

J. C.

(2007). Mixed methods in social inquiry. Thousand Oaks, CA: Sage.

23.

Hogle

J. A.

Moberg

D. P.

(2013). Success Case Studies Contribute to Evaluation of Complex Research Infrastructure. Evaluation and the Health Professions. Advance online publication. doi:10.1177/0163278713500140

24.

Institute of Medicine. (2013). The CTSA program at NIH: Opportunities for advancing clinical and translational research. Washington, DC: The National Academies Press.

25.

Kane

Alexander

Hogle

J. A.

Parsons

H. M.

Phelps

(2013). Heterogeneity at Work: Implications of the 2012 Clinical Translational Science Award Evaluators Survey. Evaluation and the Health Professions, 6, 447–463.

26.

Mann

. (2013). Team-based science increases, but promotion and tenure lag behind: AAMC Reporter. Retrieved August 2013, from https://www.aamc.org/newsroom/reporter/jan2013/325940/team-based-science.html

27.

Mazmanian

P. E.

Coe

B. A.

Evans

J. E.

Longo

D. R.

Wright

B. A.

(2013). Are researcher development interventions, alone or in any combination, effective in improving researcher behavior? A systematic review. Evaluation and the Health Professions. Advance online publication. doi: 10.1177/0163278713510375

28.

Medical Research Council. (2000). A framework for development and evaluation of complex interventions to improve health. London, England: Author.

29.

Mercer

S. L.

DeVinney

B. J.

Fine

Green

L. W.

Dougherty

(2007). Study designs for effectiveness and translation research: Identifying trade-offs. American Journal of Preventive Medicine, 33, 139–154.

30.

Mills

B. A.

Caetano

Rhea

A. E.

(2013). Factor structure of the clinical research appraisal inventory (CRAI). Evaluation and the Health Professions. Advance online publication. doi: 10.1177/0163278713500303

31.

National Research Council. (2004). Facilitating Interdisciplinary Research. Washington, DC: The National Academies Press.

32.

National Research Council. (2012a). Research universities and the future of America. Washington, DC: The National Academies Press.

33.

National Research Council. (2012b). Using science as evidence in public policy. Washington, DC: The National Academies Press.

34.

National Research Council. (2013). Best care at lower cost: The path to continuously learning health care in America. Washington, DC: The National Academies Press.

35.

Office of Science and Technology Policy, National Science and Technology Council. (2008). The science of science policy: A federal research roadmap. Retrieved from: http://scienceofsciencepolicy.net/sites/default/files/public_files/0257.SoSP_Roadmap.pdf

36.

Olsen

Aisner

McGinnis

(Eds.). (2007). The learning health care system: Workshop summary. Washington, DC: National Academy Press.

37.

Patton

M. Q.

(2008). Utilization focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

38.

Pawson

Tilley

(1997). Realistic Evaluation. Thousand Oaks, CA: Sage.

39.

Rawlins

(2008). De testimonial: On the evidence for decisions about the use of therapeutic interventions. Lancet, 372, 2152–2161.

40.

Rubio

D. M.

(2013). Common Metrics to Assess the Efficiency of Clinical Research. Evaluation and the Health Professions, 36, 432–446.

41.

Scott

C. S.

Nagasawa

P. R.

Abernethy

N. F.

Ramsey

B. W.

Martin

P. J.

Hacker

B. M.

Schwartz

H. D.

Brock

D. M.

Robins

L. S.

Wolf

F. M.

Carter-Dubois

Disis

M. L.

(2013). Expanding Assessments of Translational Research Programs: Supplementing Metrics With Value Judgments. Evaluation and the Health Professions. Advance online publication. doi:10.1177/0163278713500984

42.

Shadish

W. R.

Cook

T. D.

(2009). The renaissance of field experiments in evaluating interventions. The Annual Review of Psychology, 60, 607–629.

43.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and Quasi- experimental designs for generalized causal inference. Boston, MA: Houghton-Miflin.

44.

Stake

R. E.

(2006). Multiple case study analysis. New York, NY: Guilford Press.

45.

Thornton

P. H.

Ocasio

Lounsbury

(Eds.). (2012). The Institutional logics perspective: A new approach to cultures, structure, and process. Oxford, England: Oxford University Press.

46.

Tillman

R. E.

Jang

Abedin

Richards

B. F.

Spaeth-Rublee

Pincus

H. A.

(2013). Policies, activities, and structures supporting research mentoring: A national survey of academic health centers with clinical and translational science awards. Academic Medicine, 88, 90–96.

47.

Trochim

Kane

Graham

M. J.

Pincus

H. A.

(2011). Evaluating translational research: A process marker model. Clinical and Translational Science, 4, 153–162.

48.

Wooten

K. C.

Rose

R. M.

Ostir

G. V.

Calhoun

W. J.

Ameredes

B. T.

Brasier

A. R.

(2013). Assessing and evaluating multidisciplinary translational teams: A mixed methods approach. Evaluation and the Health Professions. Advance online publication. doi:10.1177/0163278713504433

49.

Yin

R. K.

(2009). Case study research: Design and methods (4th ed.). Thousand Oaks, CA: Sage.