Abstract
Assessing the value of clinical and translational research funding on accelerating the translation of scientific knowledge is a fundamental issue faced by the National Institutes of Health (NIH) and its Clinical and Translational Awards (CTSAs). To address this issue, the authors propose a model for measuring the return on investment (ROI) of one key CTSA program, the clinical research unit (CRU). By estimating the economic and social inputs and outputs of this program, this model produces multiple levels of ROI: investigator, program, and institutional estimates. A methodology, or evaluation protocol, is proposed to assess the value of this CTSA function, with specific objectives, methods, descriptions of the data to be collected, and how data are to be filtered, analyzed, and evaluated. This article provides an approach CTSAs could use to assess the economic and social returns on NIH and institutional investments in these critical activities.
Introduction
With support of the National Institutes of Health (NIH), the Clinical and Translational Science Award (CTSA) program was launched in 2006 and expanded to other academic medical institutions across the country. By 2012, there were approximately 60 CTSA-supported institutions, known as CTSAs, with the goal of the CTSA program to provide a nationwide collaborative of integrated infrastructures to support, educate, and accelerate clinical and translational health research. The CTSA program is now under the umbrella of a relatively new NIH unit that was established in 2011, the National Center for Advancing Translational Sciences (NCATS; Clinical and Translational Science Award [CTSA], 2013).
In an era of increasingly scarce resources, important decisions with respect to which resources should be maintained by a CTSA, and which should not be renewed, become crucial for the future of all CTSAs (CTSA, 2013). Effective evaluation has been subject to much discussion within NIH, the CTSA program, and individual CTSAs in recognition that it takes on average 17 years for only 14% of scientific innovations and discovery to reach clinical practice (Balas & Boren, 2000) and the consequent importance of engaging with communities and practice-based networks to accelerate translation (Westfall, Mold, & Fagnan, 2007). The aims of this article are 4-fold: (1) to examine the concept of return on investment (ROI) as it could be applied to CTSA program resources as used at individual institutions; (2) to propose a model for applying ROI formulae using data currently collected from CTSA program required financial and operating data; (3) to propose a methodology for decision making with respect to ROI in one component of a CTSA, namely a clinical research unit (CRU); and (4) to suggest how the methodology, an evaluation protocol, can be applied to other units within, and across, the various CTSAs supported by the CTSA program (Trochim et al., 2012).
Limited ability and experience assessing the value of CTSA research funding on accelerating the translation of scientific knowledge is a fundamental issue faced by both individual CTSAs and by NIH CTSA program (Rubio, Sufian, &Trochim, 2012). To address this issue, we propose investigating the ROI of one key program that is common to all CTSAs, namely the CRU (McCammon et al., 2013). By carefully examining the economic and social inputs and outputs of these units, it may be possible to produce multilevel ROI computations, at the investigator, program, institutional, and national levels. The developed methodology, or evaluation protocol, will focus on achieving specific objectives, methods, descriptions of the data to be collected, how data are to be filtered and analyzed, and how the results can be used in evaluating various units. This model, while being created using one component of an individual CTSA, is developed in such a way that it is generalizable to other CTSA program aspects at an individual institution, such as pilot projects or investigator training programs.
Background and Significance
As Botchkarev and Andru (2011) note, “ROI was conceived as a financial term and defined as a concept based on a rigorous and quantifiable analysis of financial returns and costs. At present, ROI has been widely recognized and accepted in business and financial management in the private and public sectors.” Authors recognize differences in economic concepts, based on the field of the research, namely, finance or economics. In ROI, the method allows a decision maker to evaluate the timing and magnitude of expected gains to the timing and magnitude of investment costs (National Information Center on Health Services Research and Health Care Technology, 2013). The simplest ROI divides the incremental economic gain from an action by its investment costs. When controlled for similar circumstances, the higher the ROI, the greater the financial return for the given investment and, presumably, the better use of the resources. Direct costs, such as salaries and wages, can be attributed directly to the investment, or project. The same is true for the direct returns, such as increased sales revenue. Proximal measures of cost and gains, or returns, are also included, insofar as they can be identified with the specific investment, and tracked for sufficient length of time. The analysis grows in complexity with the recognition of several important dimensions of the economic value implied by the ratio, the most important being the timing of the respective cost outlays and revenue inflows (The Cochrane Collaboration, 2013).
While ROI is fairly straightforward if costs and revenues can be directly identified to a project, difficulties arise in the use of ROI when it attempts to include indirect costs or returns, those associated with the decision, but not necessarily caused by it; for example, there are general expenses related to operating a CTSA at an institution, but it is difficult to attribute many of those expenses directly to one aspect of the CTSA, be they a project or unit. It is also difficult to quantify on the return side of the equation, the value supplied by the CTSA in generating a journal article or patent when there are multiple sources of funds available to an investigator, including grants and other outside funding. Additionally, the timing of the investment by a CTSA and the returns provided by the investment frequently differ. For example, initial clinical funding might be invested in Year 1, but the return as measured by additional grant awards may not occur until years later. These early cost/later gains scenarios require discounting future net cash flows to recognize the risk related to the uncertainty inherent in estimating those future values (Phillips & Phillips, 2008; Zhang, Wu, & Zhang, 2008). In this article, the method does not restrict ROI to a simple ratio, but rather one that accounts for the proximal and distal costs and benefits of investments in CRUs. It should also be noted that in those CRUs that offer services for industry-sponsored trials, the calculations can often be simplified by imposing a fixed timeline on the returns.
In addition to the economic ROI, which focuses on financial value, some formulae include social costs and value, which is commonly referred to as social return on investment (SROI; Harvard Business School, 2000). SROI is a framework for measuring and accounting for a broader concept of “value,” one that incorporates social and environmental, as well as economic costs and benefits (Gardner, 2007; International Organization for Standardization, 2013; NEF, 2013; Staiger, Richardson, & Barbara, 2005). The academic and policy-making literature have provided evidence for the importance of calculating SROI, including justification, protocols, and mechanisms for organizing and conducting a rigorous SROI in settings similar to that found in CTSAs (DeVol & Bedroussian, 2006; Pienta, Alter, & Lyle, 2010). SROI has been assessed in different fields: banking, corporate research and development, energy policy, and education policy (Blaug, 1997; Jones & Williams, 1998; Kronenberg, Kuckshinrichs, & Hansen, 2010; Nelson, Cooper, Wright, & Murphy, 2009; Raymer, 2009; Richardson, 2006; Tulchin, Gertel-Rosenberg, & Olsen, 2009). As with ROI, SROI analysis can be conducted both retrospectively, based on actual realized costs and outcomes; or prospectively, predicting how much social value will be created, for a given cost, if the activities meet their intended outcomes (Lingane & Olsen, 2004; Scottish Government, 2009).
The variation in the meaning and use of ROI, how it is calculated, and at what level, are described well in several publications. Those authors accept for their evaluation purposes an individual measure of ROI, as a metric and ratio. Other authors consider ROI “as a method of persuasive communication to senior management, a process of getting everybody’s attention to the financial aspects of the decisions and stimulating a rigid financial analysis.” In this case, actually calculated ROI numbers are of less importance compared to the processes of gathering/analyzing cost and benefit data (Botchkarev & Andru, 2011).
Approach
The approach uses quantitative and qualitative methods to determine how to extend operational protocols to assist individual CTSAs in understanding and using data representing returns on investments in research funds in CRUs. Using a discrete program within all CTSAs, the CRU, this approach encompasses unique and similar features of administrative, clinical, and research-tracking systems (Meltzer & Smith, 2011).
Basic principles drive the approach and methods: involve stakeholders; understand what changes over time; value the things that matter; only include what is relevant; be conservative; be transparent; and verify results.
The proposed evaluation protocol shows that the concept of ROI models can be adapted to better understand and manage the activities of an individual CTSA with respect to investment decisions.
Measurement
Measures of the value of research awards often include “productivity.” Productivity is commonly defined as a ratio between the output volume and the volume of inputs (Nordhaus, 2001). It measures how much inputs such as labor and capital are used in an economy to produce a given level of output (Linna, Pekkola, Ukko, & Melkas, 2010; Velentgas, Dreyer, Nourjah, Smith, & Torchia, 2013). Research productivity is often represented by the publications of research discoveries and how often the work is cited by others (Meltzer & Smith, 2011; National Institute of Mental Health, 2013). Rooted in the idea of a data life cycle, the scientific community has moved to recognize “that research data may have an enduring value on scientific progress as scientists use and reuse research data to draw new analysis and conclusions” (Jacobs & Humphrey, 2004; Levan & Stephan, 1991; Pienta et al., 2010). Some of these data sharing opportunities are encouraged by journals with the intent to replicate results (Anderson, Greene, McCullough, & Vinod, 2005; Glenditsch, Metelits, & Strand, 2003). NIH issued its final ruling in 2003 on the requirements to share data funded by the National Institutes (Carnegie Foundation for the Advancement of Teaching, 2010; National Institute of Health, 2003). Such data sharing, in ROI terms, can be considered secondary returns.
Data sources and collection techniques include literature review, in-person and telephone interviews; extraction of data from administrative and research data systems; surveys of a sample of investigators using and not using the CRU; online databases of independent scientist and career development (K) awards; and subsequent publications and employment. Because of interviews with the data managers at the CRU, and CTSA-specific data scans, the evaluation protocol guides the evaluator through standardized processes for collecting and aggregating data, validating for errors, and transmitting the data sets to the analyst.
CRUs are likely to report economic data more consistent with standard financial records for fixed assets such as property, plant, and equipment; and variable costs, such as those associated with personnel staffing the units. However, it is very likely that the institutions will provide different patterns of service (e.g., different eligibility rules or different terms of service) and account for these units in significantly different ways. Additional challenges include valuing different components of the CRUs, such as inpatient, outpatient, and mobile services. A key focus of interviews is on developing consistent and comprehensive definitions of terms and outcomes.
Analysis
The value or return will be a function of a number of characteristics: the awards through the CTSA and from other sources; the institutions at the time of the award and before and after; the investigator; the number of collaborations in the award, length and extent of “exposure” to the CRU of the research programs; all dependent on the scope and boundary discussions with stakeholders and on the synthesized model constructed.
There are several sets of potential models for each outcome; for instance, a model may include categorized data sharing status measures, wherein others may include principal investigator (PI), institution, and other award characteristics. Depending on the type of outcomes being measured and the context of the ROI calculation, regression models or Data Envelopment Analyses can be used. For instance, if the outcome were publication counts, Poisson regression models might be of use; whereas in the case of longitudinal publication outcomes, negative binomial regression models may be in order. A hierarchical set of models may help understand the extent to which differences in the outcome of interest may be attributable to characteristics of the unit, the stage of career, PI collaborations, or size and timing of the award. With such data, it will also be possible to compare relative effectiveness of investments across project times and across institutions.
Typically, ROI estimation is approached very simply. Total “returns” (e.g., monetized benefits) are divided by total “investments” (e.g., costs) to get the ratio of returns for each dollar invested. However, these simplistic analyses do not enable looking at distributions and variability or allow for statistical tests of differences. Using data that have not yet been aggregated into gross categories enables use of statistical methods rather than just reporting aggregate ROI.
Model development follows an iterative process, which follows a spiral development path (Ambler, 2002). That is, the model will begin as simple as possible, uncovering the basic issues involved in model development. Once these issues are resolved and tested using the data provided in the data collection phase discussed above, the model is enhanced and detailed at the next level of complexity and performance. Using such a process allows for both the development of a rapid, more local, decision tool and for the continuing development of a more complex and generalizable decision tool.
As stated before, while the basics of ROI are simple, other issues can make the use of ROI more challenging. This is particularly true when a return can only be realized years or decades after the investment (National Center for Advancing Translational Sciences, 2012). Discounted ROI is well known for being highly biased toward rapid investment returns. This is a major issue for CTSAs as, for example, investing “time” in new researchers today by allowing them to use a CRU should provide “return” in terms of new discoveries in the future; but how exactly should each be quantified? As one of the four transformative aims of the CTSA program is to provide a foundation of shared resources that could reduce costs, delays, and difficulties experienced in clinical research, including trials, this timing is particularly crucial (http://www.ncats.nih.gov/research/cts/ctsa/about/about.html).
Additionally, nonfinancial characteristics of both investment and return can be difficult to identify in commensurate terms. For example, time is the most inelastic and finite of all resources but it must be expended in teaching new investigators in hopes that they do better and more meaningful research in their subsequent careers. While it is possible to achieve significant “return” with completely one-on-one responsiveness to the researcher’s demands in the CRU, the investment in time usually is prohibitive with respect to the relative investment. But, only offering group instruction or supervision (a lower investment alternative) may not provide the necessary quality (i.e., return) required. One major emphasis of the proposed modeling approach is identifying comparable metrics within a function.
Methodology Development: Evaluation Protocol
The proposed evaluation protocol addresses both standard ROI and SROI estimation methodologies but focuses on the economic ROI. Work with key stakeholders helps establish the scope and boundaries of the analysis for each program. This is not a trivial issue in ROI analysis. For example, there are a number of direct and indirect potential outcomes of given clinical trial projects: subsequent research publications; patent applications and patents received; subsequent grants received; and even the economic effects of spending the funds such as their stimulus to the local economy. There is no effective way to monetize all of these outcomes and the decision regarding which to include in ROI analysis is to some extent a matter of judgment. After meeting with stakeholders and determining their within-center approach to boundary conditions, data elements can be selected.
Results: Process and Structure for ROI Analysis
A multistep process for structuring the ROI analysis is summarized as follows:
Create alternative conceptual frameworks to estimate the impacts and value.
Survey CTSA on available sources and formats of economic and social impact data; determine costs of collecting and analyzing data.
Collect selected financial, service utilization, and community encounter and impact data from collaborating sites.
Test usability of each framework and efficacy of resulting metrics.
Create evaluation protocol for use by CTSA in pilot and final testing.
In planning the project, it is important to identify a conceptual model that is acceptable to the CTSA; this can be determined through interviews with staff and investigators within the CTSA and the CRU itself.
The process for collecting and analyzing the data to calculate the ROI is detailed in Figure 1. Here, evaluators define relevant data, examine the quality of existing data, and standardize methods for collecting and analyzing data; these steps result in selected mechanisms for further testing and adoption. The analysis uses accounting, financial, economic, and SROI principles to identify outcomes and value impact.
Estimating the ROI: process steps. ROI = return on investment.
Types of costs and gains, or benefits, are listed in Figure 2a and 2b. Limiting the initial work to direct and indirect costs, and not including incidental costs, is less complex and may allow the CTSA to move forward more quickly with these types of analyses.
The standard model of ROI estimates:
(a) ROI financial flows. (b) Categories of financial flows. ROI = return on investment.
Considering the timing and magnitude of cash flows recognizes the impact of early versus later gains and costs. This recognition is captured in the discount rate, the percentage used to discount future net cash flows to recognize the risk or uncertainty of estimating net gains into the future. Discounting cash flows requires selecting a percentage rate based on an estimate of how “risky” the investment is relative to other projects in which the funder invests. In standard businesses, the discount rate is the average of the interest rate on their debt and the return shareholders expect on their investments in the company. This is an average weighted by the portion of the company that is financed by debt and financed by equity. In the CRU methodology, the discount rate can be selected using a sensitivity analysis varying from the interest rate on medium term interest rates in the commercial loan market to inflation rate plus 1–3%. This level of riskiness of the federal investment in the CTSA is conservative, but realistic.
Summary and Conclusion
This article proposes using several approaches to study quantitatively the availability, accessibility, and quality of data used to define ROI; and qualitatively seek additional process input into the financial and social models as they are developed and tested. The protocol includes identifying types of costs, impacts and values (external, internal, financial, social); creating alternative conceptual frameworks to estimate the impacts and value of translational research on individual researchers, the research enterprise, consumers of research and clinical care, and the public; surveying CTSAs on available sources and formats of economic and social impact data; determining costs of collecting and analyzing financial data; testing usability of each framework and efficacy of resulting metrics; and creating protocols for use by CTSAs.
Creating and sustaining the next generation of clinical and translational research, researchers, and practitioners within a culture of innovation and excellence requires thoughtful and fair allocation of resources. While not the only criteria for investment, the outputs in productivity, creativity, efficiency, and better health status warrant measurement. As in business in general, CTSAs would benefit from the ability to use standardized methods and tools to measure ROI. Realizing this need, some CTSAs are embarking on efforts to identify the investment, benefits, and ROI for their CRUs. Through this testing of the proposed model, the NIH can assure that this method of accountability and resource allocation can become one of the several tested criteria to help make difficult but crucial decisions on the future of science and public health.
Footnotes
Authors’ Note
The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by awards to the University of Michigan CTSA (Grant number 2UL1TR000433); Weill Cornell Medical College CTSC (Grant number UL1RR024996); and Oregon Clinical and Translational Research Institute (OCTRI), Grant number (UL1TR000128) from the National Institutes of Health (NIH) National Center for Advancing Translational Sciences (NCATS).