Multi-Dimensional Impact of the Public–Private Center for Translational Molecular Medicine (CTMM) in the Netherlands: Understanding New 21 st Century Institutional Designs to Support Innovation-in-Society

Introduction

Investment in biomedical research has fueled the development of new medical technologies (i.e., drugs, diagnostic methods, drug delivery systems, and medical devices). Herewith, astonishing medical advances have been achieved, with stimulation of high-wage job growth and economic health.

Public investments have been an important enabler of biomedical research and development (R&D). Between 2007 and 2012, the United States contributed the largest share of total global public sector expenditures (circa 52%), followed by Europe (circa 27%) and Asia-Oceania (circa 18%). The private sector has even shouldered a larger proportion of biomedical R&D expenditure, on balance approximately 64% of overall spending (Chakma et al., 2014).

In order for societies to enjoy the expected benefits of these investments fully, biomedical research findings have to be moved from “bench to bedside” and into the community (Rubio et al., 2010). Motivated by the assertion that a time lag of circa 17 years for research evidence to reach clinical practice is unnecessarily long, the concept of translational research has attracted large interest from both public and private parties (Balas and Boren, 2008; Contopoulous-Ioannidis et al.; 2008, Trochim et al.; 2011; Westfall et al., 2007).

Various countries established programs to incentivize translational research. A widely known example is the Clinical and Translational Science Awards (CTSA) Program of the National Institutes of Health (NIH) in the US. In the Netherlands, the Center for Translational Molecular Medicine (CTMM) was established in 2008. It was born out of recognition by academia, industry, and the Dutch government that translating fundamental research into patient benefit and economic activity requires a truly multidisciplinary end-to-end approach that involves all parties in the value chain. The public–private partnership, including universities, academic hospitals, and pharmaceutical and medical technology companies, aims to accelerate molecular diagnostics and imaging technologies to enable determination of predisposition, early diagnosis, and personalized treatment of patients.

Ultimately, CTMM aspires to reduce mortality, morbidity, as well as health care costs significantly, by funding translational research projects in high-burden disease areas. With a total research budget of circa €300 million, including 50% government, 25% university, and 25% commercial funding, CTMM was one of the first and largest public–private biomedical research partnerships internationally. Herewith the need arose to assess its impact. The key findings regarding CTMM's scientific, translational, clinical, and economic impact, as achieved 5 years after becoming operational in 2008, are described in this report.

Methods

Study setting

At the beginning of 2012, CTMM had brought together 124 active project partners, including 27 academic institutes and 97 industrial partners. Figure 1 shows the development of CTMM's partner portfolio over time. Notably, CTMM has attracted a relatively large number of Small and Medium Enterprises (SMEs) from the start, up to 74 SMEs in year 2012. Partners are clustered into project consortia, with circa 9–11 partners per project, and are supplied with the resources and infrastructures needed to achieve results collectively. The number of projects increased from 7 in year 2008 to 22 in year 2012 (Fig. 2). Projects examined in this report have had a timeframe of 4–5 years and were finalized no later than 2015.

OPEN IN VIEWER

FIG. 1.

Development of CTMM Partner Portfolio 2008–2012.

OPEN IN VIEWER

FIG. 2.

Number of CTMM projects by year and disease area.

CTMM projects focus on four disease areas that are associated with high human burden and high societal (oncology, cardiovascular, neurodegenerative, infection, and immunity-related diseases). In addition, an overarching Information Technology (IT) project was established in October 2011, to build a long-lasting IT infrastructure for the Netherlands to facilitate the collection, storage, analysis, archiving, and securing of the vast amounts of data generated in CTMM's scientific program.

Results

Investigators were given full access to the required administrative data of CTMM. A total of 197 responses to the survey made for a small (17%), though arguably representative, sample of CTMM participants considering its public–private composition, including 25% private partner responses, 70% public partners responses (23% from universities, 47% from university medical centers and general hospitals), and 5% responses from health foundations. Further, each CTMM project was represented in the sample by both private and public partner respondents and the majority of respondents came from the cardiovascular and oncology projects (i.e., 77%). This appropriately reflects the amount of projects in these disease areas compared to the neurodegenerative, immunology, and infection, and the IT projects that make up 23% of the total projects. Twelve CTMM representatives participated in the group interviews, including 7 project leaders and 5 management team/board members of CTMM.

Scientific impact

Between 2008 and 2012, the CTMM projects collectively published 139 papers in scientific journals. The absolute majority of the papers was published by cardiovascular and oncology consortias (55 and 69, respectively), which reflects the larger number of projects in these disease areas. The average number of publications per project was 7.7 (SD = 5.4). Important to note is that, at the time of this evaluation, another 306 papers were submitted, likely resulting in a total of 445 full papers by the end of 2013.

The average impact factor of journals in which CTMM publications have appeared is consistently higher than the average JCR 2012 journal impact factor of that disease area (Fig. 3). The associated standard deviations, however, are substantial (6.8 ± 5.8 for cardiovascular; 6.1 ± 5.5 for infection and immunity; 5.7 ± 5.5 for oncology, and 4.5 ± 2.3 for neurodegenerative disease articles). This is because translational articles, such as produced within CTMM, are not necessarily published in the specific disease-oriented journals but also in general (for example, translational, multidisciplinary, or methodological) journals. Such a broad collection of journals likely sees a larger variation in their impact factors compared to disease-specific journals.

OPEN IN VIEWER

FIG. 3.

Average impact factor CTMM publications vs. average JCR (2012) journal impact factors, by disease area.

As the Dutch scientific output typically exceeds international standards in the aforementioned disease areas, the CTMM's relative output versus international standards were compared to the Dutch relative output versus the international standards. In line with a previous assessment of the Dutch scientific output versus international output, this comparison regards the combined volume and impact of journal publications (i.e., number of publications in specific journals multiplied by the impact factor of those journals). The international standard reference value is set at 100, and a value above that indicates larger output relative to the international standard. Figure 4 shows that the scientific output of CTMM is on par with the level of Dutch biomedical research output, with scientific output 15%–79% above the international reference value.

OPEN IN VIEWER

FIG. 4.

Scientific output of CTMM and Dutch biomedical research compared to International Reference Value = 100

The survey respondents perceive CTMM's impact on the volume and excellence of scientific output as high, each with a median of 4 (range: 3–4), and its impact on publishing in high impact journals as moderate (median = 3; range: 3–4). Notably, CTMM has established a strong network structure between public and private parties involved (median = 4; range: 3–4), indicated by all respondents answering affirmative to this statement. This is further illustrated by a quote from the group interviews, which was shared among the participants, that CTMM “has managed to bring together parties that otherwise would not have found effective ways to collaborate.”

Translational and clinical impact

Eleven of the 21 disease-oriented projects had progressed from pre-clinical development in 2008 to clinical testing in 2012, and another three projects were scheduled to start clinical trials in 2013. The other projects were in the phases of employing laboratory or animal experiments to develop their technologies further.

Survey respondents indicate that they perceive the impact of CTMM on accelerating translational research in general as high (median = 4; range: 3–4). The impact of CTMM on the development of biomarkers and imaging specifically is perceived by respondents as moderate to high, with somewhat larger variation between disease areas for biomarker development compared to imaging (Fig. 5a, b). In addition, CTMM is perceived to be a strong stimulus for translating new technologies to the clinic with a median of 4 (range: 3–5) across all disease areas (Fig. 5c).

OPEN IN VIEWER

FIG. 5.

(a) Perceived impact of CTMM on biomarker development by disease area; (b) Perceived impact of CTMM on imaging development by disease area; (c) Extent to which CTMM stimulates translation of new technologies to the clinic, by disease area.

By developing technologies that span more than one phase in the care continuum from prevention to treatment monitoring, CTMM aims to maximize clinical impact while exposing SME partners to a wider range of clinical environments. The majority of the CTMM projects (n = 16 of 22) develop technologies that are applicable to ≥2 of four different phases in the health care process, defined as:

• screening and prevention (n = 14): detection and assessment of risk factors;

• early diagnosis (n = 14): in vitro and in vivo detection and assessment of early disease biomarkers;

• patient stratification (n = 15): selecting the most effective therapy for the individual patient (prognostic and predictive medicine);

• image-guided treatment (n = 8): targeted drug delivery, minimally-invasive interventions, and treatment monitoring.

Economic impact

In order to guide the development of new technologies from early development stages towards solutions that will be effective as well as cost-effective in clinical practice, CTMM facilitated early health economic modelling as part of every project. Until 2012, 21 decision analytic models as well as three disease models (simulating the natural course of disease and the (likely) impact of new technology on this) have been developed.

Experience with and impact of early modeling on R&D decision-making varied widely between projects (median = 3; range: 2–4). In the project with the highest scores regarding the influence of early modeling, 50% of respondents indicated that early modeling supported R&D decision making and that, as a result of the model outcomes, changes were made in the further development of the technology (i.e., as a risk stratification instead of a screening tool) (De Graaf et al., 2012). In the project with the lowest score regarding influence of early modeling, 15% of respondents indicated impact on R&D decision making and no changes were made to technology development. Results of the group interviews indicated that, especially in the very early development stages when no concrete technology had formed yet, the integration of the health economic analyses in the projects was experienced as difficult.

One of CTMM's objectives was to generate employment growth of 300 FTE in year two. This was realized in 2010 with a capacity of 328 FTE. After that CTMM continued to grow to nearly 1900 FTE of translational R&D capacity in 2014, most of which were in academia (Fig. 6). Employee distribution by job type is 34% PhD students, 29% scientific/technical staff, 21% postdocs, and 17% senior scientists. Eleven postgraduate students achieved their PhD degree in 2012. While eleven postgraduate students achieved their PhD degree in 2012, another 246 PhD candidates are on track to graduate in 2013 to 2015, resulting in a total of 258 PhDs.

OPEN IN VIEWER

FIG. 6.

Development of Academic and Industry Full Time Equivalents (FTEs) from 2008–2014.

CTMM staff included 33 nationalities in 2012. The majority of non-Dutch staff came from Europe (Western Europe 18%, Eastern Europe 20%, and Southern-Europe 20%), followed by Asia (25%), North America (9%), Northern Europe (6%), and South America (1%).

By the end of 2012, twelve patents had been filed and four spin-off companies were founded by the CTMM members.

Discussion

Translational research moves in a bidirectional manner from one type of research to another—from basic research to patient-oriented research, to population-based research, and back—and involves collaboration between multiple public and private parties, all with their own agendas. Successfully managing a public–private partnership in translational research is a huge challenge as one has to find a way to incentivize participants, primarily for sharing expertise, ideas, data etc., and push aside the typical “publish first” or otherwise protectionist attitude that is all too common in both academia and industry. The approach to assessing the impact of translational research partnership must therefore also be flexible enough to accommodate the assessment needs of public and private partners within the consortia, but it must also be rigorous enough to document that the program is meeting its short-, intermediate-, and long-term objectives.

Against this background, an exploratory assessment of the scientific, translational, clinical, and economic impact of CTMM, as achieved 5 years after becoming operational in 2008, is described in this report. To the best of our knowledge, this is one of the few published articles across the innovation analysis literature describing the impact of a public–private partnership using the program's administrative data and additional survey information.

The results show that CTMM's scientific impact is as high as the overall impact of Dutch biomedical research (i.e., 15%–80% above international standards in terms of volume and journal impact factors). This is an important finding in light of the view held by some that translational research, particularly when carried out in public–private consortia, would be of lower quality than purely academic-driven research. In order to assess the generalizability of this finding, it is important to realize that the scientific impact of CTMM is largely driven by the quality of the academic groups involved. As such, scientific impact of a PPP is likely to be as good as the relative quality of the scientists it can attract.

Regarding translational impact, it became apparent that the majority of CTMM projects moved from bench to clinical trials within the 5-year evaluation timeframe. Notably, participants perceive that CTMM has contributed importantly to accelerating this process. While no comparison could be made with specific other PPPs, a median translation lag of 24 years reportedly exists between first description of the discovery and the earliest highly cited article (Contopoulous-Ioannidis et al., 2008). Others stated that “it takes an average of 17 years for research evidence to reach clinical practice at a rate of 50% use in the relevant population” (Balas and Boren, 2008). Compared to this, it seems that partnerships like CTMM can speed up at least the “bench to bedside”, or T1 phase of this process as described by Westfall (Westfall et al., 2007), before moving the research to practice (phase T2).

The (future) clinical impact of CTMM is large as its projects focus on disease areas with high human and economic burden. Furthermore, CTMM emphasizes the development of medical technologies that are applicable to more than one phase of the health care continuum, in order to maximize patient value and increase the probability for return on investment. To what extent the expected value of these technologies will be harvested in practice strongly depends on whether technology development and implementation continue beyond CTMM's finalization in 2015.

A particular strength of CTMM is that it has aimed to drive collaboration in research methodologies across the development cycle, notably by facilitating early health economic modeling as part of the research projects. This was done by assigning a team of health economists to each project, with the aim to explore and project potential cost-effectiveness of the technologies as an integral part of the technology development process. While the impact of such early health economic modeling on the R&D decision making has been lower than was hoped for at the start, a few champion projects (De Graaf et al., 2012; Handles et al., 2012) illustrate the potential added value of this approach.

One specific aspect to consider in order to increase the usability of early cycle economic evaluation is the choice of metrics to assess. Whereas health economists typically advocate the use of the Quality Adjusted Life Year as their primary outcome measure—as this metric can be compared across diseases and is commonly applied at later stages—other metrics may be more informative in the early stages, such as false positive and false negative rates in case of diagnostics. We suggest that future work in this area should be “strong on principles and soft on metrics,” (i.e., the metrics that resonate most clearly with the various decision makers should be used; Steuten and Ramsey, 2014).

Another barrier may have been that health economic and technology development activities ran ‘out of sync,’ for example, when modeling activities would be outpaced by rapid technology iterations or when “classical” health economic modeling was considered too data intense. Other health technology assessment approaches, including, for example, cost-effectiveness gap analyses, multi-criteria decision analyses (Ijzerman and Steuten, 2011), or minimal modeling (Meltzer et al., 2011), may be solutions in this regard.

Beyond the infrastructure created by the CTMM to integrate health economic modeling into medical technology development, there may also be a greater role for Technology Transfer Organizations to supporting early stage economic evaluations. University TTOs generally aim to facilitate, enhance, and implement the transfer of knowledge and technology created and developed within the institution towards economic value creation, by translating the results of scientific activity into formats readily adoptable and absorbable by industry and commerce. For achieving this, they deploy activities in three main “business” areas: (1) collaborative research with companies and provision of consultancy services; (2) patenting and licensing; and (3) nurturing and developing spin-off companies (Debackere, 2012). Early health economic modelling, however, is a relatively new strand of health economics that may not be commonly known among all TTOs, let alone be routinely included in TTO activities. Education about early economic modeling, and providing or connecting to consultancy services in this area, should be intensified under activity area (1) mentioned above, in order to accelerate high value new medical technologies to the market.

The economic impact of CTMM so far is mainly indicated by the nearly 1900 FTEs generated since 2008. Besides the high number of FTE's, CTMM has created a pool of young international professionals that are well-trained in multidisciplinary team-based approaches to translational research across institutions, which is an asset for the future of biomedical R&D. The long-term impact of CTMM on health care productivity, health care costs, and wider socio-economic impact has not been quantified in this retrospective assessment, but may be part of ongoing research.

As any research, this retrospective assessment has its weaknesses, such as the low response rate to the survey. Proven effective design strategies that were applied to attain good response included pilot-testing of the length of the questionnaire, personalized e-mail invitations, and showing respondents the percentage progress towards finalization of the questionnaire. Furthermore, two reminders were sent after 2 weeks and after 1 month. Future studies should budget for incentive-based strategies, as even small monetary incentives are very effective in improving response (VanGeest et al., 2007).

Also, while the production of a scientific research publication remains a good indication of research progression and knowledge generation (Allen et al., 2009), a full citation analysis would have shed more light on the contribution that individual articles made to further research (Jones et al., 2012).

More importantly, perhaps, it became apparent that the structured assessment to link the inputs and outputs of CTMM, to quantify its measurable impact and to draw lessons for improvement, was not always as informative as expected and findings were sometimes met with resistance. This is not unique to the CTMM impact assessment and similar experiences have been reported previously (Kaplan and Garrett, 2005), for example, after the assessment of the Institute for Translational Sciences, one of the CTSA sites (Scott et al., 2014). By using indicators alone, such as publications, FTEs, and even impact as perceived by participants, the current assessment frameworks do not provide the necessary depth of understanding how well a consortium such as CTMM has nurtured and supported multidisciplinary team science and translational research. In short, leadership would have appreciated more illumination of the “value added” by CTMM and sought more actionable information regarding the relevance and adequacy of their approach.

Nevertheless, the operationally-defined process indicators approach was considered useful and pragmatic, in lieu of observing longer term human, economic, and societal impact of this PPP, which would be much more resource-intense. For future evaluations, however, it is recommended to design and incorporate an impact evaluation plan from the start, identifying the most appropriate process indicators beforehand and collecting the data accordingly, instead of commissioning a somewhat ad hoc evaluation. This may also help to engage the PPP's participants more positively in the evaluation, and to achieve a deeper understanding of the critical success factors for maximizing impact. Recently, a combination of the Kellogg Logic Model and the World Health Organisation's Health Services Program Evaluation Model was proposed for this purpose, and was shown to have the desired results in the evaluation of NIH's CTSA program (Scott et al., 2014).

Conclusion

The retrospective assessment of the impact of CTMM indicates that it succeeded in bringing together parties that otherwise would not have found effective ways to collaborate and share ideas, expertise, and data. Five years after becoming operational in 2008, the CTMM has demonstrated a solid scientific impact that is on par with the overall high-quality of Dutch biomedical research. By moving more than 50% of technologies in development from bench to clinical trials in 5 years, it likely accelerated translational research. Early health economic modeling along the technology development cycle has supported R&D decision-making in some projects and provided insight in the likely cost-effectiveness of specific technologies. Overall, the (future) clinical impact of CTMM is generally perceived as large, according to academic and industrial partners alike, but could not be quantified sufficiently yet.

For future evaluations of PPPs or other institutional and organizational innovations conceived to move discovery science rapidly and effectively to clinical and societal applications, it is recommended to design and incorporate an impact evaluation plan from the start, and engage the partnership's participants in the evaluation, to achieve a deeper understanding of the critical success factors for maximizing human, economic, and societal impacts.