Medical technology innovation and the importance of comparative effectiveness research

Abstract

Recent attention has focused on comparative effectiveness research (CER) to determine the clinical effectiveness and safety as well as the economic impact of health care interventions, including health care reform legislation that provides for new funding to develop the infrastructure to support CER. The demand for CER will significantly impact the competitive landscape for many sectors of the health care industry. The development of convincing and credible evidence to support products and services has become a critical competitive issue. In some cases, it has become a requirement for new health technologies to gain market access, achieve and sustain premium pricing, and achieve targeted levels of market acceptance. This article is intended for senior business management in the health care industry to facilitate efforts to make strategic decisions about the level and type of clinical and economic evidence needed to achieve market success in this new landscape. This article focuses primarily on the strategy for medical technology in the prism of CER.

Keywords

Comparative effectiveness research cost-effectiveness clinical marketing research methods

Rationale for comparative effectiveness research

In 2007, the Congressional Budget Office (CBO) issued a report addressing the effectiveness of medical treatments.¹ Their report focused on the burden of escalating health care expenditures for government and private payers as well as disparities in health care spending within the US compared with other countries. Greater expenditures were not associated with improved health outcomes¹ The CBO suggested US medical practitioners adopted more costly interventions in the absence of rigorous efficacy data. The CBO concluded that better information on costs, risks, and benefits of different therapies was essential in order to change the practice patterns of health care delivery in the US and decrease health care costs.¹

The CBO analysis was followed by two reports by the Institute of Medicine (IOM) in 2008² and 2009.³ Similar conclusions were made by the IOM, including that the US has the most expensive health care worldwide with expenditure not correlating with improved outcomes, and a notable wide regional variation in patterns of clinical care.^2,3 The IOM report also recommended a priority list of 100 research areas where comparative evidence for diagnosing, treating, or monitoring specified medical conditions were lacking. This list will most likely serve as a foundation for future focus and funding efforts by US Government and private payer organizations.

Definitions of comparative effectiveness research

The call to action by the IOM established the momentum for comparative effectiveness research (CER) in the US health care reform efforts. The diversity in definitions of CER is attributable, in part, to variations in the perspectives of multiple stakeholders.⁴ In Europe, CER is often associated with evidence developed in the context of formal health technology assessment (HTA), which may include an explicit consideration of cost-effectiveness.⁵ In the US, the construct of HTA was formally established by the Office of Technology Assessment in 1978 and defined as ‘a multidisciplinary field of policy analysis, studying the medical, economic, social, and ethical implications of the development, diffusion, and use of health technology.’^6,7 According to this definition, technology includes medications, devices, procedures, and systems of organization for the delivery of health care.⁷ The process of HTA is intended to evaluate effectiveness, appropriateness, and cost in order to provide decision makers with information to inform policy options.

The global perspective on HTA varies significantly due to differences in culture, history, politics, and methods of financing. Canada, Europe, and other parts of the world have established agencies charged with responsibility for HTA activities, including evaluations of the cost-effectiveness of specific technologies. The United Kingdom’s National Institute for Health and Clinical Excellence is the most mature example of a centralized HTA body that provides guidance, sets quality standards, and manages a national database to improve health.⁸ Other national bodies have multiple agencies that reflect various regional or bureaucratic divisions with different responsibilities that fall into the domain of HTA. For example, in Germany, the German Institute of Medical Documentation and Information (DIMDI) establishes research priorities, (HTA at DIMDI)⁹ while the Institute for Quality and Efficiency in Health Care (IQWiG) conducts technology assessments and the Federal Joint Decision Making Body (G-BA) makes coverage and reimbursement decisions and establishes treatment guidelines.

Use of the term HTA is less common in the US where there is a greater emphasis on CER as well as evidence-based medicine (EBM), outcomes research, and effectiveness research.⁷

Prominent organizations in the US including the Agency for Healthcare Research and Quality, the IOM Committee on CER Priorities, and the Federal Coordinating Council for Comparative Effectiveness Research (FCC-CER) have established definitions for CER.^3,10,11 The IOM defines CER as:

‘the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition, or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.’^3,7

Despite the differences in definitions for HTA, EBM, and CER, they share a common emphasis on research that yields clinical and economic evidence of the comparative benefits, risks, and values of alternative therapeutic interventions. These results are intended to provide information to clinicians, purchasers, policy-makers, patients, and other stakeholders that will improve the quality and safety of health care at the individual and population level.¹² Such information is considered essential to translate new research findings into clinical practice, improve patient outcomes, speed the provision of health care interventions with proven benefit, control costs, and deliver ‘the right treatment to the right patient at the right time.’^13,14 However, practitioners should be mindful that even within HTA assessments there are differences in assessments between medical devices and pharmaceuticals (incremental technology gains, inability to blind in certain surgical scenarios, etc.).¹⁵

The development of a CER program for the US

The American Recovery and Reinvestment Act of 2009 (ARRA) was a significant federal commitment that allocated $1.1 billion to support CER and established the FCC-CER to complement ongoing research conducted in the public and private sectors through the support and development of data and research methods, provision of training for researchers, and timely dissemination of results to health care provides, patients, and other stakeholders.¹¹

Following this investment, the Patient Protection and Affordable Care Act of 2010 established CER as the central feature of efforts undertaken by federal agencies to improve health care with oversight of federal CER activities assigned to the Patient-Centered Outcomes Research Institute (PCORI).¹⁶ PCORI will be funded going forward from the new health care legislation, with estimates as much as $400–500 million per year.

Stakeholders affected by CER

The 2008 IOM report noted that multiple individuals/entities contribute to, and are affected by, health care decision-making. These include:

Health care providers making decisions about therapeutic interventions for individual patients.

Consumers selecting health plans.

Patients or their caregivers making treatment choices.

Payers and employers making decisions about the type and scope of services to be covered and reimbursed.

Professional medical organizations developing clinical practice guidelines.

Regulatory agencies responsible for the assessment of new drugs or devices, and public health programs developing population-based interventions.

Decisions made by these various stakeholders will be based upon knowledge that is explicit, systematic, and replicable with an emphasis on determination of the type of services that are most effective in real-world settings with real-world patients. Such patients will include those with comorbid health conditions, diverse risk factors, variations in adherence rates to therapeutic interventions, stage and severity of disease, variations in health care coverage, and demographic characteristics. In addition, factors related to the health care setting, level of training of health care providers, and the timing and quality of interventions can have an impact on effectiveness.

Implications of CER for medical technology companies

The emphasis on CER and the increased demand for evidence to improve health care decision-making, clinical outcomes, and the value of health care interventions has created new challenges for the health care industry. Business lines now face critical decisions regarding the amount and timing of investments in CER initiatives. Depending on the specific type of products or services offered, individual companies will be challenged to balance the evidence needs established by regulatory, reimbursement, and market stakeholders in diverse therapeutic areas, clinical settings, geographic regions, and patient populations. Medical technology companies are now required to determine market success and assess the value of evidence-based claims. This will require trade-offs between minimizing cost and development cycle time while maximizing the market value that comes from incremental investments in evidence development (i.e. disruptive, highly utilized costly technology will require more evidence than a replication of a technology already on the market).

Furthermore, competing potential investments in evidence development must be prioritized. For example, medical technology companies will need to weigh the potential gains and losses associated with more costly, long-term, large studies that are expected to have a large impact compared with multiple, smaller studies that can be conducted at lower cost, in less time, with fewer subjects, and at potentially lower risk to patients and other customers. Evidence development alternatives (observational studies, claims mining, retrospective studies, etc.) that are lower in cost do exist and may address the same needs as the “gold standard” of the randomized controlled trial (RCT) in some circumstances (Methods and data sources for CER section).

In addition, medical technology companies must develop the expertise to anticipate and respond to information from clinical and economic studies, including comparative effectiveness (CE) studies that are publicly funded or sponsored by competitors. Such responses will have to address the interpretation of results that may negatively affect a specific company’s initiatives for utilization, reimbursement, or evidence development. The optimal strategy may include trade-offs between ‘defensive’ and ‘offensive’ investments in CER. Specifically, defensive investments may address evidence gaps in response to other published studies, while offensive strategies will rely on collection and dissemination of evidence from head-to-head comparisons to improve positioning of specific products or services.

Furthermore, large companies may face the potential for strategic conflicts across business units. For example, CER will require studies that evaluate treatments, products, or services targeted to specific patient sub-populations to identify the right intervention for the right patient at the right time. This focus will have implications for alternative interventions developed by different business units within the same company. In addition, comparative studies based on head-to-head comparisons of treatments or products are likely to be conducted by external parties, which will require each company to determine their level of support for, and involvement in, such studies and determine the best response to findings from such studies, particularly if the results are not favorable to the company’s product or service.

Strategic decisions about CER studies must also take into consideration regional and global variations in evidence needs and priorities that will be determined by diverse factors such as the differential epidemiologic burden of specific health conditions, access to health care services, and variations in the demographic and clinical characteristics of patient populations. While some core evidence will have value in all markets, allocation of research budgets will have to take into account regional differences in evidence needs, standards, and opportunities within the context of a global business milieu.

Additional expertise, changes in standard processes used by industry for product and business planning, and development of new external relationships will also be needed to support the development of the strategic evidence required by CER. Of particular importance will be expertise to plan and execute studies, as well as disseminate findings to address the evidence needs of clinicians, regulatory agencies, payers, patients, and other stakeholders. Effective dissemination of evidence-based results will support regulatory approval, support payer coverage, increase market access, and promote clinical adoption. Process and organizational capabilities will be essential to efficiently coordinate these efforts. In addition, there will be an imperative to build and sustain external relationships with investigators and other opinion leaders to ensure that investments in evidence development have maximum impact. There has never been a time when it will be more important to collaboratively and effectively work across business units toward common goals in evidence development that will benefit each company as a whole. A summary of potential strategies to achieve this type of collaborative initiative are briefly described below.

First, it is essential to realize that evidence strategy is an integral part of the product creation process. Importantly, evidence strategy should be initiated early in the product creation process, beginning with an evaluation of the evidence needed to make specific product claims (clinical, mechanical, and economic). The development of evidence for new claims close to product launch is not always possible or desirable, especially when complex, time-consuming studies are needed. Therefore, early planning efforts should identify the optimal study design to respond to specific evidence claims. Claims that must be based on complex studies enrolling large number of patients and following these patients for long periods of time should be identified and initiated early in the process of product development. This may also include evidence mapping, which would include review of various geographical needs, or if studies could be completed in one geography to support the needs of a different geography (example: CE marking a technology in Europe and initiating a study for strategic reasons while a US trial is being planned). Furthermore, a particular evidence strategy may be associated with a different investment profile and different product development cycle time and cost metrics. Thus, early consideration and proactive decision-making about the various types of evidence that is desired and needed to make specific product claims are the key elements of the evidence strategy.

Finally, evidence strategy should encompass all claims and value propositions needed for regulatory approval, favorable reimbursement and pricing, and market uptake. The evidence needs (value proposition) of various stakeholders and the claims desired to address these needs should be considered together. It is no longer sufficient to obtain regulatory approval and clearance first and then address claims support following approval. Consolidated evidence needs should be addressed in the context of benefits for patients, caregivers, and customers, including clinical outcomes, patient preferences, care continuum, clinical setting and workflow, and cost avoidance or improvement. The business case for evidence development should be explicit and priorities established for competing evidence needs and responses. This will include decisions regarding expected study outcomes and consideration of return on the resource investment (ROI). The business case should consider the full range of options for clinical and economic studies.

It is also likely that additional organizational capabilities may be needed to respond more effectively to demands for evidence. Of particular importance will be expertise in areas related to the design and conduct of robust studies, as well as dissemination of evidence. This includes, but is not limited to, biostatistical knowledge, Health Economics and Reimbursement (HER), publication planning, trial design, medical writing, scientific communications, and medical informatics. Strong collaborations between marketing, regulatory, HER, and clinical personnel will optimize the ability of an organization to launch a proactive response for evidence development. Internal organizational strengths and capabilities will be further enhanced by ongoing external relationships such as those with academic medical centers, key opinion leaders, professional societies, and independent investigators. Other partnerships that may enhance an organization’s ability to respond to the demand for evidence can include other stakeholders who share an interest in demonstrating improved clinical and economic outcomes, such as payers, professional organizations, patient advocacy groups, or other industry participants.

To summarize, strategic evidence development begins with a coordinated assessment of evidence needs early in the product development process, including a comprehensive assessment of requirements for regulatory and reimbursement approval as well as evidence required to promote market uptake/utilization and clinical adoption. An assessment of evidence needs should be grounded in the potential claims needed for the success of products, services, and solutions and the importance of these claims should be validated with business members who are functionally linked to each stakeholder. For example, claims necessary to meet the evidence requirements of regulatory and reimbursement authorities should be developed with business members associated with these functions and coordinated with similar claims for other stakeholders. Identification of claims necessary for market success is particularly important early in product development when large or complex clinical studies may be required or advantageous. The importance of these claims and the supporting evidence needed should be re-evaluated as the product advances in development, with revisions made based either on changes to product characteristics, the market landscape, or stakeholder evidence needs. Table 1 provides a guideline for assessment of evidence needs and identification of potential claims.

Table 1.

Guidelines for assessment of evidence needs

Stakeholder	Potential evidence planning objectives	Potential claims based on evidence
Regulatory agencies	Evidence to support applications to receive authorization for specific markets or indications	• Effectiveness
		• Efficacy
		• Safety
		• Other patient outcomes (e.g. quality of life)
Payers	• Clinical or economic evidence to support market access, favorable reimbursement, and premium pricing in target markets	• Product is reasonable and necessary for the diagnosis, monitoring, or treatment of a condition
		• Clinical and economic evidence support reimbursement
		• Product is cost-effective
		• Product has a modest or net positive impact on payers’ budgets
Providers	• Clinical evidence of the efficacy, effectiveness, and safety for the treatment of patients	• Evidence of greater safety or effectiveness compared with alternatives
Providers		• Product is appropriate for defined patient sub-groups
Hospital administrators	• Clinical or economic evidence to support purchasing decisions	• Current evidence or treatment guidelines support use of the product
		• Favorable impact on the care pathway
		• Cost savings or modest impact on budget
Patients	• Clinical evidence of effectiveness	• Improved health or other outcomes of relevance to patients
	• Favorable side-effect or tolerability profile
	• Improvements in clinical outcomes	• Product superiority to alternatives regarding effectiveness and safety
		• Cost-effectiveness
		• Easy to use

Strategic framework for industry investment in CER

Despite numerous challenges, the demand for CER and demonstrations of the strategic value of evidence present positive business opportunities, particularly for organizations that anticipate evidence requirements and develop a proactive response plan. Substantial efficiencies in product development and research cycle time and costs can emerge from long-range, systematic planning. For example, evidence desired by multiple stakeholders can be collected in a single, well-designed study that generates both clinical and economic outcomes data.

Industry leadership must assess evidence needs in order to determine the business imperative and rationale for studies to generate evidence that responds to the informational needs of diverse stakeholders. Lines of business face critical decisions about the timing and amount to invest in studies to generate evidence of various types. This section briefly reviews concepts to be considered in the process of decision-making about allocation of resources to support decisions about research needs. A strategic framework to support decisions for the investment in evidence development studies is presented in Figure 1.

Figure 1.

Strategic planning framework.

There are two key components of a business case for evidence development including (1) economic value of evidence and (2) level of investment required to generate evidence. The economic value of evidence depends on the impact of potential new claims, the expected financial benefit of establishing those claims, and the attractiveness of the market opportunity. Factors that contribute to decisions regarding the level of investment required to generate evidence include the evidence needs of key stakeholders and the type and cost of resources required to develop and disseminate evidence-based findings.

First, the business must determine the attractiveness of the market opportunity associated with new claims to be supported by evidence. Second, the business must estimate the cost of the studies required to develop the evidence needed to support these new claims. Such estimates should take into consideration both direct and indirect costs. Indirect costs include those associated with delayed market entry while the research is conducted or emergence of new evidence about competitive products or services. Direct components of costs will include elements of study design, conduct, management, analysis, and dissemination of findings. Evidence for multiple claims can be obtained from a single study while evidence for certain critical claims may require a series of multiple studies or modeling efforts. Key questions to be considered by business leaders to assess these two components for a given business case are presented in Table 2.

Table 2.

Key questions to assess the business case for evidence development

Key questions	Comments/analysis
1 What will be the impact of new evidence-based claims?	• Consider if new technology has the potential to significantly improve clinical practice and patient outcomes
2 Which claims will have greatest impact/value?
• Clinical outcomes
• Workflow and efficiency
• Cost
3 What is the expected financial benefit of new claims?	• Calibrate investments in evidence development with market size and anticipated position
4 What is the attractiveness of the market opportunity?
• Impact on market share
• Potential to expand the market
• Expected profit margin and potential for premium pricing
• Potential to establish and sustain a leading position
5 What are the evidence needs of key stakeholders to establish claims?	• Consider if diverse stakeholders share common evidence needs to meet their goals and requirements
• Regulatory agencies (FDA, EMEA)
• Public and private payers
• Health care providers
• Medical facility administrators
• Patients/consumers
6 What resources will be required to develop and disseminate the evidence?	• Consider the direct costs of research, internal resources, key external relationships, approaches for dissemination of findings
• Regulatory approval (510k or PMA findings)
• Reimbursement (HTA submissions, payer reviews, formulary approval)	• New technologies or product or significant changes in existing technologies/products may require greater initial investment while yielding potentially greater long-term returns
• Market uptake/utilization

Positioning on the strategic planning framework reveals different clinical and market contexts that suggest varying evidence development strategies and investment options (Figure 2). In general, business cases located in the upper two quadrants of the framework (high economic value of evidence) support more attractive investment opportunities compared with those located in the two lower quadrants (low economic value of evidence). Specifically, location in the upper left quadrant implies a higher potential ROI from a targeted evidence development strategy that captures high value while not incurring high or unnecessary investment of resources. Investments should be targeted to development of evidence that addresses the needs of key stakeholders as well as coordinated with pricing and marketing strategies to achieve maximum impact. Additional investments in evidence development may also be required to ensure that the long-term strategic advantage is sustained. Studies that fall into the upper right quadrant require a high level of investment but this is potentially offset by the potential for high yield in terms of evidence-based claims. A higher premium on planning and coordination is essential for studies that fall into this quadrant. Thus, business leaders must develop a comprehensive plan that generates new evidence to demonstrate clinical effectiveness and cost-effectiveness for well-defined patient populations.

Figure 2.

Range of potential evidence development strategies and investment options.

Evidence development strategies located in the lower left quadrant imply minimal investments, to meet the requirements for market success, which typically is focused on authorization by regulatory agencies. The lower right quadrant also implies a focus on regulatory approval, but also that the substantial investment in evidence required should be repurposed wherever possible. Due to the highest costs associated with evidence development in this quadrant, changes in market dynamics should be closely monitored. Furthermore, business teams should avoid investment in studies that address stakeholder evidence needs but have an unacceptable expected ROI.

Methods and data sources for CER

Ultimately, CER is expected to have significant implications in three primary areas. First, findings will better inform the current process of health care decision-making by diverse stakeholders. Second, the investment in translation of CER results into clinical practice has the potential to improve both the quality and value of health care for all populations in diverse health care settings. Third, CER is considered essential for the support of high-value innovation and the practice of personalized medicine.^11,13,16 Distinguishing features of CER include comparisons of alternatives that are most relevant to real-world clinical practice settings. This means that CER studies should represent the treatment(s) typically offered to patients in typical settings and should not include comparisons with placebo unless no intervention is an acceptable option in current clinical settings. In addition, patients included in CER should be representative of those most likely to receive the intervention in real-world clinical settings; this is a major difference from trials currently conducted to obtain approval of new products or devices by the Food and Drug Administration and other regulatory bodies.^11,13,16 In addition, CER should identify patient-specific characteristics that are associated with differential responses to therapeutic interventions. These might include age, gender, comorbidities, severity of illness, genetic history, and other demographic or clinical characteristics that might affect the outcome associated with a diagnostic test or therapeutic intervention. Further, CER should include all outcomes that are important to patients such as the frequency and severity of side effects, tolerability, preferences for treatment regimens, and quality of life. When appropriately conducted, CER has the potential to yield information about cost-effectiveness of interventions by providing high-quality comparative outcome data.¹⁶

RCTs have long been considered the gold standard to evaluate the efficacy of a product compared with placebo. However, the results from RCTs may not be representative of the effectiveness of therapies delivered in real-world clinical settings to real-world patient populations. Furthermore, RCTs that compare new products with existing therapies are rare.^17,18 CE studies include assessments of health outcomes attributed to alternative medical treatments or services that are of particular interest to payers, providers, patients, and other stakeholders. Typically, CER studies are designed to answer questions about treatment in ‘real-world’ medical settings. In contrast, clinical trials conducted for regulatory approval frequently do not incorporate comparisons with therapeutic alternatives, known as active comparators. In addition, such trials typically have exclusion criteria that limit the study population in order to minimize variations in response to treatment. These criteria increase internal validity while decreasing external one, as patients studied in clinical trials may differ significantly from the ‘real-world’ patient populations.

In addition, CER studies are expected to provide information about the relative effectiveness of different interventions, such as pharmacologic treatments versus surgical procedures or competing behavioral, care delivery, or policy interventions.¹⁹ For device and diagnostic manufacturers, the distinction between comparative studies and more traditional study designs can be critical because effectiveness is often a function of extrinsic factors, such as how the device or diagnostic is used. Understanding the challenges and limitations of CE studies is necessary for the development of an effective evidence strategy and to support evaluation of the results of studies conducted by competitors and other health service researchers.

Alternative study designs are available to respond to the demands for evidence required for regulatory improvement, reimbursement, or market analysis. The FCC-CER suggests a number of alternative study designs to achieve the objectives of CER (Table 3).¹¹ Each of these study types serve to accomplish different ends. Systematic reviews typically rely on published information to make comparisons between different medical interventions for diverse population administered in a variety of clinical settings. Systematic reviews synthesize existing published evidence in order to give a more comprehensive picture of the product in relation to others.²⁰ Decision analyses rely on existing evidence to simulate and compare the benefits, risks, and costs of alternative treatment pathways as well as describe overall costs, benefits, and risk of one product versus other options. Observational data analysis, on the other hand, generates new evidence of CE based on data collected in clinical practice for treatment comparisons. Finally, CE-focused RCTs expand traditional RCT methods to better reflect real-world conditions. This may include, but is not limited to, multiple comparator arms other than placebo (i.e. head-to-head comparisons) and inclusion of patients who typically might be excluded from a traditional RCT, such as older patients or those with comorbid health conditions.

Table 3.

Forms of comparative effectiveness research defined by the federal coordinating council for comparative effectiveness research

Format	Description	Examples
Systematic review	• Comprehensive analysis of published studies to compare diagnosis or treatment options	• Evaluation of transcatheter closure versus medical intervention in patent foramen ovale that was based on a comparative analysis of data presented in relevant literature listed in Medline database from 1985 to 2003²¹
	• Varying levels of quality between systematic reviews
	• Rigorous reviews most influential (UK NHS Centre for Reviews)
Decision analysis	• Simulation using published evidence to compare the benefits, risks, and costs of alternative pathways, describing overall costs and benefits of one product versus comparators	• Decision modeling for colorectal cancer screening that provided a cost-effectiveness evaluation of competing diagnostic options²²
Decision analysis		• Decision modeling for carotid artery imaging that provided a cost-effectiveness evaluation of diagnostic procedures for carotid artery stenosis²³
Observational data analysis	• Analysis of data collected in clinical practice that yields comparisons of treatments and generates new evidence on CE	• Evaluation of assistive technology devices for veterans following stroke that included an assessment of costs and variations in devices provided post-stroke based on a retrospective analysis of VA databases²⁴
CE-focused RCT	• Trial using RCT methods enhanced by design modifications to better reflect real-world conditions with a focus on comparisons of multiple treatment arms	• COURAGE trial to compare rates of death and cardiac events in patients with angina randomized to percutaneous coronary intervention or medical intervention²⁶
		• CHOICE trial to evaluate the effect of two evidence-based methods of informing patients and physicians of the benefits and risks of colorectal cancer screening²⁷
	• RCTs typically provide the highest level of evidence and strongest inferences regarding causality
	• Guidelines are available for grading the quality of RCT²⁵

The study types described in Table 3 are likely to have very different costs and time required for completion. The choice of study type should reflect careful consideration of the level of investment required and the value of evidence that is likely to be obtained. In many cases, the best choice that delivers the most value will address a specific evidence need and may not be the most costly. For example, systematic reviews of clinical and economic literature are typically conducted in the context of HTAs. Manufacturers can conduct and publish such reviews to critically assess the published evidence either to demonstrate effectiveness or to highlight an unmet medical need for patients treated with existing alternatives. Such a study is almost certainly less costly than a new decision analysis or observational study and may better highlight comparative evidence based on from multiple separate studies.

Clinical research for regulatory approval typically demonstrates safety and clinical efficacy, but it does not necessarily establish real-world effectiveness or demonstrate value in comparison with other options. Comparative clinical trials, particularly those that evaluate real-world use can address the evidence needs of payers and customers and potentially position a product for market success. The investment required to develop this type of comparative evidence must be balanced against the expected impact on the market. Similarly, economic studies may serve as an important source of information about evidence of value. Prior to product launch, decision analyses may provide information to support decisions about pricing strategy and identify potential evidence gaps. In one example, the Medicare Evidence Development and Coverage Advisory Committee reviewed screening CT colonography for coverage, but found that while the procedure was clinically effective, it would need to be less expensive in order to justify coverage. Additional criticisms included a lack of evidence demonstrating long-term benefit and the potential for risk associated with radiation exposure.²⁸ This case confirms that development of evidence demonstrating cost-effectiveness prior to coverage determination decisions may be a worthwhile investment. Going forward, more evidence will be required to change coverage policies.

Numerous methodological challenges arise from efforts to develop real-world evidence of effectiveness. Potential changes in study methods are a significant concern to pharmaceutical and device manufacturers, given the substantial investments in ongoing and future studies. Recent work has addressed these challenges with a number of methodological innovations, including designs for both interventional and database-oriented studies, data collection methods, statistical and modeling techniques. In addition, a significant component of the funding from the ARRA administered the AHRQ has been applied to improve methods for conducting CER. One particular approach that has received considerable attention is pragmatic or practical clinical trials (PCTs).^29,30 Key features of PCTs are inclusion of clinically relevant alternative interventions, inclusion of a diverse population of study participants, recruitment of participants from heterogeneous practice settings, and data collection that includes a variety of health outcomes.

Additional study designs that may inform CER include outcome studies and cost-of-illness research. Outcomes studies are designed to evaluate the effects of a treatment in terms of patient outcomes, including adverse events, ability to function, and improvements in disease state. Cost of illness research is designed to demonstrate the impact of an illness or treatment for an illness on monetary cost. It has also become accepted practice for investigators to use secondary databases or patient registries, such as those available from state Medicaid programs, state-wide hospital databases, or the Department of Veterans Affairs to evaluate initiation, adherence, continuity of product use, and clinical outcomes regarding effectiveness, safety, and cost, including disparities in these measures associated with specific population groups.³¹ However, the potential for confounding and bias is significant for analyses of administrative or other types of observational data. Strategies to minimize these threats include propensity scoring, inverse probability weighting, instrumental variables, and use of direct or indirect methods to accumulate and synthesize CE information.^31,32

The International Society for Pharmacoeconomics and Outcomes Research Health Science Policy Council has established a task force to recommend state-of-the-art analytic techniques to achieve statistical control of potential sources of confounding for non-randomized studies based on secondary databases.³³ Recommended strategies include stratification analysis before multivariable modeling, multivariable regression including model performance and diagnostic testing, propensity scoring, instrumental variable, and structural modeling techniques. Residual confounding refers to confounding that has not been completely controlled, which increases the likelihood that confounding effects of some factors may alter the observed treatment-outcome effect. It is essential to assess residual confounding and estimate its effects with methods such as sensitivity analyses.³³

Conclusions

Requirements for evidence from multiple stakeholders have changed and continue to be modified. Decisions about the clinical and economic evidence needed for market success will primarily depend on each product’s particular situation, but all businesses will likely be influenced by key trends, including increasing state and national budget constraints. Fiscal crises and growing budget deficits are likely to impose additional pressure on limiting the growth in health care spending, particularly for spending to develop new health technologies. There are also increasing evidence requirements for effectiveness from payers worldwide, particularly for products with a large overall budget impact on payers and for products with less costly substitutes. In the US, concerns about the public health and safety of the 510k Premarket Notification process could result in changes that increase evidence requirements for US regulatory approval, with a significant potential impact on teams developing evidence strategy for products in development. There are also likely to be additional requirements imposed on diagnostic and therapeutic interventions. For example, imaging is increasingly being used not only for diagnosis, but also for planning and conducting medical and surgical therapeutic interventions. Imaging systems used in such a context may be subjected to additional rigorous evidence requirements that are more typical for Class III devices. Furthermore, as clinician hospitals and other medical settings increase their adoption of integrated electronic health records due, in part, to the financial positive and negative incentives imposed by ARRA, the value of integration with those systems will increase. Along with the growing use of clinical decision support systems, this could improve productivity and lower the marginal cost with each additional user. This changing context is likely to affect the cost-effectiveness of every manufacturer’s products and services, and CER is likely to play a fundamental role in measuring these changes.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Conflict of interest

The authors declare there are no conflicts of interest.

Author's Biographies

Karen K. Giuliano, RN, PhD is a Principal Scientist in Outcomes Research and Director of Corporate Business Development at Philips Healthcare.

Michael Ferguson is currently the Director of Health Economics and Comparative Effectiveness Research at Boston Scientific. At the time of manuscript preparation, Michael Ferguson, PhD was the Director of Clinical Outcomes, medical affairs, at Philips Healthcare.

Erica Silfen, MD, is Senior Vice President and Chief Medical Officer for Philips Healthcare.

References

Congressional Budget Office. Research on the Comparative Effectiveness of Medical Treatments. US Congress. Washington, DC: 2007. http://www.cbo.gov/ftpdocs/88xx/doc8891/12-18-ComparativeEffectiveness.pdf (accessed 13 February 2011).

Institute of Medicine. Knowing What Works in Health Care: A Roadmap for the Nation, Washington, DC: National Academies Press, 2008.

Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research, Washington, DC: National Academies Press, 2009.

Eddy

. Health technology assessment and evidence-based medicine: What are we talking about? Value Health 2009; 12(Suppl. 2): S6–S7.

Nielsen

Funch

Kristensen

. Health technology assessment: Research trends and future priorities in Europe. J Health Serv Res Policy 2011; 16(Suppl. 2): 6–15.

(2009) International Network of Agencies for Health Technology Assessment. HTA Resources. http://www.inahta.org/HTA/ (accessed 13 February 2011).

Luce

Drummond

Jonsson

Neumann

Schwartz

Siebert

. EBM, HTA, and CER: Clearing the confusion. Milbank Q 2010; 88(2): 256–276.

(2012) National Institute for Health and Clinical Excellence. About NICE. Who We Are. http://www.nice.org.uk/aboutnice (accessed 30 December 2011).

Deutsches Institut für Medizinische Dokumentation und Information. HTA at DIMDI. http://www.dimdi.de/static/en/hta/basisinfo_en.pdf (accessed 30 December 2011).

10.

(2012) Agency for Health Care Research and Quality. Effective Health Care Program. http://effectivehealthcare.ahrq.gov/.

11.

Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and Congress. Washington, DC: Department of Health and Human Services, 2009. http://www.hhs.gov/recovery/prgrams/cer/cerannualrpt.pdf (accessed 30 December 2011).

12.

Snyder

Clancy

Steinwachs

. Adding the patient perspective to comparative effectiveness research. Health Aff 2010; 29(10): 1863–1871.

13.

Conway

Clancy

. Comparative-effectiveness research—implications of the Federal Coordinating Council’s report. N Engl J Med 2009; 361(4): 328–330.

14.

Sox

Greenfield

. Comparative effectiveness research: A report from the Institute of Medicine. Ann Intern Med 2009; 151(3): 203–205.

15.

Drummond

Griffin

Tarricone

. Economic evaluation for devices and drugs – same or different. Value Health 2009; 12(4): 402–404.

16.

Garber

Sox

. The role of costs in comparative effectiveness research. Health Aff 2010; 29(10): 1805–1811.

17.

Berger

Mamdani

Atkins

Johnson

. Good research practices for comparative effectiveness research: Defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—Part I. Value Health 2009; 12(8): 1044–1052.

18.

Concato

Shah

Horwitz

. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000; 342(25): 1887–1892.

19.

Volpp

Das

. Comparative effectiveness—thinking beyond mediation A versus medication B. N Engl J Med 2009; 361(4): 331–333.

20.

Center for Reviews and Dissemination. Systematic Reviews. CRD’s Guidance for Undertaking Reviews in Health Care. York, United Kingdom: York Publishing Services Ltd, 2009.

21.

Khairy

O’Donnell

Landzberg

. Transcatheter closure versus medical therapy of patent foramen ovale and presumed paradoxical thromboemboli: A systematic review. Ann Intern Med 2003; 139(9): 753–760.

22.

Khandker

Dulski

Kilpatrick

Ellis

Mitchell

Baine

. A decision model and cost-effectiveness analysis of colorectal cancer screening and surveillance guidelines for average-risk adults. Int J Technol Assess Health Care 2000; 16(3): 799–810.

23.

Buskens

Nederkoorn

Buijs-Van der Woude

Mali

Kappelle

Eikelboom

. Imaging of carotid arteries in symptomatic patients: Cost-effectiveness of diagnostic strategies. Radiology 2004; 233(1): 101–112.

24.

Winkler

Vogel

Hoenig

Ripley

Fitzgerald

. Cost, utilization, and policy provision of assistive technology devices to veterans poststroke by Medicare and VA. Med Care 2010; 48(6): 558–562.

25.

Jadad

Moore

Carroll

Jenkinson

Reynolds

Gavaghan

. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Control Clin Trials 1996; 17(1): 1–12.

26.

Boden

O’Rourke

Teo

Hartigan

Maron

Kostuk

. Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med 2007; 356(15): 1503–1516.

27.

Lewis

Pignone

Schild

Scott

Winquist

Rimer

. Effectiveness of a patient- and practice-level colorectal cancers screening intervention in health plan members: Design and baseline findings of the CHOICE trial. Cancer 2010; 116(7): 1664–1673.

28.

Centers for Medicare and Medicaid Services. Decision Memo for Screening Compute Tomography Colonography (CTC) for Colorectal Cancer (CAG-00396N). http://www.cms.hhs.gov/mcd/viewdecisionmemo.asp?id=220 (accessed 2 March 2011).

29.

Luce

Kramer

Goodman

Connor

Tunis

Whicher

. Rethinking randomized clinical trials for comparative effectiveness research: The need for transformational change. Ann Intern Med 2009; 151(3): 206–209.

30.

Tunis

Stryer

Clancy

. Practical clinical trials: Increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003; 290(12): 1624–1632.

31.

Lohr

. Emerging methods in comparative effectiveness and safety: Symposium overview and summary. Med Care 2007; 45(10 Suppl. 2): S5–S8.

32.

Psaty

Siscovick

. Minimizing bias due to confounding by indication in comparative effectiveness research: The importance of restriction. JAMA 2010; 304(8): 897–898.

33.

Johnson

Crown

Martin

Dormuth

Siebert

. Good research practices for comparative effectiveness research: Analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—Part III. Value Health 2009; 12(8): 1062–1073.