Early Stage Health Technology Assessment for Precision Biomarkers in Oral Health and Systems Medicine

Abstract

Health technology assessment (HTA) is a crucial science that influences the responsible and evidence-based transition of new discoveries from laboratory to applications in the clinic and society. HTA has recently moved “upstream” so as to assess technologies from their onset at their discovery, design, or planning phase. Biomarker research is relatively recent in oral health, but growing rapidly with investments made to advance dentistry and oral health and importantly, to build effective bridges between oral health and systems medicine since what happens in oral health affects systems pathophysiology, and vice versa. This article offers a synthesis of the latest trends and approaches in early phase HTA, with a view to near future applications in oral health, systems medicine, and biomarker-guided precision medicine. In brief, this review underscores that demonstrating health outcomes of biomarkers and next-generation diagnostics is particularly challenging because they do not always influence long-term outcomes directly, but rather impact subsequent care processes. Biomarker testing costs are typically less of a barrier to uptake in practice than the biomarker's impact on longer term health outcomes. As a single biomarker or next-generation diagnostic in oral health can inform decisions about numerous downstream diagnosis-treatment combinations, early stage “upstream” HTA is crucial in prioritizing the most valuable diagnostic applications to pursue first. For the vast array of oral health biomarkers currently developed, early HTA is necessary to timely and iteratively assess their comparative effectiveness and anticipate the inevitable questions about value for money from regulators and payers.

Introduction

Finding timely, accurate, cost-effective, and preferably noninvasive diagnosis and monitoring methods are important goals for clinicians and scientists alike. As biomarkers and next-generation diagnostics can contribute significantly to achieving these goals, their (increasing) role and potential value has received ample attention over the last two decades (Hagen, 2012). Biomarkers, defined as “an indicator of a normal biological process, a pathogenic process or a pharmacologic response to a therapeutic intervention” (Pham et al., 2014), represent a wide variety of technologies that are used in various stages of the disease process (Biomarkers Definitions Working Group, 2001). In earlier stages of the disease process, biomarkers are valuable because they allow for a timely diagnosis or staging of a disease, which can meaningfully impact prognosis, choice of therapeutic intervention, patient outcomes, and also health care costs. In later stages, biomarkers may serve to indicate surrogate and clinical endpoints in order to predict clinical benefit from specific therapies and to monitor patients during and after treatment.

There are a host of biomarker candidates (multi-omics, pan-omics, and others) in transition to the clinic, particularly in the field of oral health and systems medicine, as this special issue illustrates. The clinical realization of a next generation diagnostic or biomarker discovery, however, is an arduous journey that can best be characterized as ‘long, costly, and uncertain.’ Biomarkers or biomarker panels discovered in any of the –omics libraries are subjected to comprehensive assessments at various stages of the R&D process, from preclinical validations up to FDA evaluation and approval. Along this journey the challenge is to sort out, from this multitude of candidate biomarkers, those that are most likely to perform well in the real-world complexity and do so in an economically sustainable way.

As previously argued in this journal, the traditional Phase 1 to 4 biomarker development process is in need of strategies for “rapid falsification,” (i.e., rapid removal of biomarker candidates from further development that are unlikely to provide sufficient added value in real-world clinical settings or public health practice) (Sardas et al., 2014). One emerging approach for doing so is called “early stage Health Technology Assessment.” Firmly grounded in health economic and decision analytic theory, this approach iteratively analyzes the prospects of a new biomarker in terms of safety and (cost-) effectiveness comparative to current practice or competitor biomarkers, using the information available at that stage with the aim to inform short and longer term investment, research and development decisions.

This review relates how the early Health Technology Assessment approach can help to move next generation diagnostics and biomarkers from lab to patient more efficiently, and discusses its application to oral and health systems medicine.

Health Technology Assessment (HTA) and Early HTA: Back to the Future

Technologies have been studied for safety, effectiveness, cost, and other concerns long before the advent of formal technology assessment. In fact, the early works by later Nobel Prize winner Frederick Soddy—who in the year 1915 forecast the social consequences of atomic energy (i.e., an atomic bomb) long before atomic energy became a mainstream idea—is a very powerful example of technology assessment avant la lettre. Importantly, to achieve his insights, Soddy supplemented scientific knowledge and logical argument with so-called “nonscientific” sources including contemporary politics, social context, emotion, and imagination. This example provides strong evidence for the informative value of multiple sources of knowledge in technology assessment, most eloquently described by Sclove, 1989: “After all, many scientists shared Soddy's scientific knowledge, but none became as committed as he to investigate the social implications of that knowledge, much less reached conclusions or comparable power.”

Similarly, the origins of HTA as a specific strand of research were fueled by the emergence and diffusion of technologies that evoked social, ethical, legal, economic, and political concerns. Among these technologies are, for example, contraceptives, organ transplantation, artificial organs, life-sustaining technologies for critically or terminally ill patients, and, more recently, genetic testing, genetic therapy, and stem cell research. Health technology assessment, as defined in 1994 by the US Congress, is a structured analysis of a health technology, a set of related technologies, or a technology-related issue that is performed for the purpose of providing input to a policy decision (National Information Center on Health Services Research and Health Care Technology, n.d.).

Indeed, HTA asks important questions about health technologies (whether this be drugs, devices, procedures, settings of care, or screening) such as: When is counseling better than drug treatment for depression? What is the best operation for aortic aneurysms? Should we screen for human papilloma virus when doing cervical smears? Should aspirin be used for the primary prevention of cardiovascular disease? It answers these questions by investigating four main factors: whether the technology works, for whom, at what cost, and how it compares with the alternatives (National Institute for Health Research Health Technology Assessment Programme, n.d.).

HTA, therefore, is not a narrowly focused science, but rather a broad multidisciplinary process that summarizes information about the medical, social, economic, and ethical issues related to the use of a health technology in a systematic, transparent, unbiased, robust manner. Its aim is to inform the formulation of safe, effective, health policies that are patient focused and seek to achieve best value.

Despite its policy goals, HTA must always be firmly rooted in research and the scientific method (European Network for Health Technology Assessment, n.d.). Despite the comprehensive approach originally intended for HTA, its practitioners recognized early on that “partial HTAs” may be preferable in circumstances where selected impacts are of particular interest or where necessitated by resource constraints (US Congress. Office of Technology Assessment, 1977). In practice, few HTAs have encompassed the full range of possible technological impacts; most focus on certain sets of impacts or concerns notably health economic ones. As evaluating the cost-effectiveness of a health technology to inform coverage and reimbursement decisions is among the most common aims of an HTA, health economic evaluation and HTA are sometimes referred to interchangeably. We will use the term HTA here to connote cost-effectiveness analysis.

HTA of diagnostics and other technologies are typically undertaken to inform coverage and reimbursement decision (Severens and van der Wilt, 1999) at the time that a health technology is being introduced into the marketplace. This timing, however, poses problems for developers and payers. Developers have at this point made a substantial capital investment in the technology, both in terms of developing the product itself and the evidence supporting its clinical role in care. An unfavorable result of the HTA creates severe problems for the developer, particularly if the negative assessment is based on uncertainties regarding key aspects of performance (e.g., diagnostic specificity) or the impact of the diagnostic on clinical outcomes. In fact, any factor that ‘drives' an unfavorable assessment beyond price implies that the developer will have to make additional investments in research, causing delays in access and further costs.

From the payer/insurer perspective, negative assessments will create tensions with advocates of the technology, particularly if that technology has been introduced into clinical practice prior to the assessment. These issues, combined with the rapidly increasing range and expense of new biomarkers and next generation diagnostics, compel consideration of a more proactive approach to HTA, in which we begin to estimate cost-effectiveness at an early stage of technology development and iterate this evaluation as the development progresses.

The rationale of the early HTA approach is to identify key drivers of diagnostic value, according to various stakeholders, as early as possible and herewith steer evidence development along the innovation process (Steuten and Ramsey, 2014).

(Early) HTA of Biomarkers and Next Generation Diagnostics

As payers have become more critical and put more weight on examining the added value of a diagnostic when making coverage decisions, the need for HTA has increased. This implies that nowadays there is more attention to the cost of biomarkers than there was before the HTA spotlight fell on them. Important to note, however, is a stakeholder analysis by Cohen et al. (2013) showing that the cost of a biomarker is less important for implementation than the biomarker's impact on (longer term) health outcomes.

While this may sound as a relief to some, it shouldn't be. Evaluating the health outcomes of biomarkers and other diagnostics is particularly challenging because diagnostics themselves do not influence long-term outcomes directly, but rather impact the subsequent care process (Gazelle et al., 2005), which may or may not be very effective. In examining the effectiveness of a diagnostic biomarker, one needs to take into account (1) the accuracy of the diagnostic test; (2) the impact of the diagnostic on therapeutic decisions; and (3) the effectiveness of the therapies selected (Cohen et al., 2013; European Network for Health Technology Assessment, 2008).

Developing the evidence for test performance and resulting health outcomes requires substantial resources and time investment; an unfavorable evaluation at the final stages of development is costly. The uncertainties that developers face until biomarkers are tested in the actual environment for which they are intended, can be reduced by performing early HTA at the time when major investment and design decisions are made. Assessment at that stage allows changes that will improve the performance and ultimate health outcomes of the test.

In the context of an early HTA, this requires structuring a decision model that compares the diagnostic with an alternative, typically in the form of a decision tree. For this, the current diagnostic pathway needs to be characterized as well as the potential place of the new diagnostic in that pathway. In early stages of diagnostic test development, it is of crucial importance to understand the diagnostic performance that would be required from a test at each possible place in the pathway, notably in terms of false positives and false negatives. Once the alternative diagnostic pathways have been structured in a decision tree, the model representing those pathways has to be populated with data.

Notably, the (expected) accuracy of the diagnostic technology needs to be estimated as well as its downstream impact on health care provision, therapeutic options, and subsequent patient outcome. Model inputs may be based on empirical data, when available, or experts' opinions. By varying input parameters sequentially in one-way sensitivity analysis, the model can identify parameters that are likely to drive the comparative effectiveness and cost of the new diagnostic compared with the alternative strategy. By performing sensitivity analyses, key areas of uncertainty are highlighted, which form the basis for prioritizing further research (Vallejo-Torres et al., 2011).

During the early HTA, input from key stakeholders such as patient advocates, test developers, clinicians, payers, and regulators is required to sketch out the alternative diagnostic strategies, identify key decision criteria, and to provide estimates of effectiveness and/or costs where no empirical data are available.

Further, the evaluation of the health economic impact of a diagnostic biomarker by definition hinges upon the choice of comparator, whether that is “no testing” or using a different test strategy. Because diagnostic biomarkers are often combined with other (biomarker) tests and because they may potentially be used at different places in the care process, this can result in an unwieldy number of realistic test strategies to be compared.

A recent systematic review of health economic evaluations of diagnostic biomarkers indeed found that the number of comparators in the 33 studies included, ranged from two to seventeen (Oosterhoff et al., 2015). Notably, the amount of strategies to be compared was larger in evaluations of genetic tests, with a mean of six alternative strategies to be compared versus three in evaluations of biomarker tests for diagnosing a disease. The immediate advantage of an early assessment of the potential health outcomes and cost impact of a new biomarker is that it forms a basis for prioritizing between several potential diagnostic strategies, which is efficient in the face of scarce developmental resources.

By focusing on those new biomarkers or test strategies most likely to be cost-effective, the failure rate at each stage of the development process should be reduced, as should be the development costs. If further investment on research is driven by identifying the parameters for which more information is most valuable (for example, accuracy or long-term health impacts), this likely enhances efficient use of research and development resources. For patients, this might translate in earlier access to the most beneficial new biomarkers and other next generation diagnostics (Vallejo-Torres et al., 2008).

Role of (Early) HTA in Oral Health and Systems Medicine

The importance of oral health can be illustrated simply by looking into the two leading dental diseases, caries (tooth decay) and the periodontal diseases. Notwithstanding tremendous improvement in (access to) oral care and public health measures such as fluoridating water supplies, they remain common and widespread, affecting nearly everyone at some point in the life span. What has changed, however, is what we can do about them. The application of oral science to improved diagnostics, treatment, and prevention strategies has saved billions of dollars per year in the US annual health bills.

What remains costly, however, to the individual and to society are the expenses associated with oral health problems that go beyond dental diseases. Associations exist between chronic oral infections and, for example, heart and lung diseases and stroke, while periodontal disease has since long been associated with diabetes. A thorough oral examination can detect signs of nutritional deficiencies as well as a number of systemic diseases, including microbial infections, immune disorders, injuries, and some cancers. Indeed, the phrase that oral health is a mirror of overall health (National Institute of Dental and Craniofacial Health, n.d.) has been used to illustrate the wealth of information that can be derived from examining oral tissues.

From an HTA standpoint, this is interesting, as a relatively low-intensive and low-cost intervention such as an oral exam can generate substantial health benefits and cost savings by detecting potentially severe and costly diseases in their early stages when treatment is most effective. Indeed, HTAs have increasingly been undertaken in dentistry to inform policy makers regarding guideline development and to set future direction for oral health services. The quality of health economic evaluations undertaken in dentistry, however, remains low compared to pharmacoeconomic studies. A systematic review and quality appraisal of economic evaluations in dentistry found that many studies did not meet fairly basic methodological requirements—such as providing sufficient information on how costs and outcomes were measured—and importantly that significant quality improvements were not found in the more recent studies (i.e., full economic evaluations published after 2000) (Tonmukayakul et al., 2015).

While economic evaluations in dentistry and oral health become increasingly common, the potential cost-effectiveness of biomarkers for oral health or systems medicine remains largely unexplored. Yet oral fluids represent a significant source of discriminatory biomarkers for local, systemic, and infectious disorders, and the discovery of saliva-based biomarkers offers unique opportunities to substitute current invasive and sometimes costly procedures, such as biopsies or repeated blood draws, to evaluate the condition of both patients and healthy individuals (Yoshizawa et al., 2013).

Considering local disease, culture-based saliva biomarkers have long been established for risk assessment of dental caries, and dip-slide tests have been shown to be reliable methods for determining salivary levels of mutans streptococci and lactobacilli. More sensitive DNA-based methods including checkerboard DNA, DNA hybridization, genomic fingerprinting, 16S rRNA gene cloning, and sequencing, or TRFLP are also being utilized in identification and classification of dental caries microbiota (Gross et al., 2010; Hommez et al., 2004; Socransky et al., 1994).

Further, the availability of high-throughput DNA sequencing technology together with the rapid expansion of bacterial genome data has now made it feasible to identify the primary bacterial residents in saliva (Cephas et al., 2011; Lazarevic et al., 2010; Pushalkar et al., 2011). It is anticipated that such high-throughput sequencing will assist in identifying potential cariogenic species that may not have been detected using currently available technologies such as 16S rRNA analysis. For the relatively mature diagnostic approaches to assess risk of local disease such as dental caries, HTA alongside prospective studies can be performed to inform coverage and reimbursement decisions. Doing so will answer questions like 'Which price can be justified given the expected clinical and economic value of the diagnostic?' This is of course important for industry, but also government and payers need guidance on whether to reimburse a new diagnostic given current evidence and at what cost.

Moving beyond dental caries, salivary biomarkers may also enhance the detection of oral cancers, for which there are no scientifically credible early detection techniques beyond conventional clinical oral examination. Salivary proteomic and transcriptomic biomarkers have been shown to discriminate oral cancer from control subjects, and established assay technologies are robust enough to perform independently. Individual cutoff values for each of these markers and for the combined predictive model, however, need to be further defined in large clinical studies (Elashoff et al., 2012). HTA can help identify which cut-off values are optimal not only from an effectiveness point of view, but also in terms of cost-effectiveness. It can also be employed to inform the design of such clinical studies by answering questions such as ‘Which additional data to collect to demonstrate the comparative effectiveness and cost-effectiveness of salivary biomarker testing versus usual practice?’ and ‘Which sample sizes are efficient for that?’.

Most exciting perhaps is the potential clinical utility of salivary biomarkers beyond evaluating pathologies of the oral cavity, as microbial and immunologic salivary profiles may be indicative of systemic disease and infectious disorders (Yoshizawa et al., 2013). The diagnosis of such diseases is highly dependent on the evaluation of blood and//or tissue samples. Whereas these procedures are effective, they are also invasive, they may be expensive and often extensive time is required to obtain actionable results. In some practice settings, such as rural and/or developing countries, the tests may not be accessible to patients and health care providers.

Saliva-based test are particularly good candidates to alleviate the accessibility issue, whilst potentially being cheaper and possibly more effective than traditional diagnostic methods. Indeed, several salivary biomarkers are currently investigated for various infectious diseases, such as IgG for HIV-1 and -2; IgG, RNA and antigen for the Ebola virus; and DNA for herpes simplex virus, Epstein-Barr virus, human herpesvirus and cytomegalovirus (Yoshizawa et al., 2013). Saliva-based microbial biomarkers have also been linked to systemic diseases including Crohn's disease, pancreatic cancer, and obesity (Yoshizawa et al., 2013).

This line of research is still in its early stages and according to leading researchers in this field “what must be established now is how these indicators come to exist in the oral cavity and whether the oral microflora is an accurate identifier of additional systemic conditions”(Yoshizawa et al., 2013). Moreover, the authors argue that the establishment of disease-specific microbial signatures could lead to the development of simple tests targeting discriminatory microbes that can identify specific pathology (Yoshizawa et al., 2013). Early detection of systemic disease such as mentioned before, especially in high-risk populations, may allow for therapeutic intervention that inhibits disease progression or even onset, reducing the human burden of disease and the healthcare and societal costs associated with it.

Even in these early stages of diagnostic test development, exploratory economic evaluation can be applied alongside the preclinical studies to inform the most valuable diagnostic development. Typical questions for developers (researchers and industry) at this stage may include: ‘Which possible development directions should we pursue?' or ‘Which diagnostics should we prioritize for validation studies?' Government and research funders may ask at this stage ‘Should we invest public resources in this line of research and development, and if yes, how much?'

Conclusions and Future Outlook

Biomarkers and next-generation diagnostics in oral health and systems medicine provide tremendous opportunity to delivering noninvasive, low-cost, and widely accessible detection and monitoring of a wide array of local, infectious, and systemic diseases. These rapid developments require a strategy to managing development and uptake of diagnostic techniques, in order to maximize health benefit for expenditure (Table 1). Indeed, HTA has been designed to do exactly that. Firmly grounded in health economic and decision analytic theory, this approach iteratively analyzes the prospects of a new biomarker in terms of safety and (cost-) effectiveness comparative to current practice or competitor biomarkers, using the information available at that stage with the aim to inform short and longer term investment, research and development decisions.

Table 1.

Top 10 Executive Take Home Points and Considerations for Early Stage HTA for Precision Medicine in Oral Health

1. The potential value of precision biomarkers in oral health and systems medicine is tremendous as they facilitate noninvasive, low-cost, and widely accessible detection and monitoring of a wide array of local, infectious, and systemic disease.

2. Rapid developments in biomarker and next-generation diagnostics in oral health require a pro-active strategy to managing development and uptake of these techniques, in order to maximize health benefit for expenditure.

3. Health Technology Assessment (HTA) is a multidisciplinary scientific process that informs the evidence-based transition of new discoveries from laboratory to clinic, considering medical, economical and sometimes social and ethical arguments.

4. Early stage HTA is a proactive approach to health economic evaluation of diagnostic technologies, which identifies key drivers of diagnostic value as early as possible and herewith guides the efficiency of the diagnostics innovation process.

5. HTAs are increasingly undertaken in dentistry, but the quality of the evaluations remains relatively low compared to pharmaco-economic studies.

6. The potential cost-effectiveness of biomarkers for oral health or systems medicine is as yet largely unexplored.

7. Demonstrating health outcomes of biomarkers and next-generation diagnostics are particularly challenging because they do not influence long-term outcomes directly, but rather impact subsequent care processes.

8. Biomarker testing costs are typically less of a barrier to uptake in practice than the biomarker's impact on longer term health outcomes.

9. As a single biomarker or next-generation diagnostic in oral health can inform decisions about numerous diagnosis-treatment combinations, early stage HTA is crucial in prioritizing the most valuable diagnostic applications to pursue (first).

10. For the vast array of oral health biomarkers currently developed, early HTA is necessary to timely and iteratively assess their comparative effectiveness and herewith anticipate inevitable questions about value for money from regulators and payers.

While HTA is widely known for informing reimbursement and sometimes market access decisions of pharmaceuticals, it has another, more proactive, application that is increasingly gaining traction in the field of medical technology development. This so-called “early HTA” is applied iteratively during the development and scientific investigation of a new medical technology, updating the analyses as new information becomes available. As such, it helps researchers and medical technology developers steering the development and evidence generation process. As diagnostics in particular are characterized by potentially numerous diagnosis-treatment combinations that can be informed by a single assay, the use of an early HTA approach is even more efficient to prioritize the most valuable way forward.

Moreover, given the vast array of new diagnostics that are currently being developed, it clearly makes sense to start thinking about the comparative effectiveness of new diagnostics sooner rather than later and herewith anticipate the inevitable questions about value for money from regulators and payers. (Early) HTA allows doing so in a systematic and transparent way and increases the efficiency of the R&D process by anticipating future assessments of the added clinical and economic value of biomarkers and next-generation diagnostics in oral health and systems medicine.

Author Disclosure Statement

The author declares there are no conflicting financial interests.

References

Biomarkers Definitions Working Group. (2001). Biomarkers and surrogate endpoints: Preferred definitions and conceptual rramework. Clin Pharmacol Ther, 69, 89–95.

Cephas

, Kim

, Mathai

, et al. (2011). Comparative analysis of salivary bacterial microbiome diversity in edentulous infants and their mothers or primary care givers using pyrosequencing. Plos One, 6, E23503.

Cohen

, Wilson

, and Manzolillo

. (2013). Clinical and economic challenges facing pharmacogenomics. Pharmacogenomics J, 13, 378–388.

Elashoff

, Zhou

, Reiss

, et al. (2012). Pre-validation of salivary biomarkers for oral cancer detection. Cancer Epidemiol Biomarkers Prev, 21, 664–672.

European Network For Technology Assessment. What Is Health Technology Assessment (HTA) [Online]. Available: http://www.Eunethta.Eu/About-Us/Faq#T287n73; Accessed November 1, 2015.

European Network For Technology Assessment. 2008. HTA Core Model For Diagnostic Technologies V1.0r [Online]. Available: http://www.Eunethta.Eu/Outputs/Hta-Core-Model-Diagnostic-Technologies-10r; Accessed November 1, 2015.

Gazelle

, McMahon

, Siebert

, and Beinfeld

. (2005). Cost-effectiveness analysis in the assessment of diagnostic imaging technologies. Radiology, 235, 361–370.

Gross

, Leys

, Gasparovich

, et al. (2010). Bacterial 16s sequence analysis of severe caries in young permanent teeth. J Clin Microbiol, 48, 4121–4128.

Hagen

. (2012). Recent trends in biomarker research and development. Anal Biochem, 1, E108.

10.

Hommez

, Verhelst

, and Al

. (2004). Investigation of the effect of the coronal restoration quality on the composition of the root canal microflora in teeth with apical periodontitis by means of t-rflp analysis. Int Endod J, 37, 819–827.

11.

Lazarevic

, Whiteson

, Hernandez

, Francois

, and Schrenzel

. (2010). Study of inter- and intra-individual variations in the salivary microbiota. Bmc Genomics, 11, 523.

12.

National Information Center On Health Services Research And Health Care Technology. Introduction To Health Technology Assessment [Online]. Available: https://www.Nlm.Nih.Gov/Nichsr/Hta101/Ta10103.Html#Nrc; Accessed November 1, 2015.

13.

National Institute For Health Research Health Technology Assessment Programme. About The HTA Programme [Online]. Available: http://www.Nets.Nihr.Ac.Uk/Programmes/Hta; Accessed November 1, 2015.

14.

Oosterhoff

, Van Der Maas

, and Steuten

. (2015). A systematic review of health economic evaluations of diagnostic biomarkers. Appl Health Econ Health Policy. Epub ahead of print.

15.

Pham

, Tu

, Han

, et al. (2014). Early economic evaluation of emerging health technologies: Protocol of a systematic review. Syst Rev, 3, 81.

16.

Pushalkar

, Mane

, Ji

, et al. (2011). Microbial diversity in saliva of oral squamous cell carcinoma. Fems Immunol Med Microbiol, 61, 269–277.

17.

Sardas

, Endrenyi

, Gursoy

, et al. (2014). A call for pharmacogenovigilance and rapid falsification in the age of big data: Why not first road test your biomarker?. OMICS, 18, 663–665.

18.

Sclove

. (1989). From alchemy to atomic war: Frederick Soddy's “Technology Assessment” of atomic energy, 1900–1915. Sci Technol Human Values, 14, 163–194.

19.

Severens

, and Van Der Wilt

. (1999). Economic evaluation of diagnostic tests. A review of published studies. Int J Technol Assess Health Care, 15, 480–496.

20.

Socransky

, Smith

, and Al

. (1994). “Checkerboard” DNA–DNA hybridization. Biotechniques, 17, 788–792.

21.

Steuten

, and Ramsey

. (2014). Improving early cycle economic evaluation of diagnostic technologies. Expert Rev Pharmacoecon Outcomes Res, 14, 491–498.

22.

The National Institute Of Dental And Craniofacial Health. The Meaning Of Oral Health. [Online]. Available: http://www.Nidcr.Nih.Gov/Datastatistics/Surgeongeneral/Sgr/Chap1.Htm#Relation; Accessed November 5, 2015.

23.

Tonmukayakul

, Calache

, Clark

, Wasiak

, and Faggion

Jr. (2015). Systematic review and quality appraisal of economic evaluation publications in dentistry. J Dent Res, 94, 1348–1354.

24.

US Congress. Office of Technology Assessment, 1977. Technology Assessment in Business and Government. Summary and Analysis. Washington, DC: US Government Printing Office. Available: http://govinfo.library.unt.edu/ota/ota_5/DATA/1977/7711.pdf Accessed 12/14/15.

25.

Vallejo-Torres

, Steuten

, Parkinson

, Girling

, and Buxton

. (2011). Integrating health economics into the product development cycle: A case study of absorbable pins for treating hallux valgus. Med Decis Making, 31, 596–610.

26.

Vallejo-Torres

, Steuten

, Buxton

, Girling

, Lilford

, and Young

. (2008). Integrating health economics modeling in the product development cycle of medical devices: A Bayesian approach. Int J Technol Assess Health Care, 24, 459–464.

27.

Yoshizawa

, Schafer

, Farrell

, Paster

, and Wong

. (2013). Salivary biomarkers: Toward future clinical and diagnostic utilities. Clin Microbiol Rev, 26, 781–791.