Systematic Review of Expert Elicitation Methods as a Tool for Source Attribution of Enteric Illness

Abstract

Expert elicitation is a useful tool to explore sources of uncertainty and to answer questions where data are expensive or difficult to collect. It has been used across a variety of disciplines and represents an important method for estimating source attribution for enteric illness. A systematic review was undertaken to explore published expert elicitation studies, identify key considerations, and to make recommendations for designing an expert elicitation in the context of enteric illness source attribution. Fifty-nine studies were reviewed. Five key themes were identified: the expert panel including composition and recruitment; the pre-elicitation material, which clarifies the research question and provides training in uncertainty and probability; the choice of elicitation tool and method (e.g., questionnaires, surveys, and interviews); research design; and analysis of elicited data. Careful consideration of these themes is critical in designing and implementing an expert elicitation in order to reduce bias and produce the best possible results. While there are various epidemiological and microbiological methods available to explore source attribution of enteric illness, expert elicitation provides an opportunity to identify gaps in our understanding and where such studies are not feasible or available, represents the only possible method for synthesizing knowledge about transmission.

Introduction

Enteric pathogens are important contributors to the estimated 4.6 billion episodes of diarrheal disease occurring globally each year (WHO, 2008). Despite this important public health issue, the relative contribution of different enteric disease transmission routes to enteric disease is still not well known (Pires et al., 2009). Understanding these sources is key to identifying risk factors and ultimately preventing illness. Source attribution refers to the apportioning of human cases of illness to various sources or routes of exposure. Source attribution methods include epidemiological approaches, such as case–control studies and analysis of outbreak data, microbiological approaches including molecular subtyping and comparative exposure assessment, intervention studies, and expert elicitation (Hoffmann et al., 2006; Pires et al., 2009, 2013).

Since its inception (Helmer-Hirschberg, 1966), expert elicitation methodology has evolved into a range of methods that rely on behavioral and mathematical techniques to estimate unknown quantities, to characterize risk pathways, and to quantify uncertainty. Expert elicitation can be used qualitatively to rank pathways or build models (Tan et al., 2010; de Jong et al., 2012) or to produce quantitative estimates, such as proportions (Hoffmann et al., 2006), percentages (Cressey and Lake, 2005; Speirs-Bridge et al., 2010), probabilities (Cooke et al., 2007; Montangero and Belevi, 2007), and natural frequencies (Vally et al., 2014). Expert elicitation provides an opportunity to recruit experts across a range of disciplines to estimate measures of interest and is used to answer questions that are difficult to answer via other methods.

Producing source attribution estimates can be difficult and resource intensive. For many enteric pathogens, there is insufficient research or surveillance-based data available to perform source attribution using methods other than expert elicitation (Pires, 2013). Expert elicitation can be an excellent mechanism for gathering information and supplementing estimates obtained through other attribution methods. Such information can be used to guide the development of more effective food and water safety policies that target interventions at the transmission routes and reservoirs that cause the greatest burden in our communities. Expert elicitation has been used in New Zealand (Cressey and Lake, 2005), the United States (Hoffmann et al., 2006), the Netherlands (Havelaar et al., 2008), Canada (Ravel et al., 2010; Davidson et al., 2011), and Australia (Vally et al., 2014) to inform source attribution efforts toward informing enteric illness prevention strategies and burden-of-illness estimates (Thomas et al., 2013).

The purpose of this systematic review was to identify published expert elicitation studies from a range of disciplines, to explore the state of knowledge and practice in expert elicitation, and to identify key considerations that can be applied in the context of enteric illness source attribution.

Materials and Methods

This review is reported based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocol (Moher et al., 2009). Three research questions were developed to guide this systematic review.

Research questions

1. What methods have been used in previous expert elicitations?

2. What are the strengths and weaknesses of these methods?

3. What are best practices for expert elicitation of enteric illness transmission?

Review protocol

Six databases were searched using keywords to identify expert elicitation studies across any discipline (Table 1). Searches were performed January 7 to February 4, 2013. Search limits included publication dates between 1980 and 2012.

Table 1.

Databases and Search Key Words Used in Literature Review of Expert Elicitation Studies from 1990 to 2012

Database	URL	Publisher	Search key words
Agricola	http://agricola.nal.usda.gov/	United States Department of Agriculture	“expert elicitation”AND “Delphi”
OvidSP	http://gateway.ovid.com/	Wolters Kluwer Health	(“expert elicitation” OR “expert-elicitation” OR “expert opinion” OR “expert-opinion” OR “expert judgement” OR “expert-judgement” OR “attribution” OR “source attribution” OR “delphi method” OR “delphi technique”) AND (“foodborne” OR “waterborne” OR “enteric” OR “transmission” OR “food-borne” OR “water-borne” OR “animal” OR “environment^*” OR “gastrointestinal illness” OR “gastroenteritis” OR “person to person” OR “person-to-person” OR “risk analysis” OR “risk assessment” OR “methodology”)
PubMed	http://www.ncbi.nlm.nih.gov/pubmed	U.S. National Library of Medicine, National Institutes of Health
Science Direct	https://www-sciencedirect-com.web.bisu.edu.cn/	Elsevier
Scopus	https://www-scopus-com.web.bisu.edu.cn/	Elsevier
Web of Knowledge	http://webofknowledge.com/	Thompson Reuters

Study selection and data collection

Initial selection of articles was based on broad relevance screening of articles pertaining to expert elicitation on any subject. Round 1 article selection criteria were based on title and abstract (Table 2) and publication in English in 1990 or later. In round 2, articles were assigned to two categories. Category 1 articles indicate the use of formal expert elicitation methods as the primary study method, indicated by the use of phrases “formal,” “Delphi,” or “structured” in describing elicitation or through reference to specific methods (e.g., Classical Model or Delphi Method). Category 2 articles included review and discussion articles relevant to expert elicitation methods—studies where expert elicitation is a secondary component of the analysis and informal expert knowledge-gathering exercises. Only category 1 articles are reviewed here (Fig. 1).

FIG. 1.

Flowchart of publication selection, adapted from PRISMA Protocol (Moher et al., 2009) for systematic review of expert elicitation studies. 1. Articles identified via branching and hand-selection. 2. Articles excluded based on duplicate publications for the same study (n=5), not meeting screening criteria (n=3) or inability to access full-text based on institutional subscription access (n=2).

Table 2.

Round 1 Screening Criteria for Expert Elicitation Systematic Review of Articles Published 1990–2012

Title or abstract includes ANY of	AND title or abstract suggest	NOT
• Analytical Hierarchy Process (AHP)• Attribution• Delphi method• Expert elicitation• Expert judgment• Expert opinion• Expert panel• Multiple-criteria decision analysis (MCDA)• Multiple-criteria decision making (MCDM) • Nominal group technique• RAND/UCLA (appropriatenesss criteria)	• Discussion of expert elicitation methodologies • Use of expert elicitation• Use of expert opinion or expert judgment (as a source of data in decision making)	• Author's expert opinion• Book reviews • Expert assessment/judgement/opinion of patient-case data• Expert systems (decision-making constructs), except where expert elicitation is used to construct expert systems• Reviews of expert opinion papers published elsewhere

A single reviewer completed initial searches and round 1 screening (AB). Round 2 screening was performed by two reviewers (AB and KT or KP). Conflicts between reviewers were resolved via discussion. References and abstracts were stored in a RefWorks database (ProQuest LLC., RefWorks 2013; http://www.refworks.com/refworks). Critical appraisal of full-text articles was done via an MS Excel spreadsheet. Data were collected about recruitment, elicitation model, research design, question framing, analysis, identified sources of bias, and generalizability of methods for each study reviewed (Table 3).

Table 3.

Data Variables Collected During Critical Appraisal of Round 2 Category 1 Articles, Published 1990–2012

Critical appraisal
General	Author	Last name, first author
	Year	Year of publication
	Type of publication	1=Peer reviewed journal article; 2=Report; 3=Other
	Country	Country where study took place
	Associated publications	E.g., with Ravel (2010) and Davidson (2011) where two articles present the results from one study
Methods	Recruitment method	Snowball, conference, etc.
	Size of expert panel	Number of experts in panel
	Inclusion/exclusion criteria	If inclusion/exclusion criteria is described (e.g., profession, employer)
	Survey method	Postal, in person, electronic, phone, etc.
	Study protocol basis	If study is based on identified protocol from a specific publication (e.g., Hoffman, 2006)
	Background	Subject background (i.e., in literature review, provided)
	Training	Training in probability/uncertainty
	Weighting	E.g., seed questions, linear, expertise, none, other
	Expertise	If expertise is assessed: No, self-assessed, seed, other
	Model	E.g., Delphi, Classical (Cooke), etc.
	Number of rounds	How many rounds of questionnaires or workshops were used?
	Framing	E.g., rank, proportion, pairwise, best–worst, conjoint
	Limits	Maximum/minimum, quantile, credible interval
	Visual representation	Y/N/visual estimation tool
	Uncertainty	E.g., confidence in estimates, N/A
	Distribution	E.g., β distribution
	Explicit consensus process	E.g., Does workshop reach a consensus?
	Consensus statistic	Is a consensus statistic presented?
Discussion	Statistical measures	Free-text field
	Stated limitations	Free-text field
	Bias	Free-text field
	Generalizability of methods	Good, fair, poor
Other	Inclusion	Is this article of sufficient quality and relevance?
	Enteric/foodborne?	Y/N: Is this article an elicitation of the transmission routes of enteric/foodborne pathogens?
	Reviewed bibliography	Y/N: Has the bibliography of this article been reviewed for articles not already under assessment?
	Other	Plain text comments

N/A, not applicable.

Synthesis and analysis

Descriptive analysis was performed on the final selected studies; frequencies and proportions for categories were reported.

Results

Study selection

A total of 13,438 articles were identified from the database searches. From prescreening, 6299 articles were selected based on meeting the relevance criteria (Table 2); after deduplication, 4130 remained to be evaluated. Thirty additional references were added via hand searches of reference lists and through branching by examining the reference lists of relevant reports, primary and review articles, and Google searches for the full text of some publications for sources that matched the original inclusion/exclusion study criteria. The search was further restricted to those that contained the phrase “elicitation” in the title or abstract. Thus, 246 articles were screened in round 1, of which 156 articles were selected for round 2 screening. Ninety articles were excluded due to inclusion and exclusion criteria (including two previously overlooked duplicates). During round 2 screening, 69 articles were assessed as category 1 (formal expert elicitation as primary study outcome), 84 were assessed as category 2 (review or discussion articles, and expert elicitation as secondary study outcome), and 3 were excluded. An additional 10 articles were excluded during critical appraisal: duplications of studies reviewed elsewhere in the critical appraisal process (n=5); the full text did not describe the methods of an expert elicitation (n=3) and the full text was not accessible under institutional subscriptions (n=2). A total of 59 articles, covering a range of subjects (e.g., nuclear safety, health and environmental risks of climate change, health economics, etc.), were included for synthesis, 5 of which explicitly explored source attribution for enteric illness (Fig. 1; Supplementary Appendix A; Supplementary Data are available online at www.liebertpub.com/fpd). In these 59 reviewed studies, key themes in designing an expert elicitation were identified: the expert panel, the pre-elicitation material, the elicitation tool or method, question framing and analysis. We present an exploration of these themes as well as types of bias to be aware of when designing an expert elicitation.

Themes of expert elicitation methods

Key themes identified in the review included the expert panel, the background material supplied, the elicitation model, analysis methods, and research design.

Expert panel: Recruitment, panel size, definition of expert, assessment of expertise

An expert panel is defined by the method of recruitment, the number of experts, and the definition of an expert. Of the studies reviewed, 22/59 (37%) used relevance screening to recruit the expert panel, where experts were chosen based on a priori judgment of the participants' expertise, by the researchers (Table 4). Snowball recruitment was used in 20/59 (34%) of the studies, where a seed group of experts are asked to nominate their peers, who in turn are asked to nominate their peers to build a panel of experts. Experts were also recruited based on convenience (e.g., membership in a panel or attendance at a relevant workshop [n=3]). The method of recruitment was not reported in 22% (13/59) of studies. The majority (14/22; 64%) of studies that used relevance selection used external assessment methods (e.g., publication record, membership in professional group) to evaluate expertise (Table 4). For snowball recruitment, expertise was self-assessed (10/20; 50%), externally assessed (8/20; 40%), or assessed using seed questions (2/20; 10%) (Table 4). The number of experts recruited varied from 2 (Norrington et al., 2008) to 244 (Donlan et al., 2010), with an average of 26 experts (excluding a single study that involved more than 1 expert panel [Goossens and Harper, 1998]) (Fig. 2). Few studies (4/59, 7%) failed to report on expert panel size. There were no explicit inclusion or exclusion criteria for expert selection provided in 12 (20%) studies. In the remaining 47 (80%) studies, inclusion criteria included publication record; professional affiliation (including membership in relevant research groups); technical experience in relevant topics; location or regional expertise; profession; and employer. Exclusion criteria included geographic location or regional expertise and recruitment was restricted to local or regional experts due to resources or subject matter. Seed questions were used to assess and/or calibrate expertise in 4/59 (7%) studies. Other methods for defining expertise were based on external assessment by researchers or their peers (23/59; 39%), or self-assessment (15/59; 25%).

FIG. 2.

Expert panel size as reported in 57 reviewed expert elicitation studies (as frequency), published 1990–2012.

Table 4.

Method of Expert Recruitment in 59 Reviewed Expert Elicitation Studies, Published 1990–2012

	Expertise
Recruitment method	Self-assessed	Externally assessed	Seed questions	Not addressed	Total
Relevance selection	4	14	1	3	22
	(Bouma et al., 2011; Ravnum et al., 2012; Saunders et al., 2012; Zimmer et al., 2012)	(Arnell et al., 2005; Baker and Keisler, 2011; Coppersmith et al., 2009; Forsberg et al., 2012; Gordon and Gallo, 2011; Hirsch et al., 2004; Lieu et al., 2012; Marcot et al., 2012; Montangero and Belevi, 2007; Speirs-Bridge et al., 2010; Stiber et al., 1999; Tan et al., 2010; van der Fels-Klerx et al., 2000; Vally et al., 2014)	(van der Fels-Klerx et al., 2002)	(Acosta et al., 2010; Cressey and Lake, 2005; Horst et al., 1998)
Snowball recruitment	10	8	2	0	20
	(de Jong et al., 2012; Elmer et al., 2010; Evans et al., 1994; Flander et al., 2012; Garabed et al., 2009; Hoelzer et al., 2012; Hoffmann et al., 2007; Mcdaniels et al., 2012; Ravel et al., 2010; Wardekker et al., 2012)	(Cross et al., 2012; Goossens and Harper, 1998; Hoek et al., 2010; Knol et al., 2009; Leal et al., 2007; Page et al., 2012; Roman et al., 2012; Walker et al., 2001)	(Cooke et al., 2007; Tuomisto et al., 2008)
Other	1	0	0	3	4
	(Donlan et al., 2010)			(Gosling et al., 2012; Leon et al., 2003; Podofillini et al., 2010)
Not stated	3	1	1	8	13
	(Doria et al., 2009; Fauss et al., 2009; Havelaar et al., 2008)	(Hnottavange-Telleen et al., 2011)	(Tyshenko et al., 2011)	(Brito et al., 2012; Cagno et al., 2000; Kangas et al., 1998; Kuhnert et al., 2010; Norrington et al., 2008; Santori et al., 2004; van der Fels-Klerx HJ et al., 2009; Wreathall et al., 2004)
Total	18	23	4	14	59

Pre-elicitation material

Background information was provided to experts in 40/59 (68%) studies, with a briefing book most often used (Table 5). Many (23/59; 39%) studies reported training experts in probability and uncertainty (e.g., through providing worked examples); 18/23 (78%) studies provided additional background information (Table 5). Three elicitation tools were employed in the majority of studies reviewed: questionnaires (41/59; 69%), workshops (25/59; 42%), and interviews (11/59; 19%) (Table 6). Workshops were used as either the sole elicitation method (8/25; 32%), or in combination with other elicitation tools (17/25; 68%). Of those studies using workshops, 6/25 (24%) reported using a modified Delphi approach, 3/25 (12%) reported using the HENVINET elicitation framework (Bartonova, 2012), and 2/25 (8%) reported using the Cooke method (Cooke, 1991). Other methods included the SHeffield ELicitation Framework (SHELF) (O'Hagan, 2012), the ELI elicitation technique (van Lenthe, 1993), informal group processes, and probability trees. Questionnaires were administered using various methods (Table 7). One-on-one interviews included structured, questionnaire-based, and narrative approaches. Pilot testing or validation of the survey materials prior to administration was reported in 17/59 (29%) studies (Table 5). Pilot testing was conducted by co-workers or subject matter experts not part of the expert panel.

Table 5.

Three-Way Table of the Frequency of Pilot Testing, Providing Background Information and Training Experts in Elicitation Methodology in 59 Reviewed Expert Elicitation Studies, Published 1990–2012

	Pilot Testing?
	Yes		No
	Background information provided?		Background information provided?
Training in elicitation?	Yes	No	Yes	No	Total
Yes	7	3	11	2	23
	(Acosta et al., 2010; Goossens and Harper, 1998; Gosling et al., 2012; Leal et al., 2007; Mcdaniels et al., 2012; Page et al., 2012; Roman et al., 2012)	(Hoffmann et al., 2007; Tyshenko et al., 2011; van der Fels-Klerx et al., 2002)	(Cooke et al., 2007; Evans et al., 1994; Fauss et al., 2009; Gordon and Gallo, 2011; Havelaar et al., 2008; Hoek et al., 2010; Leon et al., 2003; Lieu et al., 2012; Marcot et al., 2012; Montangero and Belevi, 2007; Tuomisto et al., 2008)	(Norrington et al., 2008; Ravel et al., 2010)
No	6	1	16	13	36
	(Baker and Keisler, 2011; Coppersmith et al., 2009; Elmer et al., 2010; Hoelzer et al., 2012; Stiber et al., 1999; van der Fels-Klerx et al., 2000)	(Forsberg et al., 2012)	(Brito et al., 2012; de Jong et al., 2012; Garabed et al., 2009; Knol et al., 2009; Kuhnert et al., 2010; Horst et al., 1998; Ravnum et al., 2012; Speirs-Bridge et al., 2010; Saunders et al., 2012; van der Fels-Klerx, HJ. et al., 2009; Walker et al., 2001; Wreathall et al., 2004; Wardekker et al., 2012; Zimmer et al., 2012; Vally et al., 2014)	(Arnell et al., 2005; Bouma et al., 2011; Cagno et al., 2000; Cressey and Lake, 2005; Cross et al., 2012; Donlan et al., 2010; Flander et al., 2012; Hnottavange-Telleen et al., 2011; Hirsch et al., 2004; Kangas et al., 1998; Podofillini et al., 2010; Santori et al., 2004; Tan et al., 2010)
Total	13	4	27	15	59

Table 6.

Elicitation Tools Employed in 59 Expert Elicitation Studies Reviewed, Subdivided by Number of Elicitation Rounds Employed (Studies May Have Used More Than One Elicitation Tool)

Elicitation tool	One elicitation round	Multiple rounds	Total
Questionnaires	18	23	41
	(Baker and Keisler, 2011; Bouma et al., 2011; Cagno et al., 2000; Cressey and Lake, 2005; Cross et al., 2012; de Jong et al., 2012; Donlan et al., 2010; Garabed et al., 2009; Havelaar et al., 2008; Hoffmann et al., 2006; Horst et al., 1998; Leal et al., 2007; Kuhnert et al., 2010; Podofillini et al., 2010; Ravel et al., 2010; Speirs-Bridge et al., 2010; Stiber et al., 1999; Wardekker et al., 2012)	(Acosta et al., 2010; Arnell et al., 2005; Doria et al., 2009; Elmer et al., 2010; Forsberg et al., 2012; Hnottavange-Telleen et al., 2011; Hoek et al., 2010; Hoelzer et al., 2012; Knol et al., 2009; Leon et al., 2003; Lieu et al., 2012; Page et al., 2012; Ravnum et al., 2012; Roman et al., 2012; Santori et al., 2004; Saunders et al., 2012; Tan et al., 2010; Tyshenko et al., 2011; Vally et al., 2014; van der Fels-Klerx et al., 2000; van der Fels-Klerx et al., 2002; van der Fels-Klerx HJ, et al., 2009; Zimmer et al., 2012)
Workshops	6	19	25
	(Cressey and Lake, 2005; de Jong et al., 2012; Gosling et al., 2012; Horst et al., 1998; Mcdaniels et al., 2012; Wreathall et al., 2004)	(Brito et al., 2012; Coppersmith et al., 2009; Doria et al., 2009; Evans et al., 1994; Forsberg et al., 2012; Goossens and Harper, 1998; Gordon and Gallo, 2011; Hnottavange-Telleen et al., 2011; Hoek et al., 2010; Knol et al., 2009; Marcot et al., 2012; Ravnum et al., 2012; Saunders et al., 2012; Tan et al., 2010; Tyshenko et al., 2011; Vally et al., 2014; van der Fels-Klerx et al., 2000; van der Fels-Klerx et al., 2002; Zimmer et al., 2012)
Interviews	4	7	11
	(Cooke et al., 2007; Hirsch et al., 2004; Tuomisto et al., 2008; Walker et al., 2001)	(Evans et al., 1994; Fauss et al., 2009; Flander et al., 2012; Lieu et al., 2012; Norrington et al., 2008; Page et al., 2012; Roman et al., 2012)
Not stated	1	1	2
	(Montangero and Belevi, 2007)	(Kangas et al., 1998)

Table 7.

Method of Dissemination of Questionnaires in 41 Reviewed Expert Elicitation Studies, Published 1990–2012

Questionnaire dissemination method	Number of articles	References
Electronic (online, e-mail)	15 (37%)	(Arnell et al., 2005; Cross et al., 2012; Donlan et al., 2010; Doria et al., 2009; Elmer et al., 2010; Forsberg et al., 2012; Garabed et al., 2009; Havelaar et al., 2008; Hnottavange-Telleen et al., 2011; Kuhnert et al., 2010; Ravnum et al., 2012; Saunders et al., 2012; van der Fels-Klerx HJ, et al., 2009; Wardekker et al., 2012; Zimmer et al., 2012)
Post	7 (17%)	(Hoelzer et al., 2012; Hoffmann et al., 2007; Ravel et al., 2010; Stiber et al., 1999; Tan et al., 2010; van der Fels-Klerx et al., 2000; van der Fels-Klerx et al., 2002)
Phone interviews	3 (7%)	(Baker and Keisler, 2011; Page et al., 2012; Roman et al., 2012)
Computer program	1 (2%)	(Leal et al., 2007)
Not explicitly stated	15 (37%)	(Acosta et al., 2010; Bouma et al., 2011; Cagno et al., 2000; Cressey and Lake, 2005; de Jong et al., 2012; Hoek et al., 2010; Horst et al., 1998; Knol et al., 2009; Leon et al., 2003; Lieu et al., 2012; Podofillini et al., 2010; Santori et al., 2004; Speirs-Bridge et al., 2010; Tyshenko et al., 2011; Vally et al., 2014)

Elicitation tool: Format, number of rounds

In 33/59 studies (56%), more than 1 round of elicitation was undertaken (Fig. 3). In nearly half of these studies (n=16), different methods were used between rounds (e.g., both questionnaire and workshop) and an explicit consensus process (e.g., sharing group results from previous rounds and producing consensus statements from workshop discussions) was employed in 18/33 (55%) studies.

FIG. 3.

The number of elicitation rounds undertaken in 59 reviewed expert elicitation studies (as frequency), published 1990–2012.

Question framing

Roughly half (31/59; 53%) of the studies reviewed used expert elicitation to produce quantitative estimates, including measures of central tendency, percentiles, and probabilities across distributions (Table 8). A combination of both quantitative and qualitative measures were collected in 23/59 (39%) studies to elicit ranking and scoring, to generate model parameters, and to characterize risk factors (Table 8). Few (3/59; 5%) studies collected information using more than 1 method. Nearly half (27/59; 46%) of the studies collected information about the limits of the estimated quantities, as 5^th and 95^th percentiles or confidence intervals (n=13) and maximum and minimum values (n=12), and 3/59 (5%) studies collected information across a wider range, including the 5^th, 25^th, 50^th, 75^th, and 95^th percentiles to produce more detailed probability distribution functions. Visualization (e.g., using dynamic graphs to represent probability distributions or drawing model tree structures) was incorporated into the elicitation process in 23/59 (39%) studies, including 8/59 (14%) that specifically used visual estimation tools to collect probabilities (n=4), percentiles (n=2), ranking (n=1), and mean values (n=1) (Table 9).

Table 8.

Analysis Approaches Used as Described in 59 Reviewed Expert Elicitation Studies, Published 1990–2012

Methods	Frequency	References
Quantitative methods
Measures of central tendency	7	(Brito et al., 2012; Gosling et al., 2012; Hirsch et al., 2004; Leon et al., 2003; Tyshenko et al., 2011; Walker et al., 2001; Vally et al., 2014)
Percentiles and probabilities across distributions	23	(Arnell et al., 2005; Baker and Keisler, 2011; Bouma et al., 2011; Cooke et al., 2007; Cressey and Lake, 2005; Evans et al., 1994; Flander et al., 2012; Garabed et al., 2009; Goossens and Harper, 1998; Havelaar et al., 2008; Hoek et al., 2010; Hoelzer et al., 2012; Hoffmann et al., 2007; Leal et al., 2007; Mcdaniels et al., 2012; Montangero and Belevi, 2007; Ravel et al., 2010; Roman et al., 2012; Speirs-Bridge et al., 2010; Stiber et al., 1999; Tuomisto et al., 2008; van der Fels-Klerx et al., 2002; Wreathall et al., 2004)
Mixed methods
Ranking (e.g., of model parameters, risk factors)	13	(Acosta et al., 2010; de Jong et al., 2012; Doria et al., 2009; Knol et al., 2009; Kuhnert et al., 2010; Lieu et al., 2012; Page et al., 2012; Ravnum et al., 2012; Santori et al., 2004; Saunders et al., 2012; van der Fels-Klerx HJ et al., 2009; van der Fels-Klerx et al., 2000; Zimmer et al., 2012)
Scoring (e.g., of the relative importance of factors in a scenario)	8	(Cagno et al., 2000; Donlan et al., 2010; Elmer et al., 2010; Forsberg et al., 2012; Hnottavange-Telleen et al., 2011; Marcot et al., 2012; Podofillini et al., 2010; Wardekker et al., 2012)
Model structures and scenarios (e.g., identifying model parameters in ecological assessments)	2	(Tan et al., 2010; Gordon and Gallo, 2011)
Other measures
Open-ended interview questions	1	(Fauss et al., 2009)
Best–worst scaling	1	(Cross et al., 2012)
Length	1	(Horst et al., 1998)
Varied	3	(Coppersmith et al., 2009; Kangas et al., 1998; Norrington et al., 2008)

Table 9.

Use of Explicit Visualization Tools in Estimate Collection in 23/59 Expert Elicitation Studies That Explicitly Describe Using Visualization Tools to Enhance Expert Elicitation

Explicit visualization tool used in elicitation of estimates?	Frequency	Reference
Yes	8
Probabilities	4	(Evans et al., 1994; Flander et al., 2012; Leal et al., 2007; Mcdaniels et al., 2012)
Percentiles	2	(Hoek et al., 2010; van der Fels-Klerx et al., 2002)
Mean values	1	(Leon et al., 2003)
Ranking	1	(Page et al., 2012)
No	15	(Baker and Keisler, 2011; Fauss et al., 2009; Forsberg et al., 2012; Gordon and Gallo, 2011; Havelaar et al., 2008; Hirsch et al., 2004; Knol et al., 2009; Norrington et al., 2008; Ravnum et al., 2012; Roman et al., 2012; Saunders et al., 2012; Stiber et al., 1999; Tyshenko et al., 2011; Wreathall et al., 2004; Zimmer et al., 2012)

Analysis

In combining expert opinions, 18/59 (31%) studies reviewed used weighting based on seed or calibration questions (n=5), expertise (n=5) exclusion filters (n=3) or other methods (n=5); in 17/59 (29%) studies linear opinion pooling or simple averaging were used (Table 10); and in 7/59 (12%) studies, Bayesian networks were built using the elicited judgments. The types of distributions explicitly described in the reviewed studies for characterizing elicited quantities and probabilities varied, and included Bayesian modeling (n=6), β distributions (n=4), and normal and log-normal distributions (n=6). Cumulative distributions were not addressed in 23 studies, which included both qualitative and quantitative studies.

Table 10.

Methods Used to Weight Elicited Expert Estimates in 35 Expert Elicitation Studies in Which Explicit Weight Methods Were Described

	Frequency	References
Weighting based on:
Seed or calibration questions	5	(Cooke et al., 2007; Tuomisto et al., 2008; Tyshenko et al., 2011; van der Fels-Klerx et al., 2002)
Expertise	5	(Donlan et al., 2010; Evans et al., 1994; Garabed et al., 2009; Ravel et al., 2010; Wardekker et al., 2012)
Other	5	(Acosta et al., 2010; Baker and Keisler, 2011; de Jong et al., 2012; Hirsch et al., 1998; Horst et al., 1998)
Exclusion filters	3	(Havelaar et al., 2008; Leon et al., 2003; van der Fels-Klerx et al., 2000)
Linear opinion pooling/simple average	17	(Bouma et al., 2011; Brito et al., 2012; Coppersmith et al., 2009; Cressey and Lake, 2005; Flander et al., 2012; Goossens and Harper, 1998; Gosling et al., 2012; Hoelzer et al., 2012; Hoffmann et al., 2006; Kangas et al., 1998; Leal et al., 2007; Mcdaniels et al., 2012; Montangero and Belevi, 2007; Norrington et al., 2008; Stiber et al., 1999; Vally et al., 2014; Wreathall et al., 2004)

Discussion

This review explored 59 expert elicitation studies over a broad range of topics to develop a comprehensive summary of key themes and considerations for performing an expert elicitation in the context of enteric illness attribution. Expert elicitation can be performed with a behavioral or mathematical elicitation approach, to obtain an explicit consensus estimate or to explore the uncertainty in expert knowledge on a topic. The methods used in performing expert elicitations across the disciplines can include both formal and informal processes, which influence the final output. Identified sources of bias include the expert panel employed; the type of preparatory material provided to participants; the choice of elicitation model; the method of analysis; and the specific research design (Cooke, 1991; Walker et al., 2001). In designing an expert elicitation, researchers need to carefully consider and report the sources of bias (Cooke, 1991).

There are a number of biases common across expert elicitation studies. Anchoring derives from a failure to adjust viewpoints based on new information and can lead to fixation of estimates about initial values despite introduction of data to the contrary. Availability bias relates to the tendency to base estimates on information that is easily recalled. Base rate bias arises from failure to consider underlying population rates. Overconfidence relates to the tendency to produce estimates with too great a level of certainty or confidence bounds that are too narrow (Tversky and Kahneman, 1974; Walker et al., 2001; Kynn, 2008).

Recruitment methods can lead to bias through sampling from specific subsets of expert groups in recruiting expert panels, thus limiting the breadth of available knowledge and expertise (Tversky and Kahneman, 1974; Cooke, 1991). Relevance-based recruitment by experts can help to ensure that relevant stakeholders are recruited, but could potentially lead to overrepresentation of experts with whom researchers are familiar and who may hold similar opinions or viewpoints of the research questions (Evans et al., 1994). Snowball recruitment helps to ameliorate the influence of researcher bias in recruitment (recruiting who they know), but depending on the initial snowball sample being used, bias can still occur (e.g., do the experts represent relevant stakeholder groups who are qualified to address the research question?).

Some literature suggests that a panel size larger than seven experts does not improve the strength of the study (Ashton, 1986; Cooke, 1991; Winkler and Clemen, 2004). Others disagree, suggesting that larger panels increase the statistical power of the analysis and allow for better characterization of uncertainty (Clemen and Winkler, 1999; Moon and Kang, 1999; Hoffmann et al., 2007). Given the variety of subjects included in this review, a comparison of expert definitions used in the reviewed studies is not practical, but common concepts are presented herein. Transparent methodology, involving explicit inclusion and exclusion criteria, is critical (Ayyub, 2001). Without inclusion or exclusion criteria, it can be difficult for researchers to ensure that the recruited experts' knowledge is adequate. Expert selection from across disciplines (e.g., academia, industry, and public service) and broad organizational representation can help produce richer and more balanced elicitations (Ayyub, 2001).

Expertise can be quantitatively assessed using seed questions (van der Fels-Klerx et al., 2002; Cooke et al., 2007), though this is not always feasible, or through self-assessment (Hoffmann et al., 2006; Donlan et al., 2010). Seed questions are a tool for addressing overconfidence and providing an appropriate calibration for responses (Morgan and Henrion, 1992; Walker et al., 2003). Asking experts to estimate quantities that are known, or that will become known during the project timeframe, can measure an expert's ability to accurately produce estimates on the topic of interest (Cooke and Goossens, 2008). A problem with this approach lies in the ability to find questions for which answers are known or that can become known that are directly relevant. Self-assessed expertise and background information from the expert panel can be used to explore estimate weighting or use of threshold models to calibrate the estimates and reduce overconfidence bias (Ayyub, 2001; Ravel et al., 2010). Self-assessed expertise is considered more useful in characterizing experts' uncertainty with their own estimates (Hoffmann et al., 2007) than as a method for defining expertise.

Briefing books can present summaries or lists of relevant studies, outline the elicitation process, and frame the research questions. They can also provide guidance to experts on uncertainty and probability (e.g., through a worked example) (Kynn, 2008), which can reduce bias from misinterpretation of elicitation methods (Tversky and Kahneman, 1974; Cooke, 1991). These tools are similar to a pretest phase of a survey, which ensures that instructions are clear, questions are easily interpreted, and bias resulting from word choices is minimized (Knol et al., 2010). Providing experts with summaries or access to relevant current literature as part of a briefing book can help to address availability bias and base-rate bias (Cooke, 1991; Walker et al., 2001); however, it is necessary to ensure that these studies are representative of current knowledge on the topic and are not introducing a new layer of bias (Kahneman and Tversky, 1973). Base rate and representativeness bias can be avoided by providing experts information on the incidence or underlying frequency of exposures (e.g., surveillance data or toxin concentrations in the environment) (Koehler, 1996).

A variety of elicitation methods are available including workshops, questionnaires, and interviews. A key difference in these methods lies in the type of interaction between experts and facilitators, and the level of anonymity provided to participants (Brito et al., 2012). Anonymity of participants allows experts to provide estimates or judgments that may not be congruent with corporate or departmental views (Hetes et al., 2011).

Workshops and face-to-face discussion can be costly in both time and money and be difficult for experts to attend; however, they provide opportunity for experts to discuss sources of disagreement (Clemen and Winkler, 1999). A trained facilitator is recommended in workshops to reduce the potential for bias from group dynamics (Tan et al., 2010, Tyshenko et al., 2010, 2011). Nominal group technique is the most commonly used tool for structured information collection from workshops, involving the collection of input from all members, and discussion and ranking of suggested outcomes (Delbecq and van de Ven, 1971). Other forms of group decision making for workshops include the analytic hierarchy process, a hierarchical model for prioritizing outcomes or risk factors (Forman and Gass, 2001), and the RAND/UCLA Appropriateness criteria, created for supplementing scientific evidence in designing clinical process in medical care (Fitch et al., 2001). The choice of workshop method is dependent on research priorities and outcomes of interest.

More than one round of elicitation was used in over half the studies, which can provide the opportunity to resolve conflict and encourage consensus. The potential benefits of multiple rounds should be weighed against potential cost in time and money. Increased rounds of elicitation may also lead to loss of participants (dropouts) (Doria et al., 2009). The Cooke (Classical) model recommends using one round of elicitation (Cooke, 1991), reducing the risk of results becoming skewed in the direction of the “loudest” voices (Aspinall, 2010). Providing summary statistics between rounds can lead to an adjustment toward a central value (Aspinall, 2010). Including a discussion or workshop following quantitative (questionnaire) estimation can reduce the risk of anchoring bias by encouraging experts to discuss the reasoning behind their estimates (Goossens and Harper, 1998; Coppersmith et al., 2009).

Question framing depends on the outcome of interest and dictates which type of measures are collected (e.g., probability, mean, frequency, weighting), whether participants require an understanding of probability theory (e.g., probability distributions) and whether uncertainty will be explicitly quantified (e.g., quantitative measure, probability distribution) (Kuhnert et al., 2010). In eliciting quantitative measures, researchers should consider the potential benefits of using methods with easily interpreted language such as natural frequencies or means (Kuhnert et al., 2010).

Several methods have been used to aggregate elicited quantities. Linear opinion pools or simple averaging are commonly used as a straightforward method of combining expert opinions and are recommended by Cooke's (Classical) method (Cooke, 1991). Weighting of expert opinions in aggregate functions (e.g., through validation or seed questions) is recommended (Cooke, 1991). Alternatively, estimates can be aggregated on the basis of self-assessed expertise; however, this introduces the potential for overconfidence bias (Cooke, 1991).

Analytical tools chosen by researchers are highly dependent on research questions, the elicitation framework, and the elicited values. Presentation of summary statistics can help to characterize the data being collected and identify any trends or anomalies, such as a bimodal distribution (Ravel et al., 2010).

The distribution used to characterize summarized probabilities is also important. A normal distribution can often fit elicited data; however, as variation increases, other distributions can also be considered including the β distribution (Ravel et al., 2010) and the γ distribution (Cagno et al., 2000). Fuzzy distributions, Bayesian methods, and probabilistic inversion can also be used in producing probability distribution functions (Clemen and Winkler, 1999; Moon and Kang, 1999; Havelaar et al., 2008, Kuhnert et al., 2010).

Uncertainty in elicited data is a recurring theme in the literature. Researchers need to consider several sources of uncertainty in their estimates, as stated by Hoffman et al. (2007): (1) variability in expert judgment; (2) the level of agreement between experts' assessments and prior estimates based on primary data; (3) individual experts' uncertainty about their own assessments; and (4) variability in individual experts' uncertainty about their own best estimates.

Application of Expert Elicitation Methods: Enteric Illness Source Attribution

Of the 59 studies reviewed, 5 (8%) used expert elicitation to explore source attribution in (foodborne) enteric illness in Canada (Ravel et al., 2010), the United States (Hoffmann et al., 2007), The Netherlands (Havelaar et al., 2008), New Zealand (Cressey and Lake, 2005), and Australia (Vally et al., 2014) (Table 11).

Table 11.

Summary of Key Methodology Criteria Used in Five Enteric Illness Expert Elicitation Studies, Published 1990–2012

Methodology	New Zealand (Cressey and Lake, 2005)	United States (Hoffmann et al., 2006)	The Netherlands (Havelaar et al., 2008)	Australia (Vally et al., 2014)	Canada (Ravel et al., 2010)
Recruitment method	Relevance selection	Snowball	Not stated	Relevance selection	Snowball
Panel size	14	42	16	12	54
Elicitation tool	Questionnaire (during workshop)	Questionnaire (mail)	Questionnaire (e-mail)	Questionnaire (workshop)	Questionnaire (mail)
Background data provided?	No	No	Yes	Yes	No
Expertise	Not stated	Self-assessed	Self-assessed	Externally assessed	Self-assessed
Framing	Percentage	Proportion	Range, probability	Natural frequency	Percentage
Elicited limits	Lower and upper bound estimates	5^th and 95^th percentiles of cases	90% certainty intervals	90% certainty bounds	5^th and 95^th percentile limits of estimate
Elicitation rounds	1	1	1	3	1

Multisectoral expert panels were recruited in the Canadian (Ravel et al., 2010) and U.S. (Hoffmann et al., 2007) enteric illness studies, while the Australian (Vally et al., 2014) study used a narrower definition of experts (epidemiologists and public health professionals). Recruiting across a range of disciplines, the Canadian (Ravel et al., 2010) and the U.S. (Hoffmann et al., 2007) studies used larger expert panel sizes (54 and 42, respectively) than the Australian (Vally et al., 2014) (n=12), New Zealand (Cressy and Lake, 2005) (n=14), and The Netherlands (Havelaar et al., 2008) (n=16) studies. Panels were composed of experts from academia, public service, and industry, allowing the panels to draw from a breadth of experience (Hoffmann et al., 2006; Havelaar et al., 2008; Ravel et al., 2010; Vally et al., 2014), except in the New Zealand study, where panel composition was not described (Cressey and Lake, 2005).

Self-assessment of expertise in reference to the subject material as a whole or in the context of specific questions was employed in three of the five enteric illness elicitations (Hoffmann et al., 2006; Havelaar et al., 2008; Ravel et al., 2010; Davidson et al., 2011), externally assessed in one of the elicitations (Vally et al., 2014), or not described (Cressy and Lake, 2005) (Table 11). Generating appropriate seed questions is difficult in the context of enteric illness research and has not been reported. Other health-related expert elicitations have used surveillance data as a basis for seed questions (e.g., air pollution and mortality) (Cooke et al., 2007).

A thorough review of the current state of knowledge of the transmission of the illnesses being studied is an appropriate first step. This information helps frame the expert elicitation and provides a source of background information for the experts. The Australian elicitation provided a detailed literature review to the expert panel (Vally et al., 2014). Current illness surveillance data were provided to experts in two of the studies (Havelaar et al., 2008; Vally et al., 2014), and experts were encouraged to consider the potential sources of bias. While a useful tool for framing the research question and providing experts with the best available information from which to draw their estimates, briefing books also increase the likelihood of availability bias, by potentially providing only part of the information on a topic (Walker et al., 2001). The Canadian, U.S., and Australian elicitations provided a working example of the questionnaire as part of a briefing book (Hoffmann et al., 2006; Ravel et al., 2010; Davidson et al., 2011; Vally et al., 2014). Training experts to use the elicitation tool and in uncertainty can help address several sources of bias (e.g., anchoring, overconfidence, representativeness, and the base rate fallacy) (Cooke, 1991). Elicitation materials were only reported as being pretested in the U.S. study (Hoffmann et al., 2006). The Canadian and Australian studies (Ravel et al., 2010; Vally et al., 2014) built upon the U.S. survey tool.

Both workshops and questionnaires were employed in two of the studies (Cressey and Lake, 2005; Vally et al., 2014), and in the remaining three (Hoffmann et al., 2006; Havelaar et al., 2008; Ravel et al., 2010), questionnaires were mailed (n=2) or e-mailed (n=1) to experts. The Australian expert elicitation is the only enteric illness source attribution project that used multiple elicitation rounds to achieve consensus, and incorporated a discussion between rounds of survey administration (Vally et al., 2014).

There are many methods for measures and evaluating expertise. Researchers should consider whether the ability to predict surveillance data or other information would serve as a proxy for expertise in enteric illness transmission. In three of the studies, experts were asked to self-assess their expertise. In the Canadian elicitation study, a threshold model was created using self-assessed expertise ≥3 out of 5 as a cut-off for inclusion of estimates in the linear opinion pool. In the United States, experts were asked to produce estimates only for pathogens for which they felt their expertise was sufficient (Hoffmann et al., 2006). Linear opinion pools were used in the New Zealand, Australian, Canadian, and U.S. expert elicitations of enteric illness to combine expert opinion (Cressey and Lake, 2005; Hoffmann et al., 2006; Ravel et al., 2010; Vally et al., 2014). Cluster analysis was used to explore unexpected bimodal distributions in the Canadian elicitation data (Ravel et al., 2010). Triangular probability (n=3), normal (n=1), and PERT (n=1) distributions were produced from the individual estimates, which were combined using simple averaging (n=2), probabilistic inversion (n=1), or Monte Carlo simulation (n=2).

To elicit information about the proportion of cases of human illness transmitted by specific routes, questionnaires are beneficial for estimating natural frequencies or percentages and characterizing the surrounding uncertainty, using measures such as 90% certainty intervals. The incorporation of workshops allows experts to review the collective estimates and to discuss variation or divergence in estimated opinions. This can highlight sources of disagreement between experts and reduce the risk of unexplainable clustering such as observed in the 2009 Canadian elicitation. Readministration of a survey following discussion allows for revisions, reducing the potential bias of anchoring (Knol et al., 2009). Collecting information about self-assessed expertise can calibrate responses, to reduce the influence of overconfidence, especially when seed questions are not used.

On the basis of the reviewed literature, in designing a future expert elicitation of enteric illness source attribution, the strategies outlined in Table 12 should be used to strengthen future approaches.

Table 12.

Recommendations for the Design of a Future Expert Elicitation of Enteric Illness Source Attribution

Key step	Strategy
Recruitment	Recruit a diverse expert panel using a nonbiased sampling technique such as snowball recruitment.
Pre-elicitation material	Provide background material to participants clarifying the study objectives, providing definitions and context, and introducing basic concepts of estimating uncertainty (e.g., through a worked example of the elicitation tool).
Elicitation rounds	Incorporate multiple elicitation rounds, primarily through dissemination of a questionnaire. If resources allow, the inclusion of a discussion round can provide an important opportunity to clarify any confusion and overcome points of disagreement.
Question framing	Transmission attribution estimates should be elicited using natural frequencies as they require little understanding of underlying uncertainty and probability theory on the part of the participants. Confidence intervals around these estimates are a good way to characterize experts' certainty in their estimates.
Expertise	Assessment or calibration of participant expertise should be attempted. Seed questions would be ideal to calibrate participant expertise and accuracy in producing estimates; however, in the context of enteric illness, appropriate seed questions are difficult to develop. Experts should be asked to self-assess their expertise, such that they can indicate where they feel little to no confidence in their ability to produce accurate estimates for a given pathogen or transmission route.
Analysis	An exploration of the utility of weighting elicited distributions based on self-assessed expertise or other measures of experience is recommended (e.g., any demographic factors collected). Cumulative probability distributions produced from estimates should use dynamic and flexible distributions such as PERT or β distributions wherever possible.

Study Limitations

Sources of bias in this review included publication bias, the use of specialized terminology, and restriction of the search to health- and biological science–oriented indexing databases. Articles using the phrase “expert elicitation,” one of the primary search strings, are likely to emulate protocols outlined in relatively few seminal papers (e.g., the Cooke method) (Cooke, 1991), especially within disciplines. This can increase comparability across studies, but introduces the potential for bias toward specific practices that may not represent all established best practices or the underlying diversity of methods employed.

Due to the high volume of articles identified using the initial search string (4130), round 1 review of literature was restricted to 246 articles containing the word “elicitation” in the title or abstract. This may have eliminated several studies using alternative vocabulary such as “Delphi method,” “Analytic Hierarchy Process,” or “Nominal Group Technique.” Restriction to articles available in English may also have eliminated key studies. It is not possible to quantify the impact of this; however, the included literature comes from a variety of disciplines using a range of expert elicitation methods.

Conclusions

Expert elicitation has been used in a range of disciplines to answer questions that are difficult or expensive to answer in other ways. This review has highlighted critical aspects and considerations related to the expert panel, the pre-elicitation material, the elicitation tool or method, question framing and analysis, and how different types of bias may occur and be prevented. This review further indicates a departure from a rigid protocol and toward a flexible elicitation tool that can be adapted to fit individual research needs. Thorough consideration of these themes and following the guiding principles for conducting expert elicitation has informed the development of a framework for future expert elicitations of enteric illness source attribution.

Footnotes

Acknowledgments

Funding was provided by the Public Health Agency of Canada. Several experts in the design and analysis of expert elicitations, specifically in the context of enteric illness transmission, were consulted and their input has helped to guide the content of this review.

Disclosure Statement

No competing financial interests exist.

References

Acosta

, Wu

, Forrest

. Fuzzy experts on recreational vessels, a risk modelling approach for marine invasions. Ecol Model, 2010; 221:850–863.

Arnell

, Tompkins

, Adger

. Eliciting information from experts on the likelihood of rapid climate change. Risk Anal, 2005; 25:1419–1431.

Ashton

. Combining the judgments of experts: How many and which ones?. Organ Behav Hum Decis Process, 1986; 38:405–414.

Aspinall

. A route to more tractable expert advice. Nature, 2010; 463:294–295.

Ayyub

. Elicitation of Expert Opinions for Uncertainty and Risks. Boca Raton, FL: CRC Press, 2001.

Baker

, Keisler

. Cellulosic biofuels: Expert views on prospects for advancement. Energy, 2011; 36:595–605.

Bartonova

. How can scientists bring research to use: The HENVINET experience. Environ Health, 2012; 11(Suppl 1):S2-069X-11-S1-S2.

Bouma

, Kuik

, Dekker

. Assessing the value of Earth Observation for managing coral reefs: An example from the Great Barrier Reef. Sci Total Environ, 2011; 409:4497–4503.

Brito

, Griffiths

, Ferguson

, Hopkin

, Mills

, Pederson

, Macneil

. A behavioral probabilistic risk assessment framework for managing autonomous underwater vehicle deployments. J Atmos Ocean Technol, 2012; 29:1689–1703.

10.

Cagno

, Caron

, Mancini

, Ruggeri

. Using AHP in determining the prior distributions on gas pipeline failures in a robust Bayesian approach. Reliab Eng Syst Saf, 2000; 67:275–284.

11.

Clemen

, Winkler

. Combining probability distributions from experts in risk analysis. Risk Anal, 1999; 19:187–203.

12.

Cooke

. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press, 1991.

13.

Cooke

, Goossens

LLHJ

. TU Delft expert judgment data base. Reliab Eng Syst Saf, 2008; 93:657–674.

14.

Cooke

, Wilson

, Tuomisto

, Morales

, Tainio

, Evans

. A probabilistic characterization of the relationship between fine particulate matter and mortality: Elicitation of European experts. Envir Sci Tech, 2007; 41:6598–6605.

15.

Coppersmith

, Youngs

, Sprecher

. Methodology and main results of seismic source characterization for the PEGASOS Project, Switzerland. Swiss J Geosci, 2009; 102:91–105.

16.

Cressey

, Lake

. Ranking Food Safety Risks: Development of NZFSA Policy 2004–2005. Client Report FW0563:23, 2005.

17.

Cross

, Rigby

, Edwards-Jones

. Eliciting expert opinion on the effectiveness and practicality of interventions in the farm and rural environment to reduce human exposure to Escherichia coli O157. Epidemiol Infect, 2012; 140:643–654.

18.

Davidson

, Ravel

, Nguyen

, Fazil

, Ruzante

. Food-specific attribution of selected gastrointestinal illnesses: Estimates from a Canadian expert elicitation survey. Foodborne Pathog Dis, 2011; 8:983–995.

19.

de Jong

, Wardekker

, van der Sluijs

. Assumptions in quantitative analyses of health risks of overhead power lines. Environ Sci Policy, 2012; 16:114–121.

20.

Delbecq

, van de Ven

. A group process model for problem identification and program planning. J Appl Behav Sci, 1971; 7:466–492.

21.

Donlan

, Wingfield

, Crowder

, Wilcox

. Using expert opinion surveys to rank threats to endangered species: A case study with sea turtles. Conserv Biol, 2010; 24:1586–1595.

22.

Doria

MdF

, Boyd

, Tompkins

, Adger

. Using expert elicitation to define successful adaptation to climate change. Environ Sci Policy, 2009; 12:810–819.

23.

Elmer

, Seifert

, Kreibich

, Thieken

. A Delphi method expert survey to derive standards for flood damage data collection. Risk Anal, 2010; 30:107–124.

24.

Evans

, Gray

, Sielken

Jr , Smith

, Valdez-Flores

, Graham

. Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency. Regul Toxicol Pharmacol, 1994; 20:15–36.

25.

Fauss

, Gorman

, Swami

. Using expert elicitation to prioritize resource allocation for risk identification for nanosilver. J Law Med Ethics, 2009; 37:770–780.

26.

Fitch

, Bernstein

, Aguilar

, Burnand

, LaCalle

, Lazaro

, van het Loo

, McDonnell

, Vader

, Kahan

. The RAND/UCLA Appropriateness Method User's Manual. Santa Monica, CA: The Rand Corporation, 2001; MR1269.

27.

Flander

, Dixon

, Mcbride

, Burgman

. Facilitated expert judgment of environmental risks: Acquiring and analysing imprecise data. Int J Risk Assess Manag, 2012; 16:199–212.

28.

Forman

, Gass

. The Analytic Hierarchy Process—An exposition. Oper Res, 2001; 49:469.

29.

Forsberg

, Braback

, Keune

, Kobernus

, von Krauss

, Yang

, Bartonova

. An expert assessment on climate change and health—with a European focus on lungs and allergies. Environ Health, 2012; 11(Suppl 1):S4.

30.

Garabed

, Perez

, Johnson

, Thurmond

. Use of expert opinion for animal disease decisions: An example of foot-and-mouth disease status designation. Prev Vet Med, 2009; 92:20–30.

31.

Goossens

, Harper

. Joint EC/USNRC expert judgement driven radiological protection uncertainty analysis. J Radiol Prot, 1998; 18:249–264.

32.

Gordon

, Gallo

. Structuring expert input for a knowledge-based approach to watershed condition assessment for the Northwest Forest Plan, USA. Environ Monit Assess, 2011; 172:643–661.

33.

Gosling

, Hart

, Mouat

, Sabirovic

, Scanlan

, Simmons

. Quantifying experts' uncertainty about the future cost of exotic diseases. Risk Anal, 2012; 32:881–893.

34.

Havelaar

, Galindo

, Kurowicka

, Cooke

. Attribution of foodborne pathogens using structured expert elicitation. Foodborne Pathog Dis, 2008; 5:649–659.

35.

Helmer-Hirschberg

. The Use of the Delphi Technique in Problems of Educational Innovations, P-3499. Santa Monica, CA: The RAND Corporation, 1966.

36.

Hetes

, Richmond

, Pekar

, The Expert Elicitation Task Force. US EPA Expert Elicitation Task Force White Paper. Washington, DC: The US Environmental Protection Agency, 2011; 2013.

37.

Hirsch

, Podur

, Janser

, McAlpine

, Martell

. Productivity of Ontario initial-attack fire crews: Results of an expert-judgement elicitation study. Can J Forest Res, 2004; 34:705–715.

38.

Hnottavange-Telleen

, Chabora

, Finley

, Greenberg

, Marsteller

. Risk management in a large-scale CO2 geosequestration pilot project, Illinois, USA. 10th International Conference on Greenhouse Gas Control Technologies, 2011; 4:4044–4051.

39.

Hoek

, Boogaard

, Knol

, de Hartog

, Slottje

, Ayres

, Borm

, Brunekreef

, Donaldson

, Forastiere

, Holgate

, Kreyling

, Nemery

, Pekkanen

, Stone

, Wichmann

, van der Sluijs

. Concentration response functions for ultrafine particles and all-cause mortality and hospital admissions: Results of a European expert panel elicitation. Environ Sci Technol, 2010; 44:476–482.

40.

Hoelzer

, Oliver

, Kohl

, Hollingsworth

, Wells

, Wiedmann

. Structured expert elicitation about Listeria monocytogenes cross-contamination in the environment of retail deli operations in the United States. Risk Anal, 2012; 32:1139–1156.

41.

Hoffmann

, Fischbeck

, Krupnick

, McWilliams

. Attributing foodborne illnesses to their food sources: Using large expert panels to capture variability in expert judgment. Washington, DC: Resources for the Future, 2006; Discussion Paper 06-17-REV.

42.

Hoffmann

, Fischbeck

, Krupnick

, Mcwilliams

. Elicitation from large, heterogeneous expert panels: Using multiple uncertainty measures to characterize information quality for decision analysis. Decision Anal, 2007; 4:91–109.

43.

Horst

, Dijkhuizen

, Huirne

RBM

, De Leeuw

. Introduction of contagious animal diseases into The Netherlands: Elicitation of expert opinions. Livest Prod Sci, 1998; 53:253–264.

44.

Kahneman

, Tversky

. On the psychology of prediction. Psychol Rev, 1973; 80:237–251.

45.

Kangas

, Alho

, Kolehmainen

, Mononen

. Analyzing consistency of experts' judgment—Case assessing forest biodiversity. Forest Sci, 1998; 44:610–617.

46.

Knol

, de Hartog

, Boogaard

, Slottje

, van der Sluijs

, Lebret

, Cassee

, Wardekker

, Ayres

, Borm

, Brunekreef

, Donaldson

, Forastiere

, Holgate

, Kreyling

, Nemery

, Pekkanen

, Stone

, Wichmann

, Hoek

. Expert elicitation on ultrafine particles: Likelihood of health effects and causal pathways. Part Fibre Toxicol, 2009; 6:19-8977-6-19.

47.

Knol

, Slottje

, van der Sluijs

, Lebret

. The use of expert elicitation in environmental health impact assessment: A seven step procedure. Environ Health, 2010; 9:19-069X-9-19.

48.

Koehler

. The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges. Behav Brain Sci, 1996; 19:1–53.

49.

Kuhnert

, Martin

, Griffiths

. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecol Lett, 2010; 13:900–914.

50.

Kynn

. The ‘heuristics and biases’ bias in expert elicitation. J Roy Stat Soc A Sta, 2008; 171:239–264.

51.

Leal

, Wordsworth

, Legood

, Blair

. Eliciting expert opinion for economic models: An applied example. Value Health, 2007; 10:195–203.

52.

Leon

, VazquezPolo

, Gonzalez

. Elicitation of expert opinion in benefit transfer of environmental goods. Environ Resour Econ, 2003; 26:199–210.

53.

Lieu

, Nguyen

, Ball

, Martin

. Health outcomes of interest for evaluation in the Post-Licensure Rapid Immunization Safety Monitoring Program. Vaccine, 2012; 30:2824–2830.

54.

Marcot

, Allen

, Morey

, Shively

, White

. An expert panel approach to assessing potential effects of bull trout reintroduction on federally listed salmonids in the Clackamas River, Oregon. N Am J Fish Manage, 2012; 32:450–465.

55.

Mcdaniels

, Mills

, Gregory

, Ohlson

. Using expert judgments to explore robust alternatives for forest management under climate change. Risk Anal, 2012; 32:2098–2112.

56.

Moher

, Liberati

, Tetzlaff

, Altman

, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLOS Med, 2009; 6:e1000097.

57.

Montangero

, Belevi

. Assessing nutrient flows in septic tanks by eliciting expert judgement: A promising method in the context of developing countries. Water Res, 2007; 41:1052–1064.

58.

Moon

, Kang

. Use of fuzzy set theory in the aggregation of expert judgments. Ann Nucl Energy, 1999; 26:461–469.

59.

Morgan

, Henrion

. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge, UK: Cambridge University Press, 1992.

60.

Norrington

, Quigley

, Russell

, van der Meer

. Modelling the reliability of search and rescue operations with Bayesian Belief Networks. Reliab Eng Syst Saf, 2008; 93:940–949.

61.

O'Hagan

. Probabilistic uncertainty specification: Overview, elaboration techniques and their application to a mechanistic model of carbon flux. Environ Modell Softw, 2012; 36:35–48.

62.

Page

, Heathwaite

, Thompson

, Pope

, Willows

. Eliciting fuzzy distributions from experts for ranking conceptual risk model components. Environ Modell Softw, 2012; 36:19–34.

63.

Pires

. Assessing the applicability of currently available methods for attributing foodborne disease to sources, including food and food commodities. Foodborne Pathog Dis, 2013; 10:206–213.

64.

Pires

, Evers

, van Pelt

, Ayers

, Scallan

, Angulo

, Havelaar

, Hald

, Med-Vet-Net Workpackage 28 Working Group. Attributing the human disease burden of foodborne infections to specific sources. Foodborne Pathog Dis, 2009; 6:417–424.

65.

Podofillini

, Dang

, Zio

, Baraldi

, Librizzi

. Using expert models in human reliability analysis—A dependence assessment method based on fuzzy logic. Risk Anal, 2010; 30:1277–1297.

66.

Ravel

, Davidson

, Ruzante

, Fazil

. Foodborne proportion of gastrointestinal illness: Estimates from a Canadian expert elicitation survey. Foodborne Pathog Dis, 2010; 7:1463–1472.

67.

Ravnum

, Zimmer

, Keune

, Gutleb

, Murk

, Koppe

, Magnanti

, Lyche

, Eriksen

, Ropstad

, Skaare

, Kobernus

, Yang

, Bartonova

, Krayer von Krauss

. Policy relevant results from an expert elicitation on the human health risks of decabromodiphenyl ether (decaBDE) and hexabromocyclododecane (HBCD). Environ Health, 2012; 11(Suppl 1):S7-069X-11-S1-S7.

68.

Roman

, Hammitt

, Walsh

, Stieb

. Expert elicitation of the value per statistical life in an air pollution context. Risk Anal, 2012; 32:2133–2151.

69.

Santori

, Valente

, Cambiaso

, Ghirelli

, Gianelli Castiglione

, Valente

. Preliminary results of an expert-opinion elicitation process to prioritize an informative system funded by Italian ministry of health for cadaveric donor management, organ allocation, and transplantation activity. Transplant Proc, 2004; 36:433–434.

70.

Saunders

, Magnanti

, Correia Carreira

, Yang

, Alamo-Hernandez

, Riojas-Rodriguez

, Calamandrei

, Koppe

, Krayer von Krauss

, Keune

, Bartonova

. Chlorpyrifos and neurodevelopmental effects: A literature review and expert elicitation on research and policy. Environ Health, 2012; 11(Suppl 1):S5-069X-11-S1-S5.

71.

Speirs-Bridge

, Fidler

, McBride

, Flander

, Cumming

, Burgman

. Reducing overconfidence in the interval judgments of experts. Risk Anal, 2010; 30:512–523.

72.

Stiber

, Pantazidou

, Small

. Expert system methodology for evaluating reductive dechlorination at TCE sites. Environ Sci Technol, 1999; 33:3012–3020.

73.

Tan

, Baxter

, Newell

, Smye

, Dear

, Brownlee

, Darling

. Knowledge elicitation for validation of a neonatal ventilation expert system utilising modified Delphi and focus group techniques. Int J Hum-Comput St, 2010; 68:344–354.

74.

Thomas

, Murray

, Flockhart

, Pintar

, Pollari

, Fazil

, Nesbitt

, Marshall

. Estimates of the burden of foodborne illness in Canada for 30 specified pathogens and unspecified agents, circa 2006. Foodborne Pathog Dis, 2013; 10:639–648.

75.

Tuomisto

, Wilson

, Evans

, Tainio

. Uncertainty in mortality response to airborne fine particulate matter: Combining European air pollution experts. Reliab Eng Syst Saf, 2008; 93:732–744.

76.

Tversky

, Kahneman

. Judgment under uncertainty: Heuristics and biases. Science, 1974; 185:1124–1131.

77.

Tyshenko

, ElSaadany

, Oraby

, Darshan

, Aspinall

, Catford

, Krewski

. Expert elicitation and probabilistic inversion for the judgment of prion disease risk uncertainties using the classical model, EXCALIBUR and UNIBALANCE. Prion, 2010; 4:165–166.

78.

Tyshenko

, ElSaadany

, Oraby

, Darshan

, Aspinall

, Cooke

, Catford

, Krewski

. Expert elicitation for the judgment of prion disease risk uncertainties. J Toxicol Environ Health A, 2011; 74:261–285.

79.

Vally

, Glass

, Ford

, Hall

, Kirk

, Shadbolt

, Veitch

, Fullerton

, Mustro

, Becker

. Proportion of illness acquired by foodborne transmission for nine enteric pathogens in Australia: An expert elicitation. Foodborne Pathog Dis, 2014; 11:727–733.

80.

van der Fels-Klerx

, Horst

, Dijkhuizen

. Risk factors for bovine respiratory disease in dairy youngstock in The Netherlands: The perception of experts. Livest Prod Sci, 2000; 66:35–46.

81.

van der Fels-Klerx

, Kandhai

, Brynestad

, Dreyer

, Borjesson

, Martins

, Uiterwijk

, Morrison

, Booij

CJH

. Development of a European system for identification of emerging mycotoxins in wheat supply chains. World Mycotoxin J, 2009; 2:119–127.

82.

van der Fels-Klerx

, Goossens

, Saatkamp

, Horst

. Elicitation of quantitative data from a heterogeneous expert panel: Formal process and application in animal health. Risk Anal, 2002; 22:67–81.

83.

van Lenthe

. ELI: An interactive elicitation technique for subjective probability distributions. Organ Behav Hum Decis Process, 1993; 55:379–413.

84.

Walker

, Catalano

, Hammitt

, Evans

. Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene. J Expo Anal Environ Epidemiol, 2003; 13:1–16.

85.

Walker

, Evans

, MacIntosh

. Use of expert judgment in exposure assessment part I. Characterization of personal exposure to benzene. J Expo Anal Environ Epidemiol, 2001; 11:308–322.

86.

Wardekker

, De Jong

, Van Bree

, Turkenburg

, Van Der Sluijs

. Health risks of climate change: An assessment of uncertainties and its implications for adaptation policies. Environ Health, 2012; 11:67.

87.

[WHO] World Health Organization. The Global Burden of Disease: 2004 Update. Geneva, Switzerland: The World Health Organization, 2008.

88.

Winkler

, Clemen

. Multiple experts vs. multiple methods: Combining correlation assessments. Decision Anal, 2004; 1:167–176.

89.

Wreathall

, Bley

, Roth

, Multer

, Raslear

. Using an integrated process of data and modeling in HRA. Reliab Eng Syst Safe, 2004; 83:221–228.

90.

Zimmer

, Gutleb

, Ravnum

, Krayer von Krauss

, Murk

, Ropstad

, Skaare

, Eriksen

, Lyche

, Koppe

, Magnanti

, Yang

, Bartonova

, Keune

. Policy relevant results from an expert elicitation on the health risks of phthalates. Environ Health, 2012; 11(Suppl 1):S6-069X-11-S1-S6.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB