Testing the reliability of experts should be a key element of expert interviews.
Using the Condorcet Jury Theorem, it is shown that expert reliability can provide an
indication of the validity of expert-opinion data. The theoretical framework is
applied to expert-interview data collected in the Domestic Structures and European
Integration (DOSEI) project. Special attention is paid to the role of
‘leading’ experts and salient issues. Evaluating the DOSEI data,
the main findings are that (i) with some exceptions, there are acceptable levels of
inter-expert agreement, (ii) whether the leading expert is included or not does not
make a large difference to expert agreement, and (iii) experts are more in agreement
on salient issues.
Arregui, Javier
,
Frans N. Stokman
and
Robert Thomson
(2004)
‘Bargaining in the European Union and Shifts in Actors’
Policy Position’
, European Union Politics5(1):
47-72
.
2.
Austin-Smith, David
and
Jeffrey S. Banks
(1996)
‘Information Aggregation, Rationality, and the Condorcet Jury Theorem’
, American Political Science Review90(1):
34-45
.
3.
Bailer, Stefanie
(2004)
‘Bargaining Success in the European Union: The Impact of
Exogenous and Endogenous Power Resources’
, European Union Politics5(1):
99-123
.
4.
Ben-Yashar, Ruth
and
Jacob Paroush
(2000)
‘A Nonasymptotic Condorcet Jury Theorem’
, Social Choice and Welfare17:
189-199
.
5.
Black, Duncan
(1958) The Theory of Committees and Elections.
Cambridge: Cambridge University Press
.
6.
Bueno de Mesquita, Bruce
and
Frans N. Stokman
(eds) (1994) European Community Decision Making: Models,
Applications, and Comparisons.
New Haven, CT: Yale University Press
.
7.
Burnham, Peter
,
Karin Gilland
,
Wyn Grant
and
Zig Layton-Henry
(2004) Research Methods in Politics.
Houndmills: Palgrave
.
8.
Carletta, Jean
(1996)
‘Assessing Agreement on Classification Tasks: The Kappa Statistic’
, Computational Linguistics22(2):
249-254
.
9.
Cohen, Jacob
(1960)
‘A Coefficient of Agreement for Nominal Scales’
, Educational and Psychological Measurement20:
37-46
.
10.
Condorcet, Marquis de, M.J.A.N.de Caritat
(1785) ‘An Essay on the Application of Analysis
to the Probability of Decisions Rendered by a Plurality of
Votes’; in Classics of Social Choice, ed. and transl.
Iain McLean
and
Arnold B. Urken
, pp. 91-112.
Ann Arbor: University of Michigan Press
, 1995.
11.
Devine, Fiona
(2002) ‘Qualitative Methods’,
in
David Marsh
and
Gerry Stoker
(eds) Theory and Methods in Political Science, 2nd edn,
pp. 197-215.
Houndmills: Palgrave
.
12.
Di Eugenio, Barbara
and
Michael Glass
(2004)
‘The Kappa Statistic: A Second Look’
, Computational Linguistics30(1):
95-101
.
13.
Dyson, Kenneth
and
Kevin Featherstone
(1999) The Road to Maastricht: Negotiating Economic and
Monetary Union.
Oxford: Oxford University Press
.
Fleiss, Joseph L.
(1971)
‘Measuring Nominal Scale Agreement among Many Raters’
, Psychological Bulletin76:
378-382
.
16.
Grofman, Bernard
,
Guillermo Owen
and
Scott L. Feld
(1983)
‘Thirteen Theorems in Search of the Truth’
, Theory and Decision15:
261-278
.
17.
Holsti, Ole R.
(1969) Content Analysis for the Social Sciences and
Humanities.
Reading, MA: Addison-Wesley
.
18.
Hooghe, Liesbet
(1999)
‘Supranational Activists or Intergovernmental Agents? Explaining
the Orientations of Senior Commission Officials toward European Integration’
, Comparative Political Studies32(4):
435-463
.
19.
Hughes, Marie A.
and
Dennis E. Garrett
(1990)
‘Intercoder Reliability Estimation-Approaches in Marketing: A
Generalizability Theory Framework for Quantitative Data’
, Journal of Marketing Research27:
185-195
.
20.
King, Gary
,
Robert O. Keohane
and
Sidney Verba
(1994) Designing Social Inquiry. Scientific Inference in
Qualitative Research.
Princeton, NJ: Princeton University Press
.
21.
Kolbe, Richard H.
and
Melissa S. Burnett
(1991)
‘Content-Analysis Research: An Examination of Application with
Directives for Improving Research Reliability and Objectivity’
, Journal of Consumer Research18:
243-250
.
22.
Krippendorff, Klaus
(2004a)
‘Reliability in Content Analysis: Some Common Misconceptions and Recommendations’
, Human Communication Research30(3):
411-433
.
23.
Krippendorff, Klaus
(2004b) Content Analysis: An Introduction to Its
Methodology, 2nd edn.Thousand Oaks, CA: Sage
.
24.
Lombard, Matthew
,
Jennifer Snyder-Duch
and
Cheryl Campanella Bracken
(2004)
‘Content Analysis in Mass Communication: Assessment and Reporting
of Intercoder Reliability’
, Human Communication Research28(4):
587-604
.
25.
Moravcsik, Andrew
(1998) The Choice for Europe: Social Purpose and State Power
from Messina to Maastricht.
London: UCL Press
.
26.
Neuendorf, Kimberly A.
(2002) The Content Analysis Guidebook.
Thousand Oaks, CA: Sage
.
27.
Nitzan, Shmuel
and
Jacob Paroush
(1982)
‘Optimal Decision Rules in Uncertain Dichotomous Choice Situation’
, International Economic Review23:
289-297
.
28.
Nurmi, Hannu
(2002) Voting Procedures under Uncertainty.
Berlin: Springer
.
29.
Owen, Guillermo
,
Bernard Grofman
and
Scott L. Feld
(1989)
‘Proving a Distribution-Free Generalization of the Condorcet Jury Theorem’
, Mathematical Social Sciences17:
1-16
.
30.
Richards, David
(1996)
‘Elite Interviewing: Approaches and Pitfalls’
, Politics16(3):
199-204
.
31.
Riffe, Daniel
and
Alan A. Freitag
(1997)
‘A Content Analysis of Content Analyses: Twenty-five Years of
Journalism & Mass Communication Quarterly’
, Journalism & Mass Communication Quarterly74(4):
873-882
.
32.
Scott, William A.
(1955)
‘Reliability of Content Analysis: The Case of Nominal Scale Coding’
, Public Opinion Quarterly17:
321-325
.
33.
Siegel, Sidney
and
N. John Castellan Jr
(1988) Nonparametric Statistics for the Behavioral
Sciences, 2nd edn.New York: McGraw-Hill
.
