Best Practice Recommendations for Designing and Implementing Experimental Vignette Methodology Studies

Abstract

We describe experimental vignette methodology (EVM) as a way to address the dilemma of conducting experimental research that results in high levels of confidence regarding internal validity but is challenged by threats to external validity versus conducting nonexperimental research that usually maximizes external validity but whose conclusions are ambiguous regarding causal relationships. EVM studies consist of presenting participants with carefully constructed and realistic scenarios to assess dependent variables including intentions, attitudes, and behaviors, thereby enhancing experimental realism and also allowing researchers to manipulate and control independent variables. We describe two major types of EVM aimed at assessing explicit (i.e., paper people studies) and implicit (i.e., policy capturing and conjoint analysis) processes and outcomes. We offer best practice recommendations regarding the design and implementation of EVM studies based on a multidisciplinary literature review, discuss substantive domains and topics that can benefit from implementing EVM, address knowledge gaps regarding EVM such as the need to increase realism and the number and diversity of participants, and address ways to overcome some of the negative perceptions about EVM by pointing to exemplary articles that have used EVM successfully.

Keywords

research design experimental design quasi-experimental design

Understanding the direction and nature of causal relationships is the cornerstone of science (Shadish, Cook, & Campbell, 2002). While the majority of management research provides evidence regarding covariation between antecedent and outcome variables, covariation alone does not answer two important questions crucial for establishing causality: (a) Did the antecedent occur temporally before the effect, and (b) have other alternative explanations for covariation been ruled out (Aguinis & Vandenberg, 2014)? A primary reason for a lack of clear answers to these questions is that much of management research consists of passive observation designs and is cross-sectional (Aguinis & Edwards, 2014; P. M. Podsakoff & Dalton, 1987). In contrast, understanding causal relationships requires the use of experimental or quasi-experimental designs (Grant & Wall, 2009; Spector, 1981).

There have been several calls regarding the need to implement research designs that improve our knowledge about causal relationships (e.g., Allen, Hancock, Vardaman, & McKee, 2014; Casper, Eby, Bordeaux, Lockwood, & Lambert, 2007; Miller & Tsang, 2011; N. P. Podsakoff, Podsakoff, Mackenzie, Maynes, & Spoelma, 2014; Shepherd, 2011; Uy, Foo, & Aguinis, 2010). However, literature reviews have documented that there is only a small minority of articles published in management and related fields that have used those designs (P. M. Podsakoff & Dalton, 1987; Scandura & Williams, 2000). The paucity of experimental research is understandable given the associated practical and logistical constraints. Specifically, it takes more time and effort to conduct an experiment, involving the creation of experimental materials, recruiting of participants, administering the experimental treatments, and addressing ethical challenges in administering the treatments compared to using archival data or an online survey (Aguinis & Lawal, 2012; Grant & Wall, 2009; Highhouse, 2009; Sniderman & Grob, 1996). In addition, a concern with experimental designs is that they usually sacrifice external validity and generalizability to enhance internal validity (Argyris, 1975; Scandura & Williams, 2000). For example, experimental designs often involve participants such as students or individuals who are not students but are removed from their natural environments. Thus, researchers seem to face a seemingly inescapable dilemma: (a) implement experimental designs that yield high levels of confidence regarding internal validity but are challenged by difficulties regarding external validity (i.e., uncertainty regarding generalizability of results) or (b) implement nonexperimental designs that often maximize external validity because they are conducted in natural settings but whose conclusions are ambiguous in terms of the direction and nature of causal relationships.

The goal of our article is to discuss experimental vignette methodology (EVM) as a way to address the aforementioned dilemma. EVM consists of presenting participants with carefully constructed and realistic scenarios to assess dependent variables including intentions, attitudes, and behaviors. Thus, EVM enhances experimental realism and also allows researchers to manipulate and control independent variables, thereby simultaneously enhancing both internal and external validity (Atzmüller & Steiner, 2010; Hox, Kreft, & Hermkens, 1991). We readily acknowledge the existence of several important sources that have addressed one or more aspects of EVM (e.g., Aiman-Smith, Scullen, & Barr, 2002; Graham & Cable, 2001; Hughes & Huby, 2002; Ludwick & Zeller, 2001; Shepherd & Zacharakis, 1999; Wason, Polonsky, & Hyman, 2002; Wilks, 2004). Our article relies on the foundational knowledge accumulated in those and other sources but also goes beyond by making the following unique value-added contributions. First, it offers a comprehensive treatment regarding how to design and implement various types of EVM studies, including offering best practice recommendations distilled from a multidisciplinary literature review of methodological and substantive sources. In other words, our article offers a “one-stop shopping” source for researchers contemplating the possibility of conducting an EVM study; in addition, we hope that this aspect of our article will also make it attractive for instructors of methods courses. Second, our article discusses specific areas and topics that can benefit from implementing EVM. As such, it makes a contribution to specific substantive domains that may be advanced by the use of EVM. Third, our article addresses knowledge gaps regarding EVM such as the need for technological advances that would lead to increased realism and improvements regarding the need to augment the number and diversity of participants in EVM studies. Fourth, our article also addresses ways to overcome some of the negative perceptions—which may have reached the level of stigma—that are associated with EVM. We do so by pointing to exemplary articles published in highly influential journals that have used EVM successfully.

Our article is organized as follows. First, we define and describe EVM and illustrate how it is useful in terms of advancing theories in management research. Second, we describe results of a literature search based on 30 management-related journals to assess the prevalence and use of EVM in the past 20 years. Third, we provide best practice recommendations for the future use of EVM. These recommendations focus on 10 decision points, together with trade-offs for each, involved in the three major steps of any EVM study: planning, implementation, and reporting of results. Finally, we offer recommendations for specific research domains and topics that would benefit from using EVM as well as future directions regarding research that would lead to improvements regarding EVM itself.

Experimental Vignette Methodology

While there are many definitions of what constitutes a vignette, we use the definition offered recently by Atzmüller and Steiner (2010): “a short, carefully constructed description of a person, object, or situation, representing a systematic combination of characteristics” (p. 128). An important characteristic of EVM is that it is not restricted to being presented solely in written format but can include images, videos, and other media (Hughes & Huby, 2002). As we will describe next, EVM includes two major types: those assessing explicit (i.e., paper people studies) and those assessing implicit (i.e., policy capturing and conjoint analysis) processes and outcomes. Before we describe these two types in more detail, we offer two illustrations of the successful use of EVM. We refer to these studies as successful not only because of their substantive contributions to knowledge regarding causal relationships but also because they have been published in highly visible and influential journals.

First, Sauer (2011) examined the causal effects of leadership status and style on team members’ perceptions of leadership effectiveness and team performance. EVM was a suitable methodological approach because it allowed for experimental control over the manipulated antecedents. Moreover, because the outcome variables were explicit (i.e., self-reported perceptions of leadership effectiveness, leader’s self-confidence), this study involved the use of the paper people type of EVM. Sauer implemented an EVM study involving video vignettes, thereby increasing experimental realism, and in addition, participants completed the experimental task online from their own environments. Also, participants were recruited through a university’s alumni database, resulting in a sample that included over 80% non-students. Because of the control provided by using EVM, Sauer gathered causal evidence regarding the interactive effect of new leader status and leadership style on perceptions of leader effectiveness and team performance.

As a second illustration, McKelvie, Haynie, and Gustavsson (2011) addressed the impact of uncertainty in the decision-making processes of entrepreneurs. They used the policy capturing/conjoint analysis type of EVM because they were interested in assessing implicit decision-making processes. This study involved a sample of new product development decision makers, and written vignettes were presented to them online. By manipulating independent variables involved in various types of opportunities, McKelvie et al. were able to gather evidence regarding which type of uncertainty had an effect on whether entrepreneurs choose to exploit or not to exploit opportunities.

The aforementioned examples illustrate that using EVM can lead to important insights and knowledge about causal relationships. Next, we describe the two types of EVM studies that are most prevalent in management and related fields: paper people studies and policy capturing/conjoint analysis. We also describe additional exemplars of each based on articles published in influential journals to illustrate that it is certainly possible to publish an article using EVM in these and other highly respected outlets.

Paper People Studies

Paper people studies consist of presenting participants with vignettes typically in written form (and hence their name) and then asking participants to make explicit decisions, judgments, and choices or express behavioral preferences. This type of EVM has existed for many decades and has been used extensively, especially in fields such as business ethics (Alexander & Becker, 1978; Finch, 1987; Hyman & Steiner, 1996; Weber, 1992).

While this type of EVM has been popular in ethical decision-making contexts, it has also been applied in other areas as well. For example, Pierce, Aguinis, and Adams (2000) conducted a paper people study in which employees in a law enforcement agency read scenarios involving workplace romances that resulted in sexual harassment allegations. Results of this study provided evidence about the causal effects of type of workplace romance on subsequent attitudes about the romance participants. As a more recent illustration, Raaijmakers, Vermeulen, Meeus, and Zietsma (in press) conducted a paper people study that allowed them to assess institutional compliance with a new law given institutional complexities. In this study, Raaijmakers et al. presented managers information about a new law requiring changes at the child care facilities. After reading this information, which set the stage for the experiment, participants were randomly assigned to read a vignette that manipulated two independent variables. Participants were then asked to respond to a short questionnaire asking when they would implement the changes required by the law, given that the government would check on their progress five years after the law was passed. Results of this EVM study led to novel insights regarding legal compliance and adoption techniques.

Policy Capturing and Conjoint Analysis Studies

Policy capturing and conjoint analysis studies present respondents with scenarios containing carefully manipulated variables (Carroll & Johnson, 1990); however, in contrast to paper people studies, participants are asked to make decisions between scenarios in order to capture implicit processes (Aiman-Smith et al., 2002). In other words, in policy capturing and conjoint analysis studies, the goal is to understand the effects of the manipulated variables on implicit judgments through ranking of vignettes or by asking participants to make choices and state preferences between them. Therefore, the specific purpose of this type of EVM is to assess participants’ choices by capturing real-time processes and decisions—which are often not made openly and with the participants’ full awareness. While policy capturing and conjoint analysis studies are often discussed separately (Priem & Harrison, 1994) and researchers in some fields use one or another label, policy capturing and conjoint analysis methods are virtually the same (Aiman-Smith et al., 2002). Therefore, we treat policy capturing studies and conjoint analyses as a single type of EVM.

As a recently published example, Skarlicki and Turner (2014) conducted a policy capturing study in which participants were presented with the task of rating potential applicants in terms of their desirability after they read scenarios depicting 32 unique applicants. In each of these scenarios, the researchers manipulated five distinct independent variables with two levels each. This type of research would be difficult to conduct without the availability of EVM. By carefully manipulating the scenarios, Skarlicki and Turner were able to ascertain specific causal antecedents of assessments of unfairness.

As an additional example, Shepherd, Patzelt, and Baron (2013) conducted a conjoint analysis study to address a sensitive topic. In their study, they experimentally investigated moral disengagement in the decision-making process of entrepreneurs. They noted that this is a rather sensitive issue that can be difficult to investigate; however, conjoint analysis provided an excellent opportunity to assess real-time decision-making processes. Participants were presented with 25 profiles that manipulated the possible opportunity available to the entrepreneur. By using conjoint analysis, Shepherd et al. were able to investigate the decision-making processes of entrepreneurs specifically as they relate to the impact of their decisions on the environment.

Next, we describe results of a literature review, the goal of which was to learn about the prevalence of EVM in management and related fields. Our review allowed us to uncover subfields and journals that are most receptive to publishing EVM studies.

Experimental Vignette Methodology Study: Literature Review

Method

Our review involved the same 30 management-related journals identified by P. M. Podsakoff, MacKenzie, Podsakoff, and Bachrach (2008) because they include influential journals across major domains. We covered the period from 1994 through 2013 and conducted a search with Google Scholar using the keywords vignette study, conjoint analysis, paper people, scenario, policy capturing, and scenario study. While many of the articles identified by our search involved the use of stimuli presented using paper and/or written materials, we did not limit our review to those; rather, we also identified instances where vignettes were presented through other media (e.g., audio, video).

Results and Discussion

Results in Table 1 show that we identified 328 articles that used EVM. Also, this table shows that the paper people type of EVM has been used more frequently than policy capturing and conjoint analysis. The dominance of paper people studies over time is documented by results displayed in Figure 1.

Figure 1.

Number of experimental vignette methodology (EVM) articles published in 30 management-related journals by year classified by type: (a) paper people and (b) policy capturing/conjoint analysis (PC/CA) studies. Total: Sum of articles using the paper people and PC/CA types of EVM.

Table 1.

Summary of Results of Literature Review of Frequency of Use of Experimental Vignette Methodology (EVM) in 30 Management-Related Journals (1994-2013).

Journal	Paper People	PC/CA	Total
1. Academy of Management Journal	11	5	16
2. Academy of Management Review	0	0	0
3. Administrative Science Quarterly	3	0	3
4. California Management Review	0	1	1
5. Decision Sciences	2	7	9
6. Group & Organization Management	5	0	5
7. Harvard Business Review	0	0	0
8. Human Relations	4	1	5
9. Human Resource Management	5	2	7
10. Industrial & Labor Relations Review	2	0	2
11. Industrial Relations Journal	0	0	0
12. Journal of Applied Psychology	33	8	41
13. Journal of Business Research	12	27	39
14. Journal of Business Venturing	1	18	19
15. Journal of Human Resources	0	0	0
16. Journal of International Business Studies	5	2	7
17. Journal of Management	9	5	14
18. Journal of Management Studies	2	6	8
19. Journal of Occupational and Organizational Psychology	7	5	12
20. Journal of Organizational Behavior	13	3	16
21. Journal of Vocational Behavior	3	3	6
22. Leadership Quarterly	20	0	20
23. Management Science	3	6	9
24. Monthly Labor Review	0	0	0
25. Organization Science	8	4	12
26. Organizational Behavior & Human Decision Processes	41	18	59
27. Organizational Research Methods	0	0	0
28. Personnel Psychology	9	2	11
29. Sloan Management Review	0	0	0
30. Strategic Management Journal	2	5	7
Total	200	128	328

Note: PC/CA = policy capturing/conjoint analysis.

Overall, in spite of the modest increase in the number of articles using EVM over the past 20 years, as shown by the solid line indicating the total number of articles in Figure 1, EVM is clearly not a very popular methodological approach. Specifically, based on results reported by Kruschke, Aguinis, and Joo (2012, Table 1), the total number of articles published in the 30 journals included in our review during the 20-year period is approximately 30,000. So, only about 1% of articles have used EVM during this period. But, this is not necessarily an unexpected or particularly surprising result. On the other hand, a more interesting and novel insight based on results in Table 1 relates to which journals and fields are most receptive to studies using EVM—and which particular type. Our results show that the majority of paper people studies were published in organizational behavior and human resource management (OB/HRM) journals such as Organizational Behavior and Human Decision Processes (OBHDP), Journal of Applied Psychology (JAP), Leadership Quarterly, and Journal of Organizational Behavior. In fact, OBHDP and JAP account for nearly 43% of all EVM articles published in the past 20 years. In contrast, conjoint analysis studies have received more attention in the Journal of Business Venturing (JBV), a flagship publication in entrepreneurship.

We can only speculate on the reasons why EVM is used so infrequently in management research—at least in the 30 journals included in our review. One reason could be lack of knowledge on how to design and execute such studies. Thus, perhaps the existence of a source that summarizes best practices based on the knowledge accumulated thus far can address this issue. Accordingly, in our article’s next section we describe best practice recommendations for designing and implementing EVM studies. A second reason is that there may be apprehension to using EVM, as reflected by results reported by Shook, Ketchen, Cycyota, and Crockett (2003), who noted that strategic management doctoral candidates felt “little confidence” in using policy capturing and similar methods associated with experimental research. This apprehension may be due to some of the practical and logistical challenges we mentioned earlier—combined with a belief that it is particularly difficult to publish EVM studies in top-tier journals. Our review shows that yes, EVM studies are not published frequently. However, our review also shows that all of the most prestigious journals in OB/HRM, strategic management studies, and entrepreneurship have published some. So, it is possible to publish EVM studies in top-tier journals.

Best Practice Recommendations for Designing and Implementing EVM Studies

Table 2 includes a list of sources addressing one or more aspects of EVM. As shown in Table 2, these articles have been published in journals across diverse fields such as nursing (Hughes & Huby, 2002) and marketing (Joseph & Chandrasekaran, 2013). In addition, most reviews have focused on a particular aspect of EVM such as design (Alexander & Becker, 1978) or analysis (Hox et al., 1991). It is from this comprehensive list of sources that we were able to distill best practice recommendations to guide future research using EVM.

Table 2.

List of Sources Addressing Experimental Vignette Methodology (EVM).

Type of EVM	Authors	Article’s Title
Paper people	Alexander and Becker (1978)	“The Use of Vignettes in Survey Research”
	Atzmüller and Steiner (2010)	“Experimental Vignette Studies in Survey Research”
	Burstin, Doughtie, and Raphaeli (1980)	“Contrastive Vignette Technique: An Indirect Methodology Designed to Address Reactive Social Attitude Measurement”
	Cavanaugh and Fritzsche (1985)	“Using Vignettes in Business Ethics Research”
	Finch (1987)	“The Vignette Technique in Survey Research”
	Ganong and Coleman (2006)	“Multiple Segment Factorial Vignette Designs”
	Hox, Kreft, and Hermkens (1991)	“The Analysis of Factorial Surveys”
	Hughes and Huby (2002)	“The Application of Vignettes in Social and Nursing Research”
	Hyman and Steiner (1996)	“The Vignette Method in Business Ethics Research: Current Uses, Limitations and Recommendations”
	Jasso (2006)	“Factorial Survey Methods for Studying Beliefs and Judgments”
	Ludwick and Zeller (2001)	“The Factorial Survey: An Experimental Method to Replicate Real-World Problems”
	Murphy, Herr, Lockhart, and Maguire (1986)	“Evaluating the Performance of Paper People”
	Sniderman and Grob (1996)	“Innovations in Experimental Design in Attitude Surveys”
	Wason, Polonsky, and Hyman (2002)	“Designing Vignettes Studies in Marketing”
	Weber (1992)	“Scenarios in Business Ethics Research: Review, Critical Assessment and Recommendations”
	Wilks (2004)	“The Use of Vignettes in Qualitative Research Into Social Work Values”
	Woehr and Lance (1991)	“Paper People Versus Direct Observation: An Empirical Examination of Laboratory Methodologies”
Policy capturing and conjoint analysis	Aiman-Smith, Scullen, and Barr (2002)	“Conducting Studies of Decision Making in Organizational Contexts: A Tutorial for Policy-Capturing and Other Regression-Based Techniques”
	Graham and Cable (2001)	“Consideration of the Incomplete Block Design for Policy-Capturing Research”
	Green (1984)	“Hybrid Models for Conjoint Analysis: An Expository Review”
	Green, Krieger, and Wind (2001)	“Thirty Years of Conjoint Analysis: Reflections and Prospects”
	Hobson and Gibson (1983)	“Policy Capturing as an Approach to Understanding and Improving Performance Appraisal: A Review of the Literature”
	Joseph and Chandrasekaran (2013)	“Conjoint Analysis: A Perfect Link Between Marketing and Product Design Functions—A Review”
	Karren and Barringer (2002)	“A Review and Analysis of the Policy-Capturing Methodology in Organizational Research: Guidelines for Research and Practice”
	Lohrke, Holloway, and Woolley (2010)	“Conjoint Analysis in Entrepreneurship Research: A Review and Research Agenda”
	Louviere (1988)	“Conjoint Analysis Modelling of Stated Preferences: A Review of Theory, Methods, Recent Developments and External Validity”
	Priem and Harrison (1994)	“Exploring Strategic Judgment: Methods for Testing the Assumptions of Perspective Contingency Theories”
	Priem, Walters, and Li (2011)	“Decisions, Decisions! How Judgment Policy Studies Can Integrate Macro and Micro Domains in Management Research”
	Roehling (1993)	“‘Extracting’ Policy from Judicial Opinions: The Dangers of Policy Capturing in a Field Setting”
	Shepherd and Zacharakis (1999)	“Conjoint Analysis: A New Methodological Approach for Researching the Decision Policies of Venture Capitalists”
	Slaughter, Richard, and Martin (2006)	“Comparing the Efficacy of Policy-Capturing Weights and Direct Estimates for Predicting Job Choice”

Figure 2 lays out the structure of the remainder of the best practice recommendations section of our article, which addresses suggestions and trade-offs involved in each of 10 decision points associated with the planning, implementation, and reporting of results stages of EVM studies. Because our recommendations may differ based on a study’s goals and other contextual and practical issues, our discussion includes an analysis of trade-offs involved in the various decisions.

Figure 2.

Summary of steps and decision points in conducting an experimental vignette methodology study.

Planning an EVM Study

Decision Point 1: Deciding Whether EVM Is a Suitable Approach

While we argue that EVM is valuable, it is not always the most appropriate methodological approach. First, EVM is particularly useful when researchers need to exercise control of independent variables to gather evidence regarding causation (Cavanaugh & Fritzsche, 1985). EVM allows researchers to include factors that are relevant to the research question while excluding those that might confound the results. This amount of control helps to test causal hypotheses that would otherwise be difficult. Therefore, EVM is particularly useful in research domains in which variables are known to correlate but there is a need to determine the nature and direction of causal relationships.

Second, EVM is also an appropriate method when researchers are faced with ethical dilemmas associated with conducting experimental research. While it is difficult to experimentally manipulate sensitive topics in an ethical manner, EVM provides researchers with the ability to create hypothetical scenarios that address sensitive topics. As an example, it would be ethically impossible to manipulate workplace romances in an experiment, but by using EVM, researchers have been able to understand causal antecedents of perceptions and attributions of blame of former workplace romance participants when the romance turns into a sexual harassment allegation (e.g., Pierce et al., 2000). In short, EVM is a good choice when the goal is to investigate sensitive topics in an experimentally controlled way.

It is also important to recognize issues that may lead to the decision to not use EVM. Specifically, because EVM requires participants to respond to hypothetical scenarios, there are some situations when those scenarios do not create the same context as would be encountered in “real life” (Lohrke, Holloway, & Woolley, 2010). This can make it difficult to use EVM when certain contextual pressures are difficult to reproduce. For instance, in some high-stakes decision-making scenarios (e.g., mergers and acquisitions), presentation of hypothetical scenarios are not likely to produce the same responses as when those same situations occur in a natural setting.

Decision Point 2: Choosing the Type of EVM

The next major decision is to choose the type of EVM. As mentioned earlier, the two major options involve paper people and policy capturing/conjoint analysis. Although we referred to them earlier, next we offer a more detailed description as well as examples of published articles that implemented each.

Paper people studies

Paper people studies focus on explicit responses to hypothetical scenarios and have been used widely in a variety of research domains such as leadership (e.g., Benjamin & Flynn, 2006; De Cremer, Mayer, van Dijke, Schouten, & Bardes, 2009; De Cremer & Van Knippenberg, 2004; Yun, Faraj, & Sims, 2005), executive behaviors (e.g., Melone, 1994; Powell, 2001), entrepreneurship (e.g., Bucar, Glas, & Hisrich, 2003), organizational citizenship behavior (e.g., Eastman, 1994; N. P. Podsakoff, Whiting, Podsakoff, & Mishra, 2011), and ethics (e.g., Hoyt, Price, & Poatsy, 2013). This type of EVM is most appropriate when the goal is to assess explicit processes and outcomes—those about which participants are aware and on which they can provide information.

Policy capturing and conjoint analysis studies

Policy capturing and conjoint analysis studies capture implicit processes and outcomes (Priem & Harrison, 1994; Shepherd & Zacharakis, 1999). In both of these types of EVM, it is assumed that important factors that make up the individual’s decision process are known a priori (Priem & Harrison, 1994). For this reason, these techniques are most useful in areas where theory provides a clear understanding of factors that are likely to influence processes and outcomes. If an area is relatively new or conceptually underdeveloped, it may not be possible to know those factors (Lohrke et al., 2010), which makes using policy capturing and conjoint analysis less appropriate in those situations.

Additionally, policy capturing and conjoint analysis allow researchers to understand the decision-making process of a single individual (Priem & Harrison, 1994). Because of this, these techniques can be used to collect large amounts of information, even from a relatively small group of participants (Hughes & Huby, 2002). This is especially useful in situations when the population of interest is small and/or inaccessible (Priem & Harrison, 1994).

Decision Point 3: Choosing the Type of Research Design

The next decision involves choosing the type of research design. Choices include a between-person, within-person, or mixed research design (Atzmüller & Steiner, 2010).

Between-person designs require that each participant read only one vignette, and comparisons are made across participants (Atzmüller & Steiner, 2010). True between-person designs are uncommon, especially when judgments are used as the dependent variable (Atzmüller & Steiner, 2010). For instance, because participants are only presented with a single rather than multiple vignettes, they lose any chance at comparison that would help to ground responses contextually. Without other vignettes to serve as referent points for their own judgments, responses may not accurately reflect the true judgments of each respondent. For this reason, it is important that in between-person designs participants be provided with sufficient information to help provide as much context as possible. For instance, in their paper people EVM study, Raaijmakers et al. (in press) first presented all of their participants with baseline information (i.e., a general description of the issues involved) to provide a similar contextual background for all participants. Accordingly, the recommendation is to provide participants with adequate contextual background when using a between-subjects design or use the within-person or mixed design options.

When using a within-person design, each participant views the same set of vignettes, and comparisons are made between vignettes within the same person (Atzmüller & Steiner, 2010). This type of design helps to show the effects of a manipulation within one individual and is useful in terms of uncovering judgment processes of a single individual. In mixed designs, different groups of participants receive different sets of vignettes; however, within each group, participants see the same vignettes (Atzmüller & Steiner, 2010). Accordingly, because multiple respondents also offer responses regarding the same vignettes, comparisons can be made across respondents.

Decision Point 4: Choosing the Level of Immersion

One of the major criticisms regarding the use of EVM is that it is unrealistic and results are not easily generalizable (Hughes & Huby, 2002). In fact, EVM studies are criticized for only showing that certain outcomes can happen but not necessarily that they do happen outside of the experimental situation. Accordingly, researchers have made calls to improve the external validity of EVM by enhancing the level of realism present in the stimulus presentation (Hughes & Huby, 2002; Roehling, 1999; Woehr & Lance, 1991). Moreover, much like the training literature has demonstrated that transfer of training is improved by increasing the similarity between the training and job contexts (Aguinis & Kraiger, 2009), improving realism by increasing the similarity between the experimental and natural settings increases the observed effects (e.g., Taylor, 2006).

One recommendation to improve realism is to increase the level of immersion experienced by participants—the subjective experience of being personally immersed in the situation described in the vignette. Technological advances have provided the means to do so by changing the method of presentation of scenarios. For example, there is the possibility of using audio, video, pictures, and other presentation methods that increase the realism of EVM studies (Aiman-Smith et al., 2002; Goldman, Gutek, Stein, & Lewis, 2006; Green, Kriege, & Wind, 2001; Hughes & Huby, 2002). These types of presentation methods are aimed at engaging participants’ senses more fully. Indeed, virtual reality technology (VRT) has become the latest type of media used to present vignettes to participants. For instance, by presenting the vignettes as fully immersing scenarios through VRT, participants are able to see, hear, touch, smell, and fully experience information that would be impossible to implement in a more traditional written scenario (Pierce & Aguinis, 1997).

In our review of the literature, we found that fewer than 6% of the studies reported using video vignettes and the majority of those were paper people studies. Examples of such studies include Aguinis and Adams (1998), Newcombe and Ashkanasy (2002), and Pelletier (2012). However, video vignettes have also been used in conjoint analyses as well (e.g., Donoho, 2003).

Increasing the immersion of participants has several benefits. First, observing actual behavior is more likely to engage participants to a greater extent, allowing them to remember and recall important information (Hughes & Huby, 2002). Additionally, as mentioned earlier, more immersive techniques enhance experimental realism (Pierce & Aguinis, 1997). As Woehr and Lance (1991) showed, paper presentations provide fewer distractors in scenarios than when behavior is directly observed. By using a more immersive medium, scenarios become more lifelike in that they are likely to provide a greater amount of “natural noise” in each scenario. If the “noise” created by the distractors is controlled by the researcher, more lifelike scenarios could be created without compromising the internal validity of the experiment (Pierce & Aguinis, 1997). For instance, in their study observing voice behaviors using a video vignette, Whiting, Maynes, Podsakoff, and Podsakoff (2012) controlled presentation of one possible source of noise by using actors who were rated as equally attractive. This helped them rule out the effect that attractiveness could have on the results of the study. Additionally, greater immersion is likely to help increase external validity as the scenarios more closely approximate experiences in the real world (Heslin, Vandewalle, & Latham, 2006). Finally, using techniques that are higher on immersion also allows researchers to explore a variety of sensitive topics that are difficult or unethical to manipulate in the real world (Pierce & Aguinis, 1997).

While there are many benefits that accompany immersive presentation techniques, there are also some trade-offs that should be taken into account. As vignette studies become more immersive, the cost associated with the experiment typically increases as well. Creating a written vignette usually requires no more than a researcher’s time and creativity. But in a study using video vignettes in place of written vignettes, Taggar and Neubert (2004) had to hire five professional actors to role play the various scenarios for their participants.

Decision Point 5: Specifying the Number and Levels of the Manipulated Factors

Theory plays a crucial role in planning an EVM study because it is the driving force in choosing what factors are relevant to the research in question as well as the number of levels for each manipulated variable (Atzmüller & Steiner, 2010; Priem & Harrison, 1994). In fact, this is often one of the criticisms leveled against vignette methodologies (Lohrke et al., 2010): Because EVM requires prespecified variables and levels, the threat of omitting important variables, especially in complex decision-making processes, is high. As Shepherd and Zacharakis (1999) keenly noted, “the key is to identify what information is critical to the decision being studied” (p. 207). The issue of model misspecification is a concern in all empirical research, but it is particularly important in the context of EVM and experimental research in general given the criticism that that such research does not test “whether a hypothesis is true, but rather whether the experimenter is a sufficiently ingenious stage manager to produce in the laboratory conditions which demonstrate that an obviously true hypothesis is correct” (McGuire, 1973, p. 449).

There are two main approaches that can be used in choosing variables to include (Shepherd & Zacharakis, 1999). The first involves an attribute-driven design, in which factors are set up so that they are orthogonal to one another. This allows researchers to more easily assess the independent effects of each manipulated factor (Aiman-Smith et al., 2002; Karren & Barringer, 2002). However, presenting orthogonal cues has limitations. For example, in some instances, combining orthogonal variables into a single vignette produces unrealistic scenarios (Atzmüller & Steiner, 2010). In this case, researchers should consider replacing the unrealistic scenarios with more realistic ones to help ensure balance in vignette representation (Wason et al., 2002).

As a second recommendation, researchers can use the “actual derived cases” approach in which the variables chosen to be manipulated and the level of those variables are selected to represent concrete values found in actual settings (Shepherd & Zacharakis, 1999). In fact, the recommendation is to ask individuals similar to the targeted sample to provide details as to realistic factor levels (Wason et al., 2002). One of the major advantages of this approach is that it presents more lifelike scenarios to respondents, which can help to increase the generalizability of results.

Decision Point 6: Choosing the Number of Vignettes

Weber (1992) noted the importance of presenting participants with the proper number of vignettes by arguing that too many vignettes could lead to “information overload and fatigue for the respondent” (p. 143) while not enough scenarios “could limit the researchers’ ability to manipulate critical variables and could result in responses biased by the few issues contained in the scenarios presented” (p. 142).

In choosing the number of vignettes, the first issue to consider is that this number is dictated by the study’s purpose (Weber, 1992). Thus, there is not one answer to the question “How many vignettes need to be presented?” Rather, a question to ask is “What is the number of variables to be manipulated and the number of levels that each of those variables includes?” (Wason et al., 2002), as the answer to this question will provide a full population of vignettes that can be used in the study. While this full population of vignettes is available, it may not be necessary to use all of the resulting vignettes. Regardless, the first step is to create a full population of vignettes. After the full population is constructed, researchers must decide on how many of the vignettes will be actually used, and we can offer the following general recommendations. First, when conducting policy capturing/conjoint analyses, the recommendation is to use a minimum of five scenarios for every attribute that is manipulated (Aiman-Smith et al., 2002; Shepherd & Zacharakis, 1999). Moreover, the recommendation is to err on the side of including more vignettes per respondent than fewer (Aiman-Smith et al., 2002). Second, including repeated vignettes allows researchers to assess reliability (Karren & Barringer, 2002; Shepherd & Zacharakis, 1999). Thus, the recommendation particularly regarding policy capturing and conjoint analyses studies is to include four or five duplicated scenarios (Aiman-Smith et al., 2002).

Implementing an EVM Study

Decision Point 7: Specifying the Sample and Number of Participants

As Cavanaugh and Fritzsche (1985) noted, “The quality of the data obtained is dependent upon the respondent” (p. 291). This is especially true in considering the need to generalize results to a larger population. So, there is a need to match the sample to the larger population of interest (Aiman-Smith et al., 2002; Hughes & Huby, 2002; Wason et al., 2002). Additionally, it is important that the situation presented to the participant be familiar to them, otherwise responses may be artificial (Aiman-Smith et al., 2002; Cavanaugh & Fritzsche, 1985). In short, the more that the respondents can approximate a more generalized population, the higher the external validity of the results of an EVM study (Aiman-Smith et al., 2002).

A frequent challenge in EVM studies is that access to an appropriate sample can be difficult. In this situation, researchers often turn to using student samples. Whether or not this is an appropriate practice has been debated for decades (Gordon, Slade, & Schmitt, 1986; McNemar, 1946) due to concerns that student samples may not provide evidence that can be generalizable to a larger population. This is especially the case in research that aims to address issues associated with high-ranking employees (e.g., CEOs, board members, top management teams) or difficult to identify samples (e.g., early-stage entrepreneurs). An important advantage of EVM is that it can be used with samples located outside of the on-campus laboratory. However, when a suitable sample cannot be found, it may be necessary to change the vignettes used in order to better match the abilities and knowledge of the sample that is actually used (Wason et al., 2002).

New technological advances have eased the collection of data from samples that match a study’s purpose. For instance, eLancing (i.e., Internet freelancing) including sites such as Mechanical Turk and guru.com provide researchers with access to large samples of working individuals at relatively low cost (Aguinis & Lawal, 2013). Note that eLancing sites are online marketplaces where individuals willing to provide a service and those seeking those services meet. Aguinis and Lawal (2012) described how eLancing can be used to recruit research participants online. At present, samples composed of working individuals can be collected almost as easily as samples of college students, which has the potential to minimize concerns regarding external validity issues (Aguinis & Lawal, 2012).

Decision Point 8: Choosing the Setting and Timing for Administration

First, researchers should be aware of the conditions in which participants are responding. The criticism that EVM lacks realism can be addressed, at least in part, by allowing respondents to participate in their natural setting (Aguinis & Lawal, 2013; Grant & Wall, 2009). For example, as mentioned earlier, Sauer (2011) conducted a video vignette experiment investigating the effects of leadership status and style on perceptions of leadership effectiveness and team performance. Rather than bringing participants to the university facilities, they were invited to participate remotely from their own work environments.

In addition to deciding on where to conduct the EVM study, timing is also important. It is best if participants are able to respond to the vignettes in a single session (Aiman-Smith et al., 2002). In some cases, such as video vignettes, presentation may require multiple sessions, but multiple sessions are more likely to be affected by history and other threats to validity.

Decision Point 9: Choosing the Best Method for Analyzing the Data

As noted earlier, we discourage the use of between-person designs. But, if such designs are used, data analytic techniques such as MANOVA, ANOVA, and ANCOVA are appropriate. When an EVM study includes the preferred within-person of mixed design types, then there is a two-level data structure: vignette level and respondent level (Atzmüller & Steiner, 2010). Because of the multilevel nature of the data, analyses that focus on both levels simultaneously need to be used. Specifically, the recommendation is to use multilevel modeling (Aguinis, Gottfredson, & Culpepper, 2013; Hox et al., 1991).

Reporting Results of an EVM Study

In the interest of replicability, researchers should describe the process of creating and administering the vignettes as transparently as possible (Jasny, Chin, Chong, & Vignieri, 2011). A high level of detail is necessary given that research in the field of management is susceptible to numerous credibility issues with regards to reporting results, including withholding or not accurately reporting procedures and results (Bedeian, Taylor, & Miller, 2010).

Decision Point 10: Choosing How Transparent to Be in the Final Presentation of Results and Methodology

Researchers should disclose as much information as possible about the vignettes used in the EVM study. In the case of text or picture vignettes studies, they should be included in the manuscript as this enhances the transparency of the research process (Asendorpf et al., 2013). When the vignettes cannot be physically included in the manuscript (e.g., in the case of video or virtual reality vignettes), researchers should make the materials available to others upon request or through the journal’s website.

While some may balk at the idea of making experimental materials free and available to others, scientific knowledge can be enhanced through collaboration. As an example, the International Personality Item Pool (IPIP) was developed due to the slow development of personality assessment field caused in part by constraints generated by copyrighted personality assessment tools (Goldberg et al., 2006). Now consisting of over 2,000 individual personality-related items, the IPIP provides free access to a large set of items that researchers can use to further the field of personality testing. In the interest of furthering scientific knowledge, we recommend that researchers make their vignettes available.

As a recent illustration of transparency, Raaijmakers et al. (in press) included an appendix with not only the vignettes used in their study but also the process that was used in creating and pilot testing them. By providing the vignettes used in the experiment, future research now has a source to draw from in conducting work to extend theory while also providing additional data to help in the validation of the vignettes used.

Discussion

Future Substantive Research Using EVM

There are several substantive domains that would benefit from the use of EVM. Addressing a different methodological issue, Krasikova and LeBreton (2012) offered a sample of the types of research questions that are relevant for constructs that unfold in a dyadic context. Following their approach, Table 3 includes a selective set of illustrative questions that can be answered by using EVM in OB/HRM, strategic management studies, and entrepreneurship. Next, we address some of the issues listed in Table 3.

Table 3.

Illustrative Research Questions that could be Uniquely Addressed by Using Experimental Vignette Methodology.

Organizational Behavior and Human Resource Management
1.	What are the most important factors in managers’ decisions to voluntarily turnover?
2.	What policies have the greatest impact on reducing strain caused by the work-family interface?
3.	Which of a series of compensation systems work best for building morale and commitment and enhancing performance?
4.	How does observing other individuals or groups of individuals perform CWBs influence an individual’s propensity to engage in CWBs?
Strategic Management Studies
1.	How do the decision-making processes used by top management teams lead to important organizational outcomes?
2.	How do decision processes of executives differ in hostile versus benign environments?
3.	What factors exist in top management teams that lead to different strategic positions (e.g., exploration vs. exploitation)?
4.	How do the aggregated decision processes of employees result in higher levels of corporate social responsibility?
Entrepreneurship
1.	Under what circumstances do internal entrepreneurs choose to become entrepreneurs and vice versa?
2.	What factors are important in deciding whether or not to pursue an entrepreneurial venture?
3.	Do internal entrepreneurs favor a different mix of compensation and benefits options than other employees?
4.	Do individual differences exist (e.g., gender, family circumstances, personality traits) in the decision to enter certain types of entrepreneurial ventures?

Note: CWBs = counterproductive work behaviors.

First, regarding OB/HRM, EVM has the potential to provide valuable insight into work behaviors that are not easily observable. For instance, while we found several studies that used EVM to investigate organizational citizenship behavior, there is a lack of research addressing counterproductive work behaviors. Because these behaviors are observed less frequently, this research domain provides a good opportunity to better understand the processes that go into these decisions. EVM would allow researchers to present scenarios of counterproductive work behaviors to assess the consequences of observing those behaviors—and possibly investigate processes that may be involved in choosing to engage in those behaviors as well. Note that EVM is particularly suited to study these issues due to their unethical nature. As an example of another research domain, conjoint analysis and policy capturing studies could be used to examine decision processes in job crafting. These types of studies could help shed light into the decision processes of employees that engage in job crafting as well as those that choose not to do so.

Regarding strategic management studies, an area that would benefit from using EVM is corporate social responsibility. As has been noted in recent reviews, there is a need to bridge the micro/macro gap in this area (e.g., Aguinis & Glavas, 2012). EVM offers a possible way to do so (Priem, Walters, & Li, 2011). Specifically, future research could use EVM to understand the decision-making processes of organizational members at all levels and how such processes may lead to corporate social responsibility policies and actions at the firm level of analysis. In addition, research on microfoundations of strategy provides a good opportunity to also bridge the micro/macro gap (Barney & Felin, 2013). In microfoundations research, there is an interest in understanding the collective, aggregated behavior of individuals, taking into account the context of the institution and the interactions among individuals. EVM offers an excellent opportunity for further research to be conducted in this area because such research can lead to a better understanding of decision processes of individuals, taking into account the decisions of their coworkers, managers, and other individuals within the organization.

Regarding entrepreneurship, in their review of conjoint analysis, Lohrke et al. (2010) provided several recommendations for future research using EVM. For instance, they pointed out that conjoint analysis is an adequate method to better understand the decision-making process involved in opportunity identification, opportunity evaluation, and opportunity exploitation. Building on these suggestions by Lohrke et al., EVM could be used to understand those processes across different types of environment that vary regarding their stability or dynamism. Moreover, EVM can be used to gain a deeper understanding of some of the preferences that accompany entrepreneurial decisions. For instance, internal entrepreneurs may be motivated by different reward systems compared to other employees. EVM could be used to explore what preferences these employees have in their reward structure as well as tease apart possible explanations for those preferences. Also, EVM can be used as a tool for understanding what factors are important in becoming an entrepreneur, especially when the option to remain an internal entrepreneur might be available.

Future Research on EVM

One major innovation that is likely to change how EVM studies are conducted in the future is the introduction of technologies that enhance experimental realism. Although we discussed the benefits of increasing the level of immersion by using audio or video vignettes, these methods do not necessarily engage participants fully. Because of this, some have advocated the use of full immersion virtual reality (Pierce & Aguinis, 1997). Full immersion would include engaging as many of the participants’ senses as possible, including “aural, olfactory, tactile and proprioceptive senses” (Pierce & Aguinis, 1997, p. 407). In a full immersion situation, researchers would use virtual reality simulators to place participants within the situation. In essence, participants could interact with the world around them in a realistic manner while still being in the controlled environment of a laboratory. Recent advances in technology allow these scenarios to be presented in much greater realism as well. For instance, a new technology allows for smells to be presented to participants that appear to emanate from specific sources of a screen (Matsukura, Yoneda, & Ishida, 2013). Finally, technological advancements such as eLancing have great potential in terms of giving researchers access to larger, more diverse, and global pools of research participants.

Conclusions

Experimental vignette methodology is a useful way to address what seems to be an inescapable dilemma of internal versus external validity. Our review provided evidence that although EVM seems to be underutilized in management and related fields, every major journal in the field has published at least some articles that relied on EVM. Our article offers best practice recommendations, including an analysis of trade-offs, associated with 10 decision points involved in planning, implementing, and reporting results of an EVM study. These recommendations offer advice for researchers interested in using EVM and also for reviewers and readers of studies that used EVM because they will allow them to critically evaluate such studies. We hope that our article will not only serve as a catalyst to inspire the future use of EVM in management and other fields but also research that will lead to methodological improvements regarding EVM itself.

Footnotes

Acknowledgements

We thank James LeBreton and two Organizational Research Methods anonymous reviewers for highly constructive feedback that allowed us to improve our manuscript substantially. A previous version of this manuscript was presented at the meetings of the Academy of Management, Philadelphia, PA, August 2014.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Aguinis

Adams

S. K. R.

(1998). Social-role versus structural models of gender and influence use in organizations: A strong inference approach. Group and Organization Management, 23, 414–446.

Aguinis

Edwards

J. R.

(2014). Methodological wishes for the next decade and how to make wishes come true. Journal of Management Studies, 51, 143–174.

Aguinis

Glavas

(2012). What we know and don’t know about corporate social responsibility: A review and research agenda. Journal of Management, 38, 932–968.

Aguinis

Gottfredson

R. K.

Culpepper

S. A.

(2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management, 39, 1490–1528.

Aguinis

Kraiger

(2009). Benefits of training and development for individuals and teams, organizations, and society. Annual Review of Psychology, 60, 451–474.

Aguinis

Lawal

S. O.

(2012). Conducting field experiments using eLancing’s natural environment. Journal of Business Venturing, 27, 493–505.

Aguinis

Lawal

S. O.

(2013). eLancing: A review and research agenda for bridging the science-practice gap. Human Resource Management Review, 23, 6–17.

Aguinis

Vandenberg

R. J.

(2014). An ounce of prevention is worth a pound of cure: Improving research quality before data collection. Annual Review of Organizational Psychology and Organizational Behavior, 1, 569–595.

Aiman-Smith

Scullen

S. E.

Barr

S. H.

(2002). Conducting studies of decision making in organizational contexts: A tutorial for policy-capturing and other regression-based techniques. Organizational Research Methods, 5, 388–414.

10.

Alexander

C. S.

Becker

H. J.

(1978). The use of vignettes in survey research. Public Opinion Quarterly, 42, 93–104.

11.

Allen

D. G.

Hancock

J. I.

Vardaman

J. M.

McKee

D. N.

(2014). Analytical mindsets in turnover research. Journal of Organizational Behavior, 35, 61–68.

12.

Argyris

(1975). Dangers in applying results from experimental social psychology. American Psychologist, 30, 469–485.

13.

Asendorpf

J. B.

Conner

De Fruyt

De Houwer

Denissen

J. J.

Fiedler

… Wicherts

J. M.

(2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.

14.

Atzmüller

Steiner

P. M.

(2010). Experimental vignette studies in survey research. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 6, 128–138.

15.

Barney

Felin

(2013). What are microfoundations? The Academy of Management Perspectives, 27, 138–155.

16.

Bedeian

A. G.

Taylor

S. G.

Miller

A. N.

(2010). Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9, 715–725.

17.

Benjamin

Flynn

F. J.

(2006). Leadership style and regulatory mode: Value from fit? Organizational Behavior and Human Decision Processes, 100, 216–230.

18.

Bucar

Glas

Hisrich

R. D.

(2003). Ethics and entrepreneurs: An international comparative study. Journal of Business Venturing, 18, 261–281.

19.

Burstin

Doughtie

E. B.

Raphaeli

(1980). Contrastive vignette technique: An indirect methodology designed to address reactive social attitude measurement. Journal of Applied Social Psychology, 10, 147–165.

20.

Carroll

J. S.

Johnson

E. J.

(1990). Decision research: A field guide. Thousand Oaks, CA: Sage Publications, Inc.

21.

Casper

W. J.

Eby

L. T.

Bordeaux

Lockwood

Lambert

(2007). A review of research methods in IO/OB work-family research. Journal of Applied Psychology, 92, 28–43.

22.

Cavanaugh

G. F.

Fritzsche

D. J.

(1985). Using vignettes in business ethics research. In Preston

L. E.

(Ed.) Research in corporate social performance and policy (pp. 279–293, Vol. 7). London: Jai Press.

23.

De Cremer

Mayer

D. M.

van Dijke

Schouten

B. C.

Bardes

(2009). When does self-sacrificial leadership motivate prosocial behavior? It depends on followers’ prevention focus. Journal of Applied Psychology, 94, 887–889.

24.

De Cremer

Van Knippenberg

(2004). Leader self-sacrifice and leadership effectiveness: The moderating role of leader self-confidence. Organizational Behavior and Human Decision Processes, 95, 140–155.

25.

Donoho

C. L.

(2003). The “top-of-the-line” influence on the buyer-seller relationship. Journal of Business Research, 56, 303–309.

26.

Eastman

K. K.

(1994). In the eyes of the beholder: An attributional approach to ingratiation and organizational citizenship behavior. Academy of Management Journal, 37, 1379–1391.

27.

Finch

(1987). The vignette technique in survey research. Sociology, 21, 105–114.

28.

Ganong

L. H.

Coleman

(2006). Multiple segment factorial vignette designs. Journal of Marriage and Family, 68, 455–468.

29.

Goldberg

L. R.

Johnson

J. A.

Eber

H. W.

Hogan

Ashton

M. C.

Cloninger

C. R.

Gough

H. G.

(2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84–96.

30.

Goldman

B. M.

Gutek

B. A.

Stein

J. H.

Lewis

(2006). Employment discrimination in organizations: Antecedents and consequences. Journal of Management, 32, 786–830.

31.

Gordon

M. E.

Slade

L. A.

Schmitt

(1986). The “science of the sophomore” revisited: From conjecture to empiricism. Academy of Management Review, 11, 191–207.

32.

Graham

M. E.

Cable

D. M.

(2001). Consideration of the incomplete block design for policy-capturing research. Organizational Research Methods, 4, 26–45.

33.

Grant

A. M.

Wall

T. D.

(2009). The neglected science and art of quasi-experimentation: Why-to, when-to, and how-to advice for organizational researchers. Organizational Research Methods, 12, 653–686.

34.

Green

P. E.

(1984). Hybrid models for conjoint analysis: An expository review. Journal of Marketing Research, 21, 155–169.

35.

Green

P. E.

Krieger

A. M.

Wind

(2001). Thirty years of conjoint analysis: Reflections and prospects. Interfaces, 31 (3), S56–S73.

36.

Heslin

P. A.

Vandewalle

Latham

G. P.

(2006). Keen to help? Managers’ implicit person theories and their subsequent employee coaching. Personnel Psychology, 59, 871–902.

37.

Highhouse

(2009). Designing experiments that generalize. Organizational Research Methods, 12, 554–566.

38.

Hobson

F. W.

Gibson

C. J.

(1983). Policy capturing as an approach to understanding and improving performance appraisal: a review of the literature. Academy of Management Review, 8, 640–649.

39.

Hox

J. J.

Kreft

I. G.

Hermkens

P. L.

(1991). The analysis of factorial surveys. Sociological Methods & Research, 19, 493–510.

40.

Hoyt

C. L.

Price

T. L.

Poatsy

(2013). The social role theory of unethical leadership. The Leadership Quarterly, 24, 712–723.

41.

Hughes

Huby

(2002). The application of vignettes in social and nursing research. Journal of Advanced Nursing, 37, 382–386.

42.

Hyman

M. R.

Steiner

S. D

(1996). The vignette method in business research: Current uses, limitations and recommendations. In Stuart

E. W.

Ortinau

D. J.

Moore

E. M.

(Eds.), Marketing: Moving toward the 21st century (pp. 261–265). Rock Hill, SC: Winthrop University School of Business.

43.

Jasny

B. R.

Chin

Chong

Vignieri

(2011). Again, and again, and again …. Science, 334, 1225.

44.

Jasso

(2006). Factorial survey methods for studying beliefs and judgments. Sociological Methods & Research, 34, 334–423.

45.

Joseph

Chandrasekaran

(2013). Conjoint analysis: A perfect link between marketing and product design functions—A review. International Journal of Management Research and Development, 3, 8–21.

46.

Karren

R. J.

Barringer

M. W.

(2002). A review and analysis of the policy-capturing methodology in organizational research: Guidelines for research and practice. Organizational Research Methods, 5, 337–361.

47.

Krasikova

LeBreton

J. M

(2012). Just the two of us: Misalignment of theory and methods in examining dyadic phenomena. Journal of Applied Psychology, 97, 739–757.

48.

Kruschke

J. K.

Aguinis

Joo

(2012). The time has come: Bayesian methods for data analysis in the organizational sciences. Organizational Research Methods, 15, 722–752.

49.

Lohrke

F. T.

Holloway

B. B.

Woolley

T. W.

(2010). Conjoint analysis in entrepreneurship research a review and research agenda. Organizational Research Methods, 13, 16–30.

50.

Louviere

J. J.

(1988). Analyzing decision making: Metric conjoint analysis. Newbury Park, CA: Sage.

51.

Ludwick

Zeller

R. A.

(2001). The factorial survey: An experimental method to replicate real world problems. Nursing Research, 50, 129–133.

52.

Matsukura

Yoneda

Ishida

(2013). Smelling screen: Development and evaluation of an olfactory display system for presenting a virtual odor source. IEEE Transactions on Visualization and Computer Graphics, 19, 606–615.

53.

McGuire

W. J.

(1973). The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology, 26, 446–456.

54.

McKelvie

Haynie

J. M.

Gustavsson

(2011). Unpacking the uncertainty construct: Implications for entrepreneurial action. Journal of Business Venturing, 26, 273–292.

55.

McNemar

(1946). Opinion-attitude methodology. Psychological Bulletin, 43, 289–374.

56.

Melone

N. P.

(1994). Reasoning in the executive suite: The influence of role/experience-based expertise on decision processes of corporate executives. Organization Science, 5, 438–455.

57.

Miller

K. D.

Tsang

E. W.

(2011). Testing management theories: Critical realist philosophy and research methods. Strategic Management Journal, 32, 139–158.

58.

Murphy

K. R.

Herr

B. M.

Lockhart

M. C.

Maguire

(1986). Evaluating the performance of paper people. Journal of Applied Psychology, 71, 654–661.

59.

Newcombe

M. J.

Ashkanasy

N. M.

(2002). The role of affect and affective congruence in perceptions of leaders: An experimental study. The Leadership Quarterly, 13, 601–614.

60.

Pelletier

K. L.

(2012). Perceptions of and reactions to leader toxicity: Do leader-follower relationships and identification with victim matter? The Leadership Quarterly, 23, 412–424.

61.

Pierce

C. A.

Aguinis

(1997). Using virtual reality technology in organizational behavior research. Journal of Organizational Behavior, 18, 407–410.

62.

Pierce

C. A.

Aguinis

Adams

S. K. R.

(2000). Effects of a dissolved workplace romance and rater characteristics on responses to a sexual harassment accusation. Academy of Management Journal, 43, 869–880.

63.

Podsakoff

N. P.

Podsakoff

P. M.

MacKenzie

S. B.

Maynes

T. D.

Spoelma

T. M.

(2014). Consequences of unit-level organizational citizenship behaviors: A review and recommendations for future research. Journal of Organizational Behavior, 35, 87–119.

64.

Podsakoff

N. P.

Whiting

S. W.

Podsakoff

P. M.

Mishra

(2011). Effects of organizational citizenship behaviors on selection decisions in employment interviews. Journal of Applied Psychology, 96, 310–326.

65.

Podsakoff

P. M.

Dalton

D. R.

(1987). Research methodology in organizational studies. Journal of Management, 13, 419–441.

66.

Podsakoff

P. M.

MacKenzie

S. B.

Podsakoff

N. P.

Bachrach

D. G.

(2008). Scholarly influence in the field of management: A bibliometric analysis of the determinants of university and author impact in the management literature in the past quarter century. Journal of Management, 34, 641–720.

67.

Powell

G. N.

(2001). Workplace romances between senior-level executives and lower-level employees: An issue of work disruption and gender. Human Relations, 54, 1519–1544.

68.

Priem

R. L.

Harrison

D. A.

(1994). Exploring strategic judgment: Methods for testing the assumptions of prescriptive contingency theories. Strategic Management Journal, 15, 311–324.

69.

Priem

R. L.

Walters

B. A.

(2011). Decisions, decisions! How judgment policy studies can integrate macro and micro domains in management research. Journal of Management, 37, 553–580.

70.

Raaijmakers

Vermeulen

Meeus

Zietsma

(in press). I need time! Exploring pathways to compliance under institutional complexity. Academy of Management Journal.

71.

Roehling

M. V.

(1993). “Extracting” policy from judicial opinions: the dangers of policy capturing in a field setting. Personnel Psychology, 46, 477–502.

72.

Roehling

M. V.

(1999). Weight-based discrimination in employment: Psychological and legal aspects. Personnel Psychology, 52, 969–1016.

73.

Sauer

S. J.

(2011). Taking the reins: The effects of new leader status and leadership style on team performance. Journal of Applied Psychology, 96, 574–587.

74.

Scandura

T. A.

Williams

E. A.

(2000). Research methodology in management: Current practices, trends, and implications for future research. Academy of Management Journal, 43, 1248–1264.

75.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

76.

Shepherd

D. A.

(2011). Multilevel entrepreneurship research: Opportunities for studying entrepreneurial decision making. Journal of Management, 37, 412–420.

77.

Shepherd

D. A.

Patzelt

Baron

R. A.

(2013). “I Care about Nature, but…”: Disengaging values in assessing opportunities that cause harm. Academy of Management Journal, 56, 1251–1273.

78.

Shepherd

D. A.

Zacharakis

(1999). Conjoint analysis: A new methodological approach for researching the decision policies of venture capitalists. Venture Capital: An International Journal of Entrepreneurial Finance, 1, 197–217.

79.

Shook

C. L.

Ketchen

D. J.

Cycyota

C. S.

Crockett

(2003). Data analytic trends and training in strategic management. Strategic Management Journal, 24, 1231–1237.

80.

Skarlicki

D. P.

Turner

R. A.

(2014). Unfairness begets unfairness: Victim derogation bias in employee ratings. Organizational Behavior and Human Decision Processes, 124, 34–46.

81.

Slaughter

J. E.

Richard

E. M.

Martin

J. H.

(2006). Comparing the efficacy of policy-capturing weights and direct estimates for predicting job choice. Organizational Research Methods, 9, 285–314.

82.

Sniderman

P. M.

Grob

D. B.

(1996). Innovations in experimental design in attitude surveys. Annual Review of Sociology, 22, 377–399.

83.

Spector

P. E.

(1981). Research designs. Newbury Park, CA: Sage.

84.

Taggar

Neubert

(2004). The impact of poor performers on team outcomes: An empirical examination of attribution theory. Personnel Psychology, 57, 935–968.

85.

Taylor

B. J.

(2006). Factorial surveys: Using vignettes to study professional judgment. British Journal of Social Work, 36, 1187–1207.

86.

M. A.

Foo

M. D.

Aguinis

(2010). Using experience sampling methodology to advance entrepreneurship theory and research. Organizational Research Methods, 13, 31–54.

87.

Wason

K. D.

Polonsky

M. J.

Hyman

M. R.

(2002). Designing vignette studies in marketing. Australasian Marketing Journal, 10, 41–58.

88.

Weber

(1992). Scenarios in business ethics research: A review, critical assessment, and recommendations. Business Ethics Quarterly, 77, 137–160.

89.

Whiting

S. W.

Maynes

T. D.

Podsakoff

N. P.

Podsakoff

P. M.

(2012). Effects of message, source, and context on evaluations of employee voice behavior. Journal of Applied Psychology, 97, 159–182.

90.

Wilks

(2004). The use of vignettes in qualitative research into social work values. Qualitative Social Work, 3, 78–87.

91.

Woehr

D. J.

Lance

C. E.

(1991). Paper people versus direct observation: An empirical examination of laboratory methodologies. Journal of Organizational Behavior, 12, 387–397.

92.

Yun

Faraj

Sims

H. P.

Jr (2005). Contingent leadership and effectiveness of trauma resuscitation teams. Journal of Applied Psychology, 90, 1288–1296.