Best Practice Recommendations for Designing and Implementing Experimental Vignette Methodology Studies

Decision Point 1: Deciding Whether EVM Is a Suitable Approach

While we argue that EVM is valuable, it is not always the most appropriate methodological approach. First, EVM is particularly useful when researchers need to exercise control of independent variables to gather evidence regarding causation (Cavanaugh & Fritzsche, 1985). EVM allows researchers to include factors that are relevant to the research question while excluding those that might confound the results. This amount of control helps to test causal hypotheses that would otherwise be difficult. Therefore, EVM is particularly useful in research domains in which variables are known to correlate but there is a need to determine the nature and direction of causal relationships.

Second, EVM is also an appropriate method when researchers are faced with ethical dilemmas associated with conducting experimental research. While it is difficult to experimentally manipulate sensitive topics in an ethical manner, EVM provides researchers with the ability to create hypothetical scenarios that address sensitive topics. As an example, it would be ethically impossible to manipulate workplace romances in an experiment, but by using EVM, researchers have been able to understand causal antecedents of perceptions and attributions of blame of former workplace romance participants when the romance turns into a sexual harassment allegation (e.g., Pierce et al., 2000). In short, EVM is a good choice when the goal is to investigate sensitive topics in an experimentally controlled way.

It is also important to recognize issues that may lead to the decision to not use EVM. Specifically, because EVM requires participants to respond to hypothetical scenarios, there are some situations when those scenarios do not create the same context as would be encountered in “real life” (Lohrke, Holloway, & Woolley, 2010). This can make it difficult to use EVM when certain contextual pressures are difficult to reproduce. For instance, in some high-stakes decision-making scenarios (e.g., mergers and acquisitions), presentation of hypothetical scenarios are not likely to produce the same responses as when those same situations occur in a natural setting.

Decision Point 2: Choosing the Type of EVM

The next major decision is to choose the type of EVM. As mentioned earlier, the two major options involve paper people and policy capturing/conjoint analysis. Although we referred to them earlier, next we offer a more detailed description as well as examples of published articles that implemented each.

Decision Point 3: Choosing the Type of Research Design

The next decision involves choosing the type of research design. Choices include a between-person, within-person, or mixed research design (Atzmüller & Steiner, 2010).

Between-person designs require that each participant read only one vignette, and comparisons are made across participants (Atzmüller & Steiner, 2010). True between-person designs are uncommon, especially when judgments are used as the dependent variable (Atzmüller & Steiner, 2010). For instance, because participants are only presented with a single rather than multiple vignettes, they lose any chance at comparison that would help to ground responses contextually. Without other vignettes to serve as referent points for their own judgments, responses may not accurately reflect the true judgments of each respondent. For this reason, it is important that in between-person designs participants be provided with sufficient information to help provide as much context as possible. For instance, in their paper people EVM study, Raaijmakers et al. (in press) first presented all of their participants with baseline information (i.e., a general description of the issues involved) to provide a similar contextual background for all participants. Accordingly, the recommendation is to provide participants with adequate contextual background when using a between-subjects design or use the within-person or mixed design options.

When using a within-person design, each participant views the same set of vignettes, and comparisons are made between vignettes within the same person (Atzmüller & Steiner, 2010). This type of design helps to show the effects of a manipulation within one individual and is useful in terms of uncovering judgment processes of a single individual. In mixed designs, different groups of participants receive different sets of vignettes; however, within each group, participants see the same vignettes (Atzmüller & Steiner, 2010). Accordingly, because multiple respondents also offer responses regarding the same vignettes, comparisons can be made across respondents.

Decision Point 4: Choosing the Level of Immersion

One of the major criticisms regarding the use of EVM is that it is unrealistic and results are not easily generalizable (Hughes & Huby, 2002). In fact, EVM studies are criticized for only showing that certain outcomes can happen but not necessarily that they do happen outside of the experimental situation. Accordingly, researchers have made calls to improve the external validity of EVM by enhancing the level of realism present in the stimulus presentation (Hughes & Huby, 2002; Roehling, 1999; Woehr & Lance, 1991). Moreover, much like the training literature has demonstrated that transfer of training is improved by increasing the similarity between the training and job contexts (Aguinis & Kraiger, 2009), improving realism by increasing the similarity between the experimental and natural settings increases the observed effects (e.g., Taylor, 2006).

One recommendation to improve realism is to increase the level of immersion experienced by participants—the subjective experience of being personally immersed in the situation described in the vignette. Technological advances have provided the means to do so by changing the method of presentation of scenarios. For example, there is the possibility of using audio, video, pictures, and other presentation methods that increase the realism of EVM studies (Aiman-Smith et al., 2002; Goldman, Gutek, Stein, & Lewis, 2006; Green, Kriege, & Wind, 2001; Hughes & Huby, 2002). These types of presentation methods are aimed at engaging participants’ senses more fully. Indeed, virtual reality technology (VRT) has become the latest type of media used to present vignettes to participants. For instance, by presenting the vignettes as fully immersing scenarios through VRT, participants are able to see, hear, touch, smell, and fully experience information that would be impossible to implement in a more traditional written scenario (Pierce & Aguinis, 1997).

In our review of the literature, we found that fewer than 6% of the studies reported using video vignettes and the majority of those were paper people studies. Examples of such studies include Aguinis and Adams (1998), Newcombe and Ashkanasy (2002), and Pelletier (2012). However, video vignettes have also been used in conjoint analyses as well (e.g., Donoho, 2003).

Increasing the immersion of participants has several benefits. First, observing actual behavior is more likely to engage participants to a greater extent, allowing them to remember and recall important information (Hughes & Huby, 2002). Additionally, as mentioned earlier, more immersive techniques enhance experimental realism (Pierce & Aguinis, 1997). As Woehr and Lance (1991) showed, paper presentations provide fewer distractors in scenarios than when behavior is directly observed. By using a more immersive medium, scenarios become more lifelike in that they are likely to provide a greater amount of “natural noise” in each scenario. If the “noise” created by the distractors is controlled by the researcher, more lifelike scenarios could be created without compromising the internal validity of the experiment (Pierce & Aguinis, 1997). For instance, in their study observing voice behaviors using a video vignette, Whiting, Maynes, Podsakoff, and Podsakoff (2012) controlled presentation of one possible source of noise by using actors who were rated as equally attractive. This helped them rule out the effect that attractiveness could have on the results of the study. Additionally, greater immersion is likely to help increase external validity as the scenarios more closely approximate experiences in the real world (Heslin, Vandewalle, & Latham, 2006). Finally, using techniques that are higher on immersion also allows researchers to explore a variety of sensitive topics that are difficult or unethical to manipulate in the real world (Pierce & Aguinis, 1997).

While there are many benefits that accompany immersive presentation techniques, there are also some trade-offs that should be taken into account. As vignette studies become more immersive, the cost associated with the experiment typically increases as well. Creating a written vignette usually requires no more than a researcher’s time and creativity. But in a study using video vignettes in place of written vignettes, Taggar and Neubert (2004) had to hire five professional actors to role play the various scenarios for their participants.

Decision Point 5: Specifying the Number and Levels of the Manipulated Factors

Theory plays a crucial role in planning an EVM study because it is the driving force in choosing what factors are relevant to the research in question as well as the number of levels for each manipulated variable (Atzmüller & Steiner, 2010; Priem & Harrison, 1994). In fact, this is often one of the criticisms leveled against vignette methodologies (Lohrke et al., 2010): Because EVM requires prespecified variables and levels, the threat of omitting important variables, especially in complex decision-making processes, is high. As Shepherd and Zacharakis (1999) keenly noted, “the key is to identify what information is critical to the decision being studied” (p. 207). The issue of model misspecification is a concern in all empirical research, but it is particularly important in the context of EVM and experimental research in general given the criticism that that such research does not test “whether a hypothesis is true, but rather whether the experimenter is a sufficiently ingenious stage manager to produce in the laboratory conditions which demonstrate that an obviously true hypothesis is correct” (McGuire, 1973, p. 449).

There are two main approaches that can be used in choosing variables to include (Shepherd & Zacharakis, 1999). The first involves an attribute-driven design, in which factors are set up so that they are orthogonal to one another. This allows researchers to more easily assess the independent effects of each manipulated factor (Aiman-Smith et al., 2002; Karren & Barringer, 2002). However, presenting orthogonal cues has limitations. For example, in some instances, combining orthogonal variables into a single vignette produces unrealistic scenarios (Atzmüller & Steiner, 2010). In this case, researchers should consider replacing the unrealistic scenarios with more realistic ones to help ensure balance in vignette representation (Wason et al., 2002).

As a second recommendation, researchers can use the “actual derived cases” approach in which the variables chosen to be manipulated and the level of those variables are selected to represent concrete values found in actual settings (Shepherd & Zacharakis, 1999). In fact, the recommendation is to ask individuals similar to the targeted sample to provide details as to realistic factor levels (Wason et al., 2002). One of the major advantages of this approach is that it presents more lifelike scenarios to respondents, which can help to increase the generalizability of results.

Decision Point 6: Choosing the Number of Vignettes

Weber (1992) noted the importance of presenting participants with the proper number of vignettes by arguing that too many vignettes could lead to “information overload and fatigue for the respondent” (p. 143) while not enough scenarios “could limit the researchers’ ability to manipulate critical variables and could result in responses biased by the few issues contained in the scenarios presented” (p. 142).

In choosing the number of vignettes, the first issue to consider is that this number is dictated by the study’s purpose (Weber, 1992). Thus, there is not one answer to the question “How many vignettes need to be presented?” Rather, a question to ask is “What is the number of variables to be manipulated and the number of levels that each of those variables includes?” (Wason et al., 2002), as the answer to this question will provide a full population of vignettes that can be used in the study. While this full population of vignettes is available, it may not be necessary to use all of the resulting vignettes. Regardless, the first step is to create a full population of vignettes. After the full population is constructed, researchers must decide on how many of the vignettes will be actually used, and we can offer the following general recommendations. First, when conducting policy capturing/conjoint analyses, the recommendation is to use a minimum of five scenarios for every attribute that is manipulated (Aiman-Smith et al., 2002; Shepherd & Zacharakis, 1999). Moreover, the recommendation is to err on the side of including more vignettes per respondent than fewer (Aiman-Smith et al., 2002). Second, including repeated vignettes allows researchers to assess reliability (Karren & Barringer, 2002; Shepherd & Zacharakis, 1999). Thus, the recommendation particularly regarding policy capturing and conjoint analyses studies is to include four or five duplicated scenarios (Aiman-Smith et al., 2002).

Decision Point 7: Specifying the Sample and Number of Participants

As Cavanaugh and Fritzsche (1985) noted, “The quality of the data obtained is dependent upon the respondent” (p. 291). This is especially true in considering the need to generalize results to a larger population. So, there is a need to match the sample to the larger population of interest (Aiman-Smith et al., 2002; Hughes & Huby, 2002; Wason et al., 2002). Additionally, it is important that the situation presented to the participant be familiar to them, otherwise responses may be artificial (Aiman-Smith et al., 2002; Cavanaugh & Fritzsche, 1985). In short, the more that the respondents can approximate a more generalized population, the higher the external validity of the results of an EVM study (Aiman-Smith et al., 2002).

A frequent challenge in EVM studies is that access to an appropriate sample can be difficult. In this situation, researchers often turn to using student samples. Whether or not this is an appropriate practice has been debated for decades (Gordon, Slade, & Schmitt, 1986; McNemar, 1946) due to concerns that student samples may not provide evidence that can be generalizable to a larger population. This is especially the case in research that aims to address issues associated with high-ranking employees (e.g., CEOs, board members, top management teams) or difficult to identify samples (e.g., early-stage entrepreneurs). An important advantage of EVM is that it can be used with samples located outside of the on-campus laboratory. However, when a suitable sample cannot be found, it may be necessary to change the vignettes used in order to better match the abilities and knowledge of the sample that is actually used (Wason et al., 2002).

New technological advances have eased the collection of data from samples that match a study’s purpose. For instance, eLancing (i.e., Internet freelancing) including sites such as Mechanical Turk and guru.com provide researchers with access to large samples of working individuals at relatively low cost (Aguinis & Lawal, 2013). Note that eLancing sites are online marketplaces where individuals willing to provide a service and those seeking those services meet. Aguinis and Lawal (2012) described how eLancing can be used to recruit research participants online. At present, samples composed of working individuals can be collected almost as easily as samples of college students, which has the potential to minimize concerns regarding external validity issues (Aguinis & Lawal, 2012).

Decision Point 8: Choosing the Setting and Timing for Administration

First, researchers should be aware of the conditions in which participants are responding. The criticism that EVM lacks realism can be addressed, at least in part, by allowing respondents to participate in their natural setting (Aguinis & Lawal, 2013; Grant & Wall, 2009). For example, as mentioned earlier, Sauer (2011) conducted a video vignette experiment investigating the effects of leadership status and style on perceptions of leadership effectiveness and team performance. Rather than bringing participants to the university facilities, they were invited to participate remotely from their own work environments.

In addition to deciding on where to conduct the EVM study, timing is also important. It is best if participants are able to respond to the vignettes in a single session (Aiman-Smith et al., 2002). In some cases, such as video vignettes, presentation may require multiple sessions, but multiple sessions are more likely to be affected by history and other threats to validity.

Decision Point 9: Choosing the Best Method for Analyzing the Data

As noted earlier, we discourage the use of between-person designs. But, if such designs are used, data analytic techniques such as MANOVA, ANOVA, and ANCOVA are appropriate. When an EVM study includes the preferred within-person of mixed design types, then there is a two-level data structure: vignette level and respondent level (Atzmüller & Steiner, 2010). Because of the multilevel nature of the data, analyses that focus on both levels simultaneously need to be used. Specifically, the recommendation is to use multilevel modeling (Aguinis, Gottfredson, & Culpepper, 2013; Hox et al., 1991).

Decision Point 10: Choosing How Transparent to Be in the Final Presentation of Results and Methodology

Researchers should disclose as much information as possible about the vignettes used in the EVM study. In the case of text or picture vignettes studies, they should be included in the manuscript as this enhances the transparency of the research process (Asendorpf et al., 2013). When the vignettes cannot be physically included in the manuscript (e.g., in the case of video or virtual reality vignettes), researchers should make the materials available to others upon request or through the journal’s website.

While some may balk at the idea of making experimental materials free and available to others, scientific knowledge can be enhanced through collaboration. As an example, the International Personality Item Pool (IPIP) was developed due to the slow development of personality assessment field caused in part by constraints generated by copyrighted personality assessment tools (Goldberg et al., 2006). Now consisting of over 2,000 individual personality-related items, the IPIP provides free access to a large set of items that researchers can use to further the field of personality testing. In the interest of furthering scientific knowledge, we recommend that researchers make their vignettes available.

As a recent illustration of transparency, Raaijmakers et al. (in press) included an appendix with not only the vignettes used in their study but also the process that was used in creating and pilot testing them. By providing the vignettes used in the experiment, future research now has a source to draw from in conducting work to extend theory while also providing additional data to help in the validation of the vignettes used.