Research Methods in Psychology: A Feminist Exercise to Facilitate Students’ Understanding of Operational Definitions,Observation,and Inter-Rater Reliability

Part 1

For the first part of this exercise, I have students watch a popular Saturday Night Live digital short called “The Creep” (over 100 million views on youtube.com). Prior to watching the video, I instruct my students that they will watch a short video clip and to take out a blank piece of paper. I ask each student to watch the video closely and write down each time they observed a “creepy” behavior. At this time in the semester, students are familiar with operational definitions and had practice developing construct parameters of their own (on a given topic of their choice). I purposefully do not provide an operational definition nor do I mention the concept. As an instructor, this is an opportunity for me to understand how my students translate an abstract concept (“creepy” behavior) into concrete forms of observation.

The song and its lyrics revolve around a dance called the “Creep” as well as “creeping” in various situations and places. The video is intended to be humorous and coins a dance where everyone is instructed to “…get up on your feet. It’s real easy to do, and it’s called ‘The Creep’. Let your hands flap around like a Marionette…Now pull your waistband up like you expectin’ a flood and slick your hair down flat like it was covered in mud” (The Lonely Island, 2011). Andy Samberg, Jorma Taccone, and Akiva Schaffer (members of The Lonely Island) are wearing suit pants that are too short for their height and aviator-style glasses. Situations in which these men are “creeping” include watching (spying on) their parents in bed and a woman alone in her bedroom through a window. Other situations include dancing the “Creep” toward a group of women on the dance floor, a woman in front of an ATM, and a women judge (the latter examples include the men wearing prison attire with accompanying lyrics “when the judge is a hottie and you can’t control your body…do the creep”). Half way through the video, Nicki Minaj appears (dressed similarly) and is shown watching men disrobe in a locker room as well as sneaking into a funeral home to kiss a man in a coffin.

After watching the video, I ask each student to share how many creepy behaviors they observed and compared across students. The average is typically one or two observed behaviors. During this large-discussion process, students begin to understand why and how inter-rater reliability is established. Students tend to agree on which behaviors were creepy (rather lack thereof), demonstrating high interrater reliability. In several cases, students do not perceive any part of the video or its lyrics to be creepy and record zero observations. In a few cases, students observe two or more creepy behaviors.

Next, I ask students how they operationalized creepy behaviors. A consensus typically emerges that the actions of the characters in the video and the song lyrics are funny, innocent, and not intended to be harmful. For instance, some students use the motivation of humor as part of their operational definition of creepy behaviors, indicating that the intention for the behavior was more important than the actual behavior. Other students defined creepy as disturbing or threatening behaviors. Further guiding this discussion, I ask students to describe examples in which they coded as a creepy behavior. Often, students discuss the actions of Nicki Minaj when she walks into a funeral home and hovers over a man in a coffin (with accompanying lyrics “when you sneak into a wake, and you see a beefcake…do the creep”). Infrequently, students cite specific examples of the men’s behaviors in the video.

Part 2

For the second part of this exercise, I ask students to consider creepy behaviors in the context of the law and social norms. Specifically, I ask students to provide examples in which stalking, spying, unwanted attention, and harassment would be protected by the law or perceived as creepy. Through this discussion, my students and I illuminate the ways in which society explicitly (legal precedent, resource allocation) and implicitly (social norms, behavioral aftermath of systemic inequity) educates us on the value of women. For instance, legislation may be in place in the United States (for instance, Title VII and sexual violence laws), though lack of prosecution, high rates of acquittal, brief sentences, and a larger victim-blaming culture illustrate the many ways in which women’s lives are not valued (Acierno et al., 1997; Armstrong et al., 2006; Berdahl & Moore, 2006; McLaughlin et al., 2012; Tjaden & Thoennes, 2006).

To contextualize this larger class discussion, I provide students with background information on gender-based harassment, stalking, and unwanted sexual attention (Breiding, 2014; Leskinen et al., 2011; Moors et al., 2014; Rothenberg, 2000). Specifically, I find this valuable as many students may not be aware of what each of these terms encompass. For instance, often the prevailing view of sexual harassment is that unwanted sexual attention is the “quintessential harassment” (Leskinen et al., 2011). Yet, unwanted sexual attention involves a broad range of expressions of romantic or sexual interest that are unwelcome, unreciprocated, and offensive (Leskinen et al., 2011; Lim & Cortina, 2005). Specifically, gender harassment is defined as a “broad range of verbal and nonverbal behaviors not aimed at sexual cooperation but that convey insulting, hostile, and degrading attitudes about women” (Fitzgerald et al., 1995, p. 430). I also include examples of gender harassment, such as offensive remarks, degrading comments, and infantilization (see Leskinen & Cortina, 2014, for additional examples and a psychological measure of gender harassment). Dovetailing discussions of harassment and unwanted sexual attention, I discuss how stalking is a “pattern of repeated and unwanted attention, harassment, contact, or any other course of conduct directed at a specific person that would cause a reasonable person to feel fear” (Rape, Abuse & Incest National Network, 2019).

Tying to earlier discussions of humor, I present background on empirical research that documents the ways in which sexist humor contributes to tolerance of gender-based harassment and gender inequity (Bill & Naus, 1992; Ford et al., 2001; Love & Deckers, 1989; Thomae & Pina, 2015). For instance, both correlational and experiment research have documented that exposure to sexist jokes is linked to victim-blaming and endorsement of rape myths (Ryan & Kanjorski, 1998; Thomae & Tendayi, 2013). Through critically examining what seems evident and “normal” attitudes and behavior toward women, students expand their conceptual framework for defining creepy behaviors.

Next, I have students re-watch the video and code for creepy behaviors (repeat the same exercise as in the first part). After watching the video, I ask each student to share how many creepy behaviors they observed and compared across students. The average this time is typically seven or eight observed behaviors. This shift in average observed behaviors is illuminating to students, as they report that new information about gender harassment, stalking, and humor provided a different set of parameters for how they operationally defined creepy behaviors. Again, I ask students to describe examples in which they coded as a behavior as creepy. Students often describe how the men’s creepy behaviors toward women (stalking, unwanted attention, spying) is normative and humorous; however, critically examining these assumptions yielded a change in perspective for many students. For instance, students describe several examples from the video, including spying on a woman through a window and aggressively approaching women in several situations. In these cases, students shifted from the focus of the humorous behavior of men to reactions of women in the video. Targets of the creepy behaviors and accompanying dance are depicted as visibly upset, shocked, and afraid. In some cases, students reflect on how they held a woman engaging in these behaviors (in the case of Nicki Minaj) to a different standard than men (see Biernat & Manis, 1994). This exercise also reinforced concepts of inter-rater reliability (this exercise again yielded high inter-rater reliability) while underscoring the importance of applying a critical feminist lens to observational design.

Conclusion

Concepts of operational definitions, observational design, and inter-rater reliability are hallmarks of psychological research methods. Many textbook examples of these concepts tend to incorporate broad examples (e.g., the reliability of a gas pump gauge, watching children play; coding facial emotions; Cozby, 2009; Haslam & McGarty, 2014; Schweigert, 2011) or self-described “silly” examples (e.g., foot intelligence scale; Cozby, 2009). To locate students as active creators of knowledge, I strive to incorporate feminist perspectives and activities within my research methods courses. The example activity outlined above was designed to help guide students through the process of shifting conceptual frameworks (Trigwell & Prosser, 1996; Weber, 1998) in an engaging way (students’ laughter and “ah-ha” moments were audible ¹ ). As an instructor, my goal is that students are engaged in class discussions, gain a deep understanding of the course material, and feel prepared to use empiricism to shed light on pressing social issues.

In addition to incorporating the present teaching tool into research methods courses, instructors can consider how this tool can be implemented in other psychology courses, such as those focused on gender, social, or development. For instance, for discussions related to gender harassment or sexual violence against women, the present teaching tool could serve as a way to illustrate how people likely recode or reinterpret information (i.e., what constitutes creepy behaviors) based on new information (i.e., learning about legal definitions and feminist research on gender harassment). This exercise could also be incorporated into classroom discussions on sexist humor and how women and men are often evaluated differently for engaging in the same behavior (e.g., shifting standards; Biernat & Manis, 1994). In a similar vein, an adapted version of this exercise could be used to illustrate selective perception and stereotyping (Biernat et al., 1996; Koomen & Dijker, 1997) by highlighting how people tend to view behavior through the dominant point of view, which often exempts men from problematic behaviors.