Abstract
Over 50 years ago, Stanley Milgram and colleagues published a short article detailing an unobtrusive experimental design they called the lost-letter technique. The technique involves placing stamped, unmailed letters in a community and using the relative rate of return to infer local attitudes toward political issues and social groups. Furthermore, the technique is simple and inexpensive enough to replicate in an introductory-level social science course as a means to familiarize students with social research methodology. This activity utilizes active learning with student-centered pedagogy, and this paper details the procedure, best practices, and student feedback. Students enjoy having a personal stake in the project and find they better understand social research through this active learning project. Instructors are encouraged to consider adopting the technique in their courses.
Keywords
Teaching research methods can be a challenging task for any instructor since the area is vast, is abstract, and can be intimidating to even the best students. Systematic reviews of the literature on research methods education echo these sentiments (Earley 2014; Wagner, Garner, and Kawulich 2011). In a recent mixed-methods study on achievement in undergraduate methods courses, Markle (2017) found many undergraduate students felt unfamiliar with social science research methods even after taking methods courses in communications, criminal justice, political science, psychology, and sociology. Students overwhelmingly failed to see the usefulness of the course and perceived science as “dull, abstract, and hard to understand” (Markle 2017:112). If students feel this way after taking an entire course on research methods, then there is good reason to believe they hold similar sentiments after completing a module on the subject in an introductory-level course. Perhaps instructors can overcome these dismal sentiments if they stress the relevance and applicability of methods through hands-on approaches rather than theoretical discussions. This paper proposes a way to do that using the lost-letter technique.
Over 50 years ago, Dr. Stanley Milgram and colleagues published a short article detailing an unobtrusive experimental design they called the lost-letter technique. The technique involves placing or dropping stamped, unmailed letters in a community. Thus, when a person discovers a letter that appears to have been lost, they have the choice of mailing it or disregarding it. The letters are all addressed to the same post office box but to different recipients. By analyzing the return rate relative to different recipients of the lost letters, we may be able to infer local attitudes toward political issues and social groups (Milgram 1969; Milgram, Mann, and Harter 1965). For example, in the original study, 72 percent of letters addressed to “Medical Research Associates” were returned, while “Friends of the Communist Party” and “Friends of the Nazi Party” had a return rate of 25 percent each. The experiment’s control, a fictitious stranger named Walter Carnap, had a return rate of 71 percent. Though these results are not surprising given the social stigma associated with Nazis and Communists, it is “possible to extend the technique to domains and issues where the answers are not so clearly known, or where it is desirable to conduct a study without directly questioning subjects” (Milgram et al. 1965:437). Moreover, the technique is simple and inexpensive enough to replicate in an introductory-level social science course as a means of familiarizing students with social research methodology.
This paper aims to address recommendations to incorporate more research education throughout undergraduate sociology programs (Markle 2017; McKinney et al. 2004). More specifically, adapting the lost-letters technique to a classroom setting provides an active learning approach to social research methods as suggested by previous scholars (Atkinson and Hunt 2008; Earley 2014; Nind, Kilburn, and Luff 2015). By transforming students into reflective practitioners, this activity engages the three core aspects of student-centered pedagogy: “engaging learners in the research process, providing hands-on experience, and offering opportunities for critical reflection” (Kilburn, Nind, and Wiles 2014:203). In other words, the lost-letter technique provides a practical way for the teacher to serve as the gatekeeper and guide to the research process in a concrete manner. Through this activity, instructors of sociology are putting on display for their students what Mills (1959) deemed the “craft” of the discipline: the systematic thinking needed to uncover facts about the social world.
Prevailing wisdom may lead one to believe students do not enjoy learning about research methods, though new evidence from Wisecup (2017) challenges the perception of student disdain for the area. Her survey of sociology majors found no positive or negative attitudes toward research, so it should not be assumed students would be put off by doing an activity like the lost-letter technique. In fact, successful integration of this experiment in class may decrease the confidence gap sociology students have regarding research (Spalter-Roth et al. 2010). This activity also has the potential to strengthen a student’s research self-concept, defined as a student’s confidence in his or her ability to understand the logic and usefulness of research methods (Markle 2017).
Done correctly, the lost-letters activity provides students with an opportunity to become experimenters while demystifying the research process and empowering their research self-concepts (see Markle 2017). The following sections offer a step-by-step guide to the activity with suggestions from the author. This paper details materials needed for the experiment, variable selection, letter placement tactics, and data-recording strategies. A discussion of assessment techniques is also included. Finally, student feedback is considered to identify strengths of this pedagogical technique in addition to suggestions for improvement.
Teaching with Lost Letters
I have used the lost-letters technique with my Introduction to Sociology classes four times since the spring of 2016. As an adjunct, I have successfully implemented this activity at a community college, a large state university, and a large research university all in a rural, midwestern setting. Though special considerations are detailed throughout this paper, student comprehension and engagement in the experiment has been uniform across these institutional settings. One rule remains consistent, however: this experiment works better with larger class sizes since this increases the sample size. I have done the activity with a class as large as 65 and while teaching two sections of Introduction to Sociology, both with roughly 30 students. The sample size can also be increased by collaborating with other instructors. Additionally, it is wise to begin this experiment with at least one month remaining in the semester to allow sufficient time for returned letters, data entry, and analysis. I do this activity after we have completed the module on social research methods in class. Research methods are typically discussed in the second chapter of most introductory textbooks, making it possible to cover foundational material before asking students to apply it concretely in this activity. Most introductory texts also include an overview of social experiments, though one may wish to incorporate the original 1965 research brief by Milgram et al. titled “The Lost-letter Technique: A Tool of Social Research.” This article is a short two pages and written in a manner understandable to most first-year college students.
Before beginning this activity, I show the 2015 film Experimenter (Almereyda 2015) in class. The movie chronicles Milgram’s career with the primary focus being the famous obedience experiments, but the audience is also shown several other experimental designs pioneered by Milgram. The lost-letter technique is detailed in a three-minute scene, which can be shown on its own (0:42:30–0:45:30). After playing the film or just the scene, I walk my classes through the experiment with the help of PowerPoint slides to serve as a step-by-step guide. Since there is much to do here, having a common visual to keep focused on each step has been very useful.
Procedure
This experiment has some minor expenses, which may be covered by one’s department or institution. The following are required.
A post office box
This can be rented from the local post office with rates beginning around $30 for a three-month period. Be sure to add the names of the categories (the control and the organizational names) to the box’s recipient list before students begin losing letters. I recommend using whichever post office is the easiest to access for the person who will be picking up the letters.
Stamps
I put in my syllabus that students will be required to bring either four U.S. Postal Service “Forever” stamps or $2 to class on the day we make the letters. I suggest doing a quick poll of the class to get an idea of how many students plan on purchasing stamps brought in by the instructor. It is wise to bring in extra stamps for students who forget.
Envelopes
Since the letters need to be uniform, it is good practice to supply envelopes to the class rather than leaving this up to individual students. Office supply stores sell large boxes of standard envelopes in various designs. Envelopes with screens or windows are good to use because this allows us to digitally print addresses on the inside letter, where they can be folded to display through the window. This is one way to eliminate handwriting as an influence in the experiment. Other possibilities include using printed address labels or printing addresses directly on the envelope.
Printed letters
The content of the letters also needs to be uniform yet at the same time resemble a legitimate letter. A generic “thank-you” letter works well here. Do not put too much time or energy into making this letter, though. In all of the times I have done this experiment, of approximately 500 lost letters, only 3 have come back opened. So, odds are no one will ever see the content of this letter.
Step 1: Decide on Variables and Categories
Deciding variables and categories is the first major area for student involvement. Generally speaking, it is wise to decide on an issue or problem first and formulate variables after. The issue must meet several criteria: (1) it must be well known to the public, (2) it must be polarizing, and (3) it must be possible to capture the opposing sides in believable organizational names. Additionally, the issue should not be something widely analyzed by survey data or polls. Keep in mind that this experiment has two major strengths: it gives us insight to an issue on a local level (where opinion data may not readily exist), and it is also mitigates social desirability bias through a truly anonymous design.
An issue that works well for this technique is marijuana reform. If that is the issue, one could name the pro-legalization organization something like “Legal Marijuana for Michigan” and the anti-legalization organization something like “Michiganders against Marijuana.” A third position could also be measured with an organization such as “Medical Marijuana for Michigan.” Regardless, the issue works well because it is widely understood by the public, it is polarizing, and polls of the issue rarely provide data on a town, city, or county level. Several other issues suited for this technique include abortion, immigration, prostitution, and assisted suicide. I have always left it up to the class to democratically decide on the issue and category names, and I actively provide feedback along the way.
The example used in this paper, the “Lives Matter” debate, was proposed by a student of mine at a community college. Two future sections of the same course at different institutions voted to replicate this design. Originally, “Blue Lives Matter” was the fourth category, and both times it was included, the rate of return matched “All Lives Matter.” As such, during the most recent rendition of this experiment, the class suggested it be replaced with something universally agreeable, just like Medical Research Associates in Milgram et al.’s (1965) original experiment. A student suggested “Kitten Lives Matter” and while this was favored by the class, we changed it to “Puppy Lives Matter” since Americans have a cultural bias toward favoring dogs over cats (Newport et al. 2006).
It’s also worthy to note here special consideration should be given to issues and organizations that have the potential to offend or upset. The goal is simply to have people discover the letters, decide whether to mail them, and think nothing twice of it. Avoid making someone outraged over the perceived existence of a local group, a likely reaction if using something like “White Lives Matter” as a category. It would not reflect well on the scientific community if a naive subject were motivated to research and/or complain about the discovery of such a letter. Not only would that be a waste of time and emotional energy, it also has the potential to create undo stress to public officials, support organizations, and any other possible group interested in public threats, such as hate groups. Before proceeding with the experiment, it is recommended one check with their institution’s institutional review board policies, as prior approval may be required.
As for this experiment’s control, the name is ultimately irrelevant as long as it is not a publicly known person. I have used the name of my mother, father, and best friend in the past, all of whom do not live in the state where the letters were lost. There is an argument that an ethnic name may elicit bias, and this would be interesting to test using this experimental design. The same holds true with gender (in fact, using a gender-neutral name for the control may be a good idea). One version of the lost-letters technique, however, found no difference between the gender of the return addressee and return rate (Kern and Yeaton 2010). It seems likely this would hold true for the intended recipient of the letter as well.
Finally, it is important to define an explicit area where the letters will be “lost.” Limiting placements to a specific city or county makes for easy comparisons to census data. Consideration should also be given to avoiding leaving letters on campus, since the demographics of most campuses do not resemble the demographics of their surrounding communities. The class can also begin to formulate hypotheses at this point using their collective knowledge of the area’s political climate.
Steps 2 and 3: Creating the Letters
The next steps of the experimental procedure are largely at the discretion of the instructor. Step 2 involves the creation of the letter inside the envelope. As previously mentioned, this letter needs to look realistic enough in the event someone opens the letter to peer inside. Nevertheless, the important part of this step is to have students sign their name in the letter since this is how one determines which letters are returned. For privacy reasons, I allow students to use an initial for either their first or last name.
Step 3 involves students stuffing, sealing, and stamping their letters. I have always done this in class as we are going over the activity. This provides a way for the instructor to ensure students complete this portion of the experiment. I recommend giving students half credit for successful completion of this step and half credit for turning in a completed data record.
Step 4: Placements and Parameters
The original experiment by Milgram et al. (1965) lost the letters in four ways: random drops on busy city sidewalks, leaving the letter behind in a shop, placing the letter in a phone booth, and placing the letter on a vehicle (under the wipers) with a note. In a critique of the lost-letters technique, Wicker (1969) found the highest rate of return was for letters placed on vehicles. This also ensures adults discover the letters rather than children. Placing letters on vehicles, however, has a major drawback. Vehicle owners can become aggravated if they discover someone touching such an expensive piece of property. Students of color have suggested this sentiment is more pronounced in their communities, and as such, they reported to me how they felt more anxiety when doing this form of placement. After this was brought to my attention the first time I did this experiment, I excluded vehicle placements from the design.
Tips for placing the letters are numerous, but a productive approach is to consider what not to do. Letters should not be discovered by friends, family, or coworkers; the researcher(s) should not know who returned or failed to return a letter. Do not leave letters outside where they may get wet or blown away. Also, make sure letters do not get swept under something or fall between cracks. Do not leave letters in tourist areas or places frequented by out-of-towners. Examples of such places include highway rest areas, popular summer beaches, major concerts or events, and anywhere else where most people are nonlocal. Do not leave letters in too high or too low of places; eye level is preferable. Finally, avoid leaving letters in restrooms. A good piece of life advice to impart on students is to never touch anything one discovers in a public restroom.
It is important the class understands that credit for the assignment is not contingent on receiving a returned letter. The integrity of the experiment will be compromised if students feel the need to have a letter returned since they may just mail it themselves. Finally, while a stranger may see the student leave the letter behind and kindly return it to them, I tell my students that if this happens, they should smile, say “thank you,” and wait until the stranger moves on. Afterward, they can place it right back where they intended.
Steps 5 and 6: Recording Data and Creating a Data Set
Step 5 involves a separate data record students complete as they lose their letters. In the past, students collected the following information as they placed each letter: city/town, county, date, time, and the name of specific location. Students turn these letter records in once they are completed. The instructor may want to add or subtract categories on this record depending on what they think is worthy of analysis. I have not yet discovered any significant correlations between return rate and any of the above-mentioned categories, but students are usually interested to see which towns have high and low return rates. It is also interesting to see how long it takes for the letters to be returned after they are lost, with the average being just 4 days and the maximum being over 20 days. One can discover this information by recording the postmark date on the letters as they are returned.
Step 6 is typically the responsibility of the instructor, as it involves the creation and analysis of a data set. Entering the letter records into a statistical program can take some time but can be streamlined by formatting the letter records in a manner similar to a spreadsheet. In the past, I simply enter students’ letter records into SPSS and analyze the data with basic measures understandable to students in an introductory-level course. Such measures include frequencies, cross-tabulations, and bar graphs. Another possibility is to have students contextualize these results with data from the U.S. Census Bureau. The instructor may also opt to involve students in the creation and analysis of the data set, though I have not yet done this in my classes.
Step 7: Results
The most recent results of the experiment were taken from two sections of an Introduction to Sociology class both taught in fall 2017 at a large, midwestern, state university. Letters were lost between two counties with similar demographics, with results combined in Table 1. Results are in line with what was hypothesized when considering the demographics and conservative climate of the area. Census data reveal each county is over 80 percent white non-Hispanic, and results from the 2016 election show each county overwhelmingly voted for Donald Trump. As such, we hypothesized “Black Lives Matter” would not fare as well as the other categories measured, though we were surprised that it was only slight lower. We furthermore correctly hypothesized that the control (Monica Zabroski) received the highest return rate. It is possible, however, for these results to be interpreted as county residents having a lack of interest in politics altogether as opposed to a lack of enthusiasm for any particular category.
Results from Two Sections of Introduction to Sociology Taught in Fall 2017.
Variations
The original experiment by Milgram et al. (1965) did not include a return address on the envelopes, and I think it is prudent to do the same. Moreover, the gender of the return addressee was found to have no impact on the return rates (Kern and Yeaton 2010). It may be a good idea, though, to have a generic, preprinted statement of “Thank You for Your Donation” or “Business Mail” in the space of the return address.
In one study using the lost-letter technique, Bouchard and Stuster (1969) dropped letters within 30 feet of a public mailbox. Though they had a relatively high rate of return, this strategy seems flawed in that it is conceivable a person (or postal worker) would post the letter without bothering to examine the recipient. Another variation of the original experiment compared the return rates for stamped and unstamped letters, with stamped letters having a significantly higher return rate (Simon and Gillen 1971).
Several more recent implementations of the lost-letter technique have found no difference in the return rate among organizations dedicated to heart diseases, diabetes, or suicide (Witte, Smith, and Joiner 2010); no difference in the return rate among chiropractic care versus established medical care (Kern and Yeaton 2010); and no difference if using altercasting (e.g., “A good person would . . .”), though such methods have been found to compound the experiment while yielding no socially significant data (Turner et al. 2010).
Assessment and Feedback
The purpose of this activity is to demystify social research through an active learning approach. As such, this activity has two general goals: heighten student understanding of social research and enhance student attitudes toward social research. Both of these are discussed in this section.
Student learning can be assessed in a variety of ways. I have used multiple-choice exam questions and brief written responses in the form of critical reflections as part of in-class evaluations used to gauge participation. I ask students to tell me in their own words what the experiment sought to measure and how it contributes to our understanding of social reality. Most students can readily provide acceptable answers to these queries, and some have even informally done so already when discussing the experiment with their friends and family. Though my assessment of student learning lacks a comparison group, students can communicate the purpose and practices of social research after partaking in this class experiment.
Student impressions were measured directly by placing questions on end-of-semester evaluations and by asking students to share their thoughts as part of a brief reflective writing assignment. Feedback from the most recent iteration of the activity was overwhelmingly positive, judging by anonymous end-of-semester evaluations completed by 48 students between two classes (87.3 percent response rate). The median response to the statement, “The Lost Letters Experiment was useful in understanding social research methods,” was 4.6 out of a 5-point scale, with 1 meaning strongly disagree and 5 meaning strongly agree. The same result (4.6 median) was found in a second query using the same scale. Qualitative feedback in the same evaluations provides more detail on this activity’s strengths and weaknesses. Students commonly described the activity as “fun and interesting” in open-ended responses when asked to share their thoughts about the class experiment. Many students thought watching the 2015 Experimenter movie was particularly helpful, and at least one student appreciated the high-profile nature of the original researcher: I liked the Lost Letters activity. It definitely helped watching the film on Stanley Milgram before completing the assignment, I had a better understanding of why we were doing what we were doing and how it plays into psychology and sociology. It was also very fun because I thought it was cool to replicate on a smaller scale an experiment that has been done before by a famous psychologist!
Additionally, students tend to report enjoying their experience placing the letters in the field. One student said, “In one case, I saw a person start to interact with the letter I had dropped. Even though she didn’t pick it up it was cool to watch undercover and see what she would do about it.” Other students echo this appreciation for taking on the role of the researcher, indicating an ability to visualize themselves doing social research in the future, something not effectively delivered through standard pedagogical approaches of reading, lecture, and discussion.
While students enjoyed the local focus of the experiment (and most are interested in what the results reveal about the local area), some students thought it would have been better to include campus placements or restrict the experiment to the campus altogether. One student suggested a list be provided of acceptable placement locations, which would increase the reliability of the design. The other major suggestion expressed by students was to be more explicit in the rationale of doing the experiment as a class. It is good practice to emphasize this activity will expose the learner to the multiple steps in conducting research, familiarizing them to the scientific process while empowering their research self-concepts. Finally, while I have made the completed data set available to students in the past, it is certainly possible for an instructor to work with their class on the final analysis as part of a statistics module. Both the experiment and analysis focus on the application of knowledge, building student skills and confidence in the material along the way.
Discussion
The lost-letters technique provides instructors with a way to illuminate social research methods through an active learning approach. It immerses students in the research process and directly involves them in data collection procedures. Moreover, a critical discussion of the approach allows students to identify limitations of the design. These are all overlapping elements of a student-centered pedagogy identified by Kilburn et al. (2014) as being essential in stimulating learning.
The experiment has many weaknesses, which are valuable components to this technique as a pedagogical tool. In discussing the results of the experiment with students, take time to explicate inadequacies of this experimental design. Students readily recognize flaws, such as inconsistent letter placements, the possibility of classmates not actually placing the letters, the probability of some letters getting destroyed before they are discovered, and the chance personal biases may compromise the results. No method is perfect and researchers are trained to acknowledge methodological shortcomings in their reports. Interrogating such limitations makes for a good class discussion, and I suspect such critical discussions are often overlooked when teaching methods from a traditional textbook.
Employing multiple styles of teaching in the classroom makes sociological content more accessible to diverse student learning styles. Calls for the reform of undergraduate research methods education have been vast and varied (Atkinson and Hunt 2008; Earley 2014; Markle 2017; McKinney et al. 2004; Nind et al. 2015; Spalter-Roth et al. 2010; Wagner et al. 2011), and this paper seeks to provide an alternative or supplement to the traditional lecture. Future advances in teaching research methods would be wise to focus on similar active learning techniques and strategies. An additional strength of this technique is that it can be easily implemented in nearly any course where social research methods are taught. By simply breaking the monotony of the traditional lecture and discussion, an instructor can actualize the relevance and applicability of research methods and perhaps increase their chances of inspiring a new generation of social scientists.
Footnotes
Editor’s Note
Reviewers for this manuscript were, in alphabetical order, Andrea Hunt, Mary Senter, and Allison Wisecup.