Analyzing Data From Studies Depicted on Video

Abstract

We describe an activity that combines the instructional benefits of video with the active and integrative learning inherent in the analysis and interpretation of realistic data. In the activity, students watched a brief video clip that showed how an actual research study was conducted. Students then analyzed raw data from the study, interpreted the results, and compared their work to published results and conclusions. The activity allows students to practice applying their knowledge of statistics and research methodology to realistic situations without having to conduct actual research. Students reported that the activity is engaging and helpful, and a comparison of pretest and posttest quiz scores suggested that it helps students learn to use and interpret statistical procedures.

Keywords

teaching activity statistics research methods active learning

Students often have difficulty applying what they have learned in methodology and statistics courses. It is likely not unusual for a student who has completed introductory courses to still have trouble analyzing and interpreting data from a simple experiment. To address this problem, many instructors supplement their lectures with a variety of student-centered activities. The effectiveness of the active learning approach is supported by a large body of empirical research (Prince, 2004; Ruiz-Primo, Briggs, Iverson, Talbot, & Shepard, 2011), some of which is specific to the teaching of statistics (Ryan, 2006; Tintle, Topliff, VanderStoep, Holmes, & Swanson, 2012). In this article, we describe an active learning exercise in which students analyze and interpret data from an actual research study shown in a brief video clip. The activity has several of the instructional benefits of doing research, while avoiding some of the drawbacks of having students do their own studies.

To make statistics and research courses more active, we and other instructors have tried at least three approaches. One tactic is to have students conduct their own studies, giving them the opportunity to acquire statistical and research skills through realistic application. Analyzing data and drawing conclusions from their own studies is perhaps the best way for students to learn about research, and many instructors have promoted this do-your-own-study approach (e.g., Ballman, 2000; Cobb, 1992; Morgan, 2001; Nolan & Speed, 1999; Thompson, 1994). From the instructor’s point of view, one drawback of having students conduct research is the amount of time that may be required to make it work. Limited resources and issues related to institutional review board approval can create additional problems. Also, Boger (2001) found that having students find or generate real data to analyze (as compared to analyzing instructor-provided data) can reduce student appreciation for statistics, perhaps because of the extra work the students had to do.

Another approach is to have students work with actual data from real research studies. Hand, Daly, Lunn, McConway, and Ostrowski (1994), echoing other instructors, argued that teaching with real data helps demonstrate that statistical methods are relevant and important, used by real people to solve real problems. Artificial data, in contrast, may contribute to the misconception that statistics is merely a collection of procedures for crunching numbers. Although there are many sources of real data (e.g., Data and Story Library, n.d.; Hand et al., 1994), perhaps the main difficulty for instructors is simply finding appropriate examples.

A third approach uses video to improve the teaching of research and statistics. There are drawbacks to video; it requires specific equipment, appropriate videos may be difficult to find, and the “activity” of watching a video can be a passive experience unless it is specifically designed to engage students. However, technological advances have made it increasingly convenient to incorporate video segments into courses, either during class meetings or as outside viewing assignments. Videos can also help instructors meet a variety of learning goals. As Moore (1993) explained, “Video, when done well and used wisely, shows rather than tells. Video connects instruction to students’ everyday experience by bringing real events into the classroom” (p. 174). Videos can provide a change of pace, capture attention, and make abstract ideas concrete. Students also tend to remember examples they have seen in videos (Cherney, 2008; VanderStoep, Fagerlin, & Feenstra, 2000). Videos are used in a variety of courses to provide vivid examples of psychological concepts and as the basis for discussion and application of course content (e.g., Anderson, 1992; Boyatzis, 1994; Christopher, Walter, Marek, & Koenig, 2004; Conner, 1996; Dorris & Ducey, 1978; Fleming, Piedmont, & Hiam, 1990; Hart & Stevens, 1995; Kelley & Calkins, 2006; Logan, 1988; Roskos-Ewoldsen & Roskos-Ewoldsen, 2001; Simpson, 2008; Thompson, Vermette, & Wisniewski, 2004). The use of video in statistics and research courses appears to be less common than in other courses, but this need not be the case, as many engaging examples of psychological research are available.

The Video Research Activity

In our activity, students analyze realistic data from studies depicted on video and draw their own conclusions. The activity allows students to simulate many of the steps in the research process—including the selection, application, and interpretation of statistics—but it requires much less time and effort than having students do their own studies. By showing the video of actual research, and through the use of realistic data, the activity adds an element of realism and is potentially more engaging and meaningful than working with contrived examples.

To illustrate, one video we use is a 7-min segment from the Scientific American Frontiers series that discusses therapeutic touch. Practitioners of therapeutic touch believe they can speed healing in a sick or injured person by massaging that person’s purported energy field. We begin the activity by showing the first few minutes of the segment to provide background information; in this case the video explains what therapeutic touch is and shows nurses performing therapeutic touch on patients in a hospital. We stop the video at that point and have our students discuss for a few minutes, in small groups, how they might design a study to find out if therapeutic touch actually promotes healing. After a brief class discussion of the students’ research ideas, we show the next portion of the video which describes a study conducted by Rosa, Rosa, Samer, and Barrett (1998). Instead of testing whether patients improve more after receiving therapeutic touch, Rosa et al. tested the assertion of practitioners that they could detect a supposed human energy field. The video shows several participants being tested. Before the video reveals the results of the study, we stop it again and discuss strengths and weaknesses of the study described in the video. We then give the students data from the study; because the actual raw data were not available, we provide realistic data that produce the same results reported by Rosa et al. Students select and perform the appropriate statistical analysis, then attempt to write their results and conclusions in American Psychological Association format. Finally, students compare their results and conclusions to those given in the video and in the article written by Rosa et al.; any differences provide fruitful points for discussion. The entire activity, including data analysis during class, takes about 30 min.

Information about 12 different studies shown on video, data for each study, and a handout for use with the activity, are available from the authors and on our website. Most of the video segments we use are between 3 and 7 min long, and most can be freely viewed on demand via streaming video. For longer videos or more complicated data sets, we make the analysis an assignment to be completed outside of class. In three cases, the actual raw data from the study could be reconstructed from information in the publication or the video. For the other studies, we generated realistic data—raw data that yield essentially the same results reported in the original publication.

Assessment of the Activity

To evaluate the effectiveness of the activity, we gave quizzes before and after three videos used in a recent research methods course in which 30 students were enrolled. The topic of the videos (and their associated statistical tests) were whether therapeutic touch is effective (one-sample t-test); whether yawning is contagious (2 × 2 chi-square test of independence); and whether the “quiet-eye” technique improves basketball shooting accuracy (paired t-test).

We wrote four pairs of short-answer test items for each video and randomly assigned them to either quiz form A or B (Bartsch, Bittner, & Moreno, 2008). As examples, the one-sample t-test questions included: (1) What is the null hypothesis for a one-sample t-test? and (2) A researcher gave a vocabulary test to a sample of 100 third grade children. Now the researcher wants to see if there is a significant difference between average vocabulary scores of the boys and girls. Should the researcher do a one-sample t-test? Explain why or why not. Half of the students were randomly assigned to take form A of the quiz as a pretest and the other students took form B as a pretest. After the activity, each student completed the other quiz form as a posttest. Quizzes were scored collaboratively by the two authors; we were blind to student identity and to whether the quiz was a pretest or a posttest. Each answer was categorized as either correct or incorrect, thus each quiz score could range from 0 to 4.

For each activity, we did a 2 (Test: pretest and posttest) by 2 (Order: form A first or form B first) mixed analysis of variance (ANOVA) on quiz scores. In all the three cases, the mean posttest score was significantly higher than the mean pretest score; F ratios ranged from 14.1 to 18.9, and the effect sizes (partial eta-square) ranged from .40 to .52. On average, across the three video activities, 64% of the students scored higher on the posttest, with an average improvement of 1.0 points. These data suggest that the video activity helped students improve their understanding of applied statistics.

At the end of the semester, all 30 students completed a survey to assess their reactions to the activity. The survey contained seven statements to which students indicated their level of agreement on a 1 (strongly disagree) to 5 (strongly agree) scale. All students agreed or strongly agreed that the activity helped them “understand how research and statistics can be applied to psychological topics that I’m interested in” and that the activity “is more interesting than analyzing made-up data.” All students also agreed or strongly agreed with the statement “The instructor should use the activity again next year,” and 26 students (87%) agreed or strongly agreed that the activity helped them “learn more about which statistical test to apply in different situations.” Finally, for each video, more than 90% of students agreed or strongly agreed that the video was interesting. Together, the quiz and survey data suggest that the video research activity was interesting to nearly all students and that it helped most students improve their ability to apply statistics to research problems.

Conclusion

In our experience, many students find that one of the most difficult aspects of statistical analysis is selecting the appropriate technique for a given research question. Students who can perform the calculations for a t-test may be unsure when to apply it or how to interpret the outcome. The video research activity helps students practice this skill; it gives students an opportunity to see the relationship between a specific research question and a testable statistical hypothesis, and to learn what conclusions can and cannot be drawn from a study. In some cases, our students discover that they thought they knew when to do a t-test or two-way ANOVA, but they did not. However, when students obtain the same results and arrive at the same conclusions as professionals, it gives them confidence and reinforces lessons learned in the course. Another important benefit of the activity is that it allows students to see very concretely how statistics and research design are relevant to their interests in psychology. Videos can include studies from diverse areas of psychology, so nearly all students will be interested in at least some of the studies. As a convenient substitute for the real thing, the video research activity gives students a taste of the excitement of discovery that can accompany scientific research.

We see two main drawbacks of the activity. First, it is limited, for the most part, to published research studies that are shown in a video. Compared to the use of contrived examples, the activity gives instructors less flexibility in the research topics, research designs, and statistical methods that they can address. The materials we provide cover a variety of situations, but they are far from comprehensive. Instructors who wish to generate new activity materials must find appropriate videos, then generate raw data (see Morse, 2013, for one method), extract it from the published account of the research, or obtain the original data from the researchers who conducted the study. A second drawback is that the activity can require substantial amounts of class time. However, students can do parts of the activity outside of class, such as data analysis or the writing of results and conclusions.

In summary, the video research activity is a potentially useful supplement to other teaching methods; it combines the instructional benefits of video with the active and integrative learning inherent in the analysis and interpretation of realistic data, while avoiding some of the difficulties that arise when students conduct actual research projects.

Footnotes

Authors’ Note

Materials for the video research activity can be downloaded from .

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Anderson

D. D.

(1992). Using feature films as tools for analysis in a psychology and law course. Teaching of Psychology, 19, 155–158. doi:10.1207/s15328023top1903_6

Ballman

(2000). Real data in classroom examples. In Moore

T. L.

(Ed.), Teaching statistics: Resources for undergraduate instructors (pp. 11–18). Washington, DC: Mathematical Association of America.

Bartsch

R. A.

Bittner

W. M. E.

Moreno

J. E.

(2008). A design to improve internal validity of assessments of teaching demonstrations. Teaching of Psychology, 35, 357–359. doi:10.1080/00986280802373809

Boger

(2001). The benefit of student-generated data in an introductory statistics class. Journal of Education for Business, 77, 5–8. doi:10.1080/08832320109599663

Boyatzis

C. J.

(1994). Using feature films to teach social development. Teaching of Psychology, 21, 99–101. doi:10.1207/s15328023top2102_9

Cherney

I. D.

(2008). The effects of active learning on students’ memories for course content. Active Learning in Higher Education, 9, 152–171. doi:10.1177/1469787408090841

Christopher

A. N.

Walter

J. L.

Marek

Koenig

C. S.

(2004). Using a “new classic” film to teach about stereotyping and prejudice. Teaching of Psychology, 31, 199–202. doi:10.1207/s15328023top3103_5

Cobb

(1992). Teaching statistics. In Steen

(Ed.), Heeding the call for change: Suggestions for curricular action (pp. 3–43). Washington, DC: Mathematical Association of America.

Conner

D. B.

(1996). From Monty Python to Total Recall: A feature film activity for the cognitive psychology course. Teaching of Psychology, 23, 33–35. doi:10.1207/s15328023top2301_6

10.

Data and Story Library. (n.d.). Retrieved March 28, 2009, from http://lib.stat.cmu.edu/DASL/DataArchive.html

11.

Dorris

Ducey

(1978). Social psychology and sex roles in films. Teaching of Psychology, 5, 168–169. doi:10.1207/s15328023top0503_17

12.

Fleming

M. Z.

Piedmont

R. L.

Hiam

C. M.

(1990). Images of madness: Feature films in teaching psychology. Teaching of Psychology, 17, 185–187. doi:10.1207/s15328023top1703_12

13.

Hand

D. J.

Daly

Lunn

A. D.

McConway

K. J.

Ostrowski

(Eds.). (1994). A handbook of small data sets. London, England: Chapman and Hall.

14.

Hart

K. E.

Stevens

(1995). The use and evaluation of video supplements in the teaching of introductory psychology. Journal of Instructional Psychology, 22, 103–114.

15.

Kelley

Calkins

(2006). Evaluating popular portrayals of memory in film. Teaching of Psychology, 33, 191–194. doi:10.1207/s15328023top3303_7

16.

Logan

R. D.

(1988). Using a film as a personality case study. Teaching of Psychology, 15, 103–104. doi:10.1207/s15328023top1502_11

17.

Moore

D. S.

(1993). The place of video in new styles of teaching and learning statistics. The American Statistician, 47, 172–176.

18.

Morgan

B. L.

(2001). Statistically lively uses for obituaries. Teaching of Psychology, 28, 56–58.

19.

Morse

B. J.

(2013). Replicate this! Creating individual level data from summary statistics using R. Teaching of Psychology, 40.

20.

Nolan

Speed

T. P.

(1999). Teaching statistics theory through applications. The American Statistician, 53, 370–375.

21.

Prince

(2004). Does active learning work? A review of the research. Journal of Engineering Education, 93, 223–231.

22.

Rosa

Samer

Barrett

(1998). A close look at therapeutic touch. Journal of the American Medical Association, 279, 1005–1010. doi:10.1001/jama.2011.1797

23.

Roskos-Ewoldsen

D. R.

Roskos-Ewoldsen

(2001). Using video clips to teach social psychology. Teaching of Psychology, 28, 212–215.

24.

Ruiz-Primo

M. A.

Briggs

Iverson

Talbot

Shepard

L. A.

(2011). Impact of undergraduate science course innovations on learning. Science, 331, 1269–1270. doi:10.1126/science.1198976

25.

Ryan

R. S.

(2006). A hands-on exercise improves understanding of the standard error of the mean. Teaching of Psychology, 33, 180–183. doi:10.1207/s15328023top3303_5

26.

Simpson

(2008). Classic and modern propaganda in documentary film: Teaching the psychology of persuasion. Teaching of Psychology, 35, 103–108. doi:10.1080/00986280802004602

27.

Thompson

W. B.

(1994). Making data analysis realistic: Incorporating research into statistics courses. Teaching of Psychology, 21, 41–43. doi:10.1207/s15328023top2101_9

28.

Thompson

W. B.

Vermette

P. J.

Wisniewski

S. A.

(2004). Ten cooperative learning activities for the cognitive psychology course. Teaching of Psychology, 31, 134–136.

29.

Tintle

Topliff

VanderStoep

Holmes

Swanson

(2012). Retention of statistical concepts in a preliminary randomization-based introductory statistics curriculum. Statistics Education Research Journal, 11, 21–40.

30.

VanderStoep

S. W.

Fagerlin

Feenstra

J. S.

(2000). What do students remember from Introductory Psychology? Teaching of Psychology, 20, 89–92. doi:10.1207/S15328023TOP2702_02