A National Survey of Teaching and Learning Research Methods: Important Concepts and Faculty and Student Perspectives

Abstract

In this study, we assessed instructor and student attitudes and knowledge toward research methods (RM). Instructors (N = 62) answered questions about course format, topic importance, and resources. Students (N = 166) of some of those instructors answered questions regarding attitudes toward research. Five major factors organize topics that instructors find most important. Only ratings of statistics importance varied by rank. Associate and full professors rated statistics as being more important than other instructors. There were significant relationships between attitudes toward and knowledge of RM together with the higher perceived utility of some course components. Requiring students to conduct their own research was not a significant predictor of attitudes or RM knowledge.

Keywords

research methods undergraduate training educational resources psychology curriculum

The research methods (RM) course holds a prominent position in the undergraduate psychology curriculum in the United States (Dunn et al., 2010; Roberts, 2016) and around the world (Field, 2010; Sümer, 2016). To a large extent, this is because it focuses on a critical learning outcome in psychology. The American Psychological Association (APA, 2013) lists “scientific inquiry and critical thinking” as one of the learning goals for an undergraduate psychology degree and as one of seven domains of the Standards for high school teaching (APA, 2012). Furthermore, the ability to read, comprehend, critically evaluate, and apply the findings of research is a prerequisite to psychological literacy (Cranney & Dunn, 2011; McGovern et al., 2010) and a foundation of the introductory psychology course (Gurung et al., 2016). In this article, we examined faculty and student perceptions of the RM course. Together with measuring how the content of this course is valued, taught, and learnt, we predicted students’ knowledge of the content.

Given the importance of RM to psychological science, it is not surprising that nearly every undergraduate psychology program offers an RM course (Norcross et al., 2016; Stoloff et al., 2010). However, RM is generally regarded as a challenging subject to teach. Ciarocco et al. (2017) gathered faculty views on the purpose and challenges of the course. Faculty rated the development of scientific thinking (79%), the increase of engagement in the research process (28%), and preparation for higher level courses (22%) as the most important reasons for the course. The most common challenge involved course design issues such as struggling with the balance between the different types of material, the available time to teach “everything,” and making the material engaging. Other challenges involved teaching students how to conduct research, teaching data analysis, and teaching writing.

Instructors have tried to address challenges differently. One study explicitly tested the efficacy of both active learning and a form of scaffolding. Students in the new classes showed better APA-style writing, a higher perceived utility of research and statistics, better attitudes toward statistics, and higher perceived skills/abilities in statistics (Ciarocco et al., 2013).

In a recent parallel study on students, Strohmetz et al. (2018) reported students see learning to think scientifically and learning to do research as equally important, compared to instructors who place a higher value on learning to think scientifically. Students also saw RM more as a preparation for graduate school than as one to cultivate employable skills. Whereas students worried about writing issues and learning how to design/conduct their own study, instructors were more likely to list student issues (e.g., lack of motivation) and an inability to engage in higher order thinking as the challenging aspects of the course. Students did not rate learning how to engage in research as challenging as instructors viewed this issue.

RM is clearly an area of weakness for many students (Murtonen, 2015). Most students (75%) are not enthusiastic to take RM (Rajecki et al., 2005) or prefer to passively read or hear about research over actively conducting research individually or with a team (Vittengl et al., 2004). Students have relatively little interest in reading and conducting research (Rottinghaus et al., 2006; Vittengl et al., 2004), hold negative attitudes toward RM and statistics (Addison et al., 2015; Murtonen, 2005; Sizemore & Lewandowski, 2009), and fail to see the future relevance or utility of methods and statistics courses (Ciarocco et al., 2013; Earley, 2014; Murtonen et al., 2008).

Ironically, exposure to RM may strengthen negative attitudes about the course. Students report lower levels of perceived usefulness of both statistics and research knowledge (Sizemore & Lewandowski, 2009) and a loss of interest in scientific activities (Manning et al., 2006) after the course. Two possible factors that may alleviate the negative attitudes are whether students collect their own data and the format of the course. Students who actively design and conduct research as part of their methods course report more interest in doing research, feel more prepared to conduct their own research, and have more favorable impressions of the course overall (Ball & Pelco, 2006; Roberts & Allen, 2012, 2013). Taking the course face-to-face is important as well. Students who did collect their own data in an online course scored significantly lower on measures of quantitative mastery of statistical concepts than those who took the face-to-face version though the size of this effect was small (Goode et al., 2018).

Past research on RM focuses on instructor challenges and student perceptions of the course. We extend prior research by examining how instructors organize and design their courses and separately examine students’ self-reported knowledge and confidence about course material. What content do instructors believe is critical for an RM course? To capture a more detailed, in-depth picture, we conducted an online national survey to offer information about course design and content and provide a needs analysis that can serve instructors of RM in psychology.

Beyond educators, we also focused on students and used a range of psychological measures to assess their RM knowledge and confidence. Specifically, we explored which factors predict knowledge of RM in students and whether a certain course design leads to a stronger confidence in RM knowledge. Here, we built on past research (Amsel et al., 2011; Roberts & Allen, 2013; Sizemore & Lewandowski, 2009) and assessed students’ perceptions of the utility of research and statistics, attitudes toward research and statistics, perceptions of research and statistical abilities, and efficacy in APA-style writing as a component of the scientific process.

Together with presenting a detailed picture of how RM is taught and learnt, we structure our article around five key questions designed to help teachers of psychology. Our major research questions included the following: How is RM designed? What are the important concepts in RM? Does the importance of concepts vary by faculty rank or background? What course components do students most learn from? What predicts student learning in RM? Given the broad research base, we expected the majority of courses to feature many hands-on components (e.g., data collection), faculty to report coverage of all the topics as most challenging, and students to report active learning assignments as most useful and predicting their knowledge.

Method

Participants

Sixty-two instructors of RM, predominantly women (68%), completed the online instructor survey. Respondents had been teaching an undergraduate RM course from less than a year to 40 years (M = 12.82 years, SD = 10.45). The range also varied for teaching at the master’s and doctoral level: Participants had been teaching at the master’s level for 2.17 years (SD = 5.55) and at the doctoral level for 1.27 years (SD = 5.12). Respondents were at all academic ranks: full (42%), associate (21%), and assistant professors (24%), with a smaller percentage at the graduate student or adjunct lecturer rank (6%). Participants had been trained in the following areas of psychology: social (31%), experimental (19%), developmental (18%), cognitive (19%), clinical (5%), and other (8%). Approximately 5.85 (SD = 6.58) sections of RM were offered each year. The number of students enrolled in RM courses ranged from 0 to 1,200 students (M = 206.30, SD = 292.35).

In total, 141 students taking RM completed the student survey. The sample was drawn from 12 colleges and universities with varying numbers from each institution. For example, 49 students participated from the University of California—Davis, 32 from the University of Wisconsin—Green Bay, 17 from Seton Hall, and 10 from Tulane. Students ranged in age from 18 to 50 years (M = 21.10 years, SD = 4.55). High school grade point average (GPA) ranged from 2.00 to 4.70 (M = 3.68, SD = 0.53). Students were predominantly women (86%) and White (48%). Other ethnicities included Asian (21%), Latinx (15%), Black (9%), and other (2%).

Materials

There were two different surveys for this study (complete surveys available on https://osf.io/yfh8u/). The instructor survey contained 36 questions. Participants provided information on the content and organization of the course, their education, rank and teaching experience, the method of assessment and evaluation, use of teaching tools in addition to textbooks, and additional teaching resources they would need or want. The student survey included 39 questions. We measured students’ attitudes toward psychology as a science, RM, and statistics; confidence about RM; ability to use research knowledge; APA-style self-efficacy; and study behavior.

Instructor survey

We asked instructors to indicate their gender, rank, area in which they received their PhD, and the number of years they have been teaching RM in psychology. We also asked how many RM courses are offered each year in their department, whether statistics is a prerequisite, how the methods class is offered (one RM class, require two RM classes [stats is separate], one combined RM and statistics class, require two classes [combined methods and stats class]), and how many undergraduate students take an RM course each year.

Course design

Survey respondents described many elements of their course (e.g., if a textbook was used) and how many times they used of each different teaching resources in their course (e.g., journal articles, media or news articles). To capture course grade breakdown, instructors described what percentage of the overall grade was assigned to different evaluation methods. We asked about the importance of teaching 25 topics (e.g., sources of information, ethics) within an RM course. The list was developed by examining current RM textbooks. Respondents indicated the importance of including a particular topic using a 1 (strongly disagree) to 5 (strongly agree) scale. We also asked for respondents’ biggest challenge in teaching RM and what works well for teaching RM. Respondents were asked to upload their syllabi and any assignments (available as a teaching resource for the course https://classroom.google.com/c/MTQ4MjQyMzEyNjda). Survey respondents also indicated whether students conducted their own research, what statistics program they used, and whether their students were required to complete an APA-style research paper.

Needs assessment

To assess the needs of instructors with respect to content, tools, and/or resources, we asked four open-ended questions regarding what topics participants felt were missing from RM in psychology textbooks, what topics they believe should have less coverage in RM textbooks, what methods worked well, and what their major challenges were.

Student survey

Besides basic demographics, we also measured student self-reported scholastic aptitude test (SAT) and American College Testing (ACT) score and number of psych classes taken. We had three major categories of student measures: attitudes toward research, science, and statistics; course design; and student characteristics. Table 1 summarizes all measures used in the study.

Table 1.

Student Surveys With Sample Items.

Psychology as a Science (Friedrich, 1996)

“It’s just as important for psychology students to do experiments as it is for students in chemistry and biology”

Attitudes Toward Statistics (Sizemore & Lewandowski, 2009)

“I feel comfortable in a statistics class”

Attitudes Toward Research (Sizemore & Lewandowski, 2009)

“Reading articles about research in psychology is something I enjoy”

Perceived Utility of Statistics (Sizemore & Lewandowski, 2009)

“Taking a statistics course is a valuable part of one’s education”

Perceived Utility of Research (Sizemore & Lewandowski, 2009)

“The concepts learned in a research class will be very helpful to me in the future”

Perceived Ability in Statistics (Sizemore & Lewandowski, 2009)

“Statistics is easy for me”

Perceived Ability in Research (Sizemore & Lewandowski, 2009)

“I usually understand what we are talking about in my research classes”

Perceived ability to write in APA Style (Ciarocco et al., 2013)

“I feel I could write an effective Introduction for an APA style paper”

Attitudes toward the Scientific Process (Ciarocco et al., 2013)

“Research design is a useful tool for answering questions”

Confidence in research skills (Allen & Baughman, 2016)

How confident are you, that with your notes and textbooks, that you could:

Run and interpret an independent-samples t test using Statistical Package for the Social Sciences?

Study behavior checklist (Gurung et al., 2010)

Students self-reported how many minutes on each of 11 strategies (e.g., made examples, reviewed figures).

Student Characteristics (Author generated)

“When I am given an academic task, I make a strong effort to find the right answers”

Psychological Research Methods Survey (Amsel et al., 2014)

Which of the following is the best description of the use of inferential statistics?

A procedure used to explain the relationship between two variables.

A method for summarizing a large amount of data with a few numbers.

A method used to determine the practical importance of research findings.

A procedure for determining if differences are due to chance or non-chance factors.

Note. APA = American Psychological Association.

Student attitudes

Students completed the Psychology as Science (PAS) questionnaire (Friedrich, 1996) to assess general psychology attitudes. Participants answered 20 items on a 7-point Likert-type scale ranging from 1 (strongly disagree) to 7 (strongly agree) with 15 items used for coding and 5 filler items. A person’s total score is the average agreement score on the select 15 items. Our Cronbach’s α inter-item reliability = .82.

Participants next completed questions assessing attitudes toward conducting research and statistics (Sizemore & Lewandowski, 2009). This 30-item scale was divided up into six subscales measuring attitudes toward research (Cronbach’s α = .74), attitudes toward statistics (α = .89), perceived utility of research (α = .91), perceived utility of statistics (α = .84), perceived ability in research (α = .78), and perceived ability in statistics (α = .61). Respondents indicated their attitudes on a scale from 1 (strongly agree) to 6 (strongly disagree).

We measured confidence in seven different skills relating to RM in psychology with a scale developed by Allen and Baughman (2016). Respondents used a 4-point scale ranging from 1 (not at all confident) to 4 (very confident). Our Cronbach’s α inter-item reliability was .79.

To analyze knowledge gained in the writing process and attitudes toward research, we used the attitudes toward scientific process measure (Ciarocco et al., 2013). Participants rated 16 statements on a 7-point scale from 1 (not at all) to 7 (definitely). The first eight questions addressed the writing process (Cronbach’s α = .94). The second half of the measure addressed attitudes toward the scientific method (Cronbach’s α = .81).

We measured knowledge of RM with the Psychological Research Methods (PRM) Survey (Amsel et al., 2014). The PRM has 10 items, each with four possible answers. Our KR-20 reliability (binary calculation due to right/wrong answers) was .55. This reliability is acceptable given the exam-like nature of the scale.

Course design

We asked students to describe their courses (e.g., was data collection involved) including which statistics program was used and whether they were required to complete an APA-style research paper. We also asked participants the extent to which class resources (i.e., textbooks, journal articles, presentations, exams, discussion, and lecture) aided their understanding of the material. Participants rated this using a Likert-type scale (1 = strongly agree, 7 = strongly disagree). Our Cronbach’s α inter-item reliability was .81. It is important to note that this is a measure of how helpful each component is viewed and not how often the course component was present.

Student characteristics

To tap into the students’ self-confidence, coping, and perceptions of instructor support for learning, we created a 10-item measure drawing items from published measures of motivation, self-efficacy, and teacher behaviors. Participants rated this using a Likert-type scale (1 = strongly agree, 7 = strongly disagree). Our Cronbach’s α inter-item reliability was .87.

Finally, we analyzed study behavior using a study behavior checklist (Gurung et al., 2010). Students listed how long they studied for a test, how far in advance they started studying, and a breakdown of study habits. Students self-reported how many minutes on each of 11 strategies (e.g., made examples, reviewed figures). We then asked how confident participants were in their knowledge for the most recent exam. Table 2 presents descriptive data for all student variables.

Table 2.

Descriptive Data for Major Student Variables (Student Survey).

Key Variables	Minimum	Maximum	M	SD
Psychology as a science	2.40	6.80	5.35	0.77
Attitudes toward statistics	2.20	5.60	3.88	0.54
Attitudes toward research	2.60	5.80	4.13	0.58
Perceived utility of statistics	2.20	6.20	4.08	0.56
Perceived utility of research	2.00	5.60	3.76	0.55
Perceived ability in statistics	2.00	6.20	3.47	0.62
Perceived ability in research	2.20	6.60	4.24	0.65
APA	1.00	7.00	5.22	1.51
Scientist	2.50	7.00	5.61	0.82
Student characteristics	1.00	7.00	5.73	0.90
Confidence	1.00	4.00	2.62	0.64
Psychological Research Methods Survey	.00	8.00	4.65	2.65
Think of mnemonic/memory devices	2.00	100.00	19.54	16.42
Review figures and tables in text	1.00	120.00	25.58	22.77
Make up examples	2.00	120.00	26.87	21.62
Memorize definitions through repetition	2.00	180.00	41.99	34.33
Review highlighted information from text	1.00	180.00	44.41	39.49
Use concept checks, chapter-end questions	1.00	240.00	46.60	47.14
Read your notes	3.00	240.00	49.99	44.80
Rewrite notes and/or skim notes	10.00	240.00	51.99	47.80
Study with a friend	3.00	500.00	54.76	93.43
Read the text	5.00	300.00	62.57	63.83
Take notes from the book	4.00	240.00	63.07	58.47
Studying behavior (Minutes)	30.00	1,100.00	299.10	238.63

Note. The first 11 constructs are rated on a 7-point scale. The final 12 items are measured in minutes. APA = perceived ability to write in American Psychological Association style; Scientist = attitudes toward the scientific process; Confidence = confidence in research skills.

Procedure

We posted an invitation to participate in our study on the Society for the Teaching of Psychology’s Facebook page, used electronic mail invitations to instructors who had signed up in the Hub for Intro Psych and Pedagogical Research, and invited reviewers for SAGE Publishing who had given their consent. To be eligible, participants had to have taught an RM course at least once. Instructors volunteering to participate were invited to send the student survey to their class members.

Results

How Is the RM Course Designed?

Taking a statistics class was a prerequisite to taking RM at 73% of institutions, and a little over half the sample institutions only required one RM class (56%; 22% required two). We found few differences related to whether RM courses required statistics as a prerequisite. A one-way analysis of variance (ANOVA) showed that instructors whose courses did not have a statistics prerequisite reported correlational approach as more important than those who didn’t, F(1, 57) = 4.54, p = .037, and preparing manuscripts more important than those who didn’t, F(1, 57) = 5.24, p = .026. Only 8% of our sample taught RM online and 3% taught it as a hybrid class.

Students were often required to conduct their own research using either an experimental design (44% of institutions) or a correlational design (38% of institutions). In some of these institutions, students did not collect their own data but used data provided (30%). In a small percentage of institutions, students did not analyze data (14%). Most of our sample required students to write a complete APA-style paper (84%). The average number of references students required to cite ranged from 2 to 20 (M = 7.65, SD = 3.83). The bulk of the sample used a textbook for class (85.7%). With respect to evaluation and grade composition, exams constituted the major method of assessment, accounting for approximately 29% of the overall grade followed by papers (15%) and quizzes (12%). Class participation (7%) and presentations (8%) were other methods of evaluation.

What Are the Important Concepts in RM?

A major focus of this article was to determine which content areas instructors of undergraduate RM courses consider most important to teach as a reflection of their importance in the field. As Table 3 displays, there was little variation in importance ratings across the 25 topics presented. Most topics were rated as important or very important. As we expected topics to be related thematically, we clustered items into five factors based on similarity: basic knowledge (BK), design, skills, statistics, and secondary topics (ST). We combined items within each factor into an aggregate score for further analyses. Each factor showed acceptable internal reliability: BK Cronbach’s α = .80, design Cronbach’s α = .85, skills Cronbach’s α = .82, statistics Cronbach’s α = .86, and ST Cronbach’s α = .72. A bivariate correlation showed factors were moderately correlated with each other with the exception of skills which was not significant. For example, BK was significantly correlated with design, r (59) = .49, p < .001; statistics, r (59) = .31, p = .016; and ST, r (59) = .54, p < .001.

Table 3.

Descriptive Statistics for Importance of Topics in Research Methods (Instructor Survey).

Factor	Items	Minimum	Maximum	M	SD
Design	Between-subjects research design	2	5	4.61	0.67
	Within-subjects research design	2	5	4.54	0.70
	Generating testable ideas	2	5	4.42	0.75
	Factorial experimental designs	1	5	4.22	1.04
Skills	Communicating research	3	5	4.58	0.59
	APA-style writing	3	5	4.34	0.80
	Preparing manuscripts	1	5	3.75	1.17
	Preparing posters	1	5	3.53	1.10
	Preparing talks	1	5	3.24	1.12
Basic knowledge	Experimental approach	3	5	4.78	0.46
	Confounding variables	3	5	4.71	0.53
	Ethics	3	5	4.69	0.50
	Sources of information	2	5	4.58	0.65
	Extraneous/control/third variables	3	5	4.56	0.65
	Using surveys	2	5	4.44	0.73
	Validities	2	5	4.44	0.86
	Replicability	2	5	4.31	0.77
	Scales of measurement	1	5	4.27	0.91
Statistics	Analyzing data	2	5	4.00	1.03
	Statistics	1	5	3.63	1.05
	Using a statistics program	1	5	3.20	1.05
Other topics	Correlational approach	2	5	4.54	0.65
	Quasi-experiments	2	5	4.20	0.87
	Using observational methods	2	5	4.14	0.88
	Case studies	1	5	3.02	1.06

Note. N = 59.

We combined graduate student, adjunct instructors, and the “other” into one category and conducted a multivariate analysis of variance with the four factors that correlated with each other and an ANOVA on skills. Although the multivariate test was significant, Pillai’s Trace F(12, 162) = 1.86, p = .043, η² = .12, the effect was driven by ratings of the importance of statistics. Associate and full professors rated statistics as being more important than the graduate student/ad hoc/other ranked instructors, M_difference = .75, 95% CI [0.03, 1.49], p = .039, and M_difference = .85, 95% CI [0.18, 1.52], p = .007. Ratings of the importance of skills did not vary across academic rank. A separate set of analyses showed ratings of importance of topics did not vary with training backgrounds of instructors.

Needs Assessment: Content and Resources

We asked four open-ended questions. All authors read through responses to each item and pulled key themes from each. Authors then compared coding and resolved discrepancies to create a list of the most commonly stated items for each question.

In regard to topics absent in books, the replication crisis was mentioned by 17% of respondents followed by evaluating news and research, the qualitative approach, and statistics (each listed by 10% of respondents). Most respondents did not list any topics needing less coverage (17%). Frequently mentioned topics for less coverage included complex design (10%), excessive details, single-case design, and types of validity (each listed by 7%).

The most cited strategy that worked well was real-life applications and labs (cited by 8%), followed by the use of group projects, research presentations, and reporting results (cited by 6%). The major challenges listed by instructors revolved around student attitudes to the course and related to low student interest (17%), not enough time to cover material (14%), differences in preparation (12%), APA-style issues (11%), and lack of statistical knowledge, student issues relating to their lives (both cited by 8%).

What Course Components Do Students Most Learn From?

To test this question, we correlated student attitude and knowledge responses with ratings of the utility of each of 12 resources often used by instructors (see Table 4). The pattern of correlations highlights resources that may work better for some students than others. For example, ACT scores were negatively correlated with the perceived utility of presentations, group work, and class participation.

Table 4.

Correlations Between Course Design and Student Attitudes.

Key Variables	Textbook	Articles	SocM.	Present.	Ex.	Discuss	Papers	Qz	GWK	Part	Lecture	Videos
ACT score	−.13	−.02	−.20	−.25*	.06	−.11	−.01	−.17	−.25*	−.26*	−.07	−.12
ATR	.14	.32**	.07	.08	.20*	.05	.35**	.12	−.09	.02	−.01	.01
PUR	−.15	.01	−.01	.10	.08	−.04	.05	.12	.11	.03	−.01	.00
PAR	.03	−.05	−.11	.19*	.20*	.08	.15	.17	−.12	−.00	.17	.04
APA	.10	.27**	.04	.35**	.30**	.38**	.47**	.31**	.22*	.24**	.29**	.10
Scientist	.08	.23*	−.20*	.08	.31**	.15	.25**	.22*	.05	.15	.26**	.10
PAScience	−.02	.21*	−.13	.13	.14	.17	.27*	.20*	.19*	.25**	.22*	.14
Confidence	.06	.13	−.01	.29**	.27**	.20*	.25**	.32**	−.01	.24**	.24**	.19*
Student char.	.28**	.30**	.10	.37**	.31**	.37**	.46**	.43**	.22*	.45**	.36**	.38**
Studying beh.	.10	−.12	.06	−.12	.06	−.10	−.10	.05	−.12	−.12	−.15	.01
PRMS	.02	.11	−.07	−.03	.15	.12	.07	.13	−.05	.02	.11	−.07

Note. ACT score includes only self-reported ACT scores. ATR = attitudes toward research; PUR = perceived utility of research; PAS = perceived ability in statistics; APA = perceived ability to write in American Psychological Association style; Scientist = attitudes toward the scientific process; PAScience = Psychology as a Science Scale; Confidence = confidence in research skills; Student char. = student characteristics; Studying beh. = studying behavior; SocM. = social media; Present. = presentations; Ex. = examples; Discuss = discussions; Qz = quiz; GWK = group work, Part = participation; PRMS = Psychological Research Methods Survey.

*p < .05. **p < .01. ***p < .001.

We note few resources were associated with attitudes toward RM, perceptions of the utility of RM, and perceptions of ability in RM. Of the 11 resources, 9 were associated with positive attitudes about writing in APA style, 8 were associated with confidence in RM, and 6 were associated with positive attitudes to the scientific method. All methods positively correlated to the student characteristics, and none related to knowledge of RM.

What Predicts Student Learning in RM?

To paint a picture of the factors associated with learning RM, we examined the correlations between measures of student attitude toward science; RM; statistics, knowledge, and confidence surrounding RM; and student academic–related characteristics. Correlations between main student variables are shown in Table 5.

Table 5.

Correlations Between Major Variables Measured in the Student Survey.

Key Variables	ATR	PUR	PAR	APA	Scientist	PAS	Conf.	StuCh.	SBeh.	PRMS
ATR	—	.10	.16	.33**	.32**	.21*	.19*	.29**	.03	.03
PUR		—	.24**	.05	−.04	−.20*	.01	.09	−.08	−.10
PAR			—	.19*	.25**	.11	.14	.46***	.06	.08
APA				—	.44**	.19*	.42**	.46***	−.13	.06
Scientist					—	.49**	.39**	.43***	.13	.13
PAS						—	.31**	.38***	.02	.24**
Confidence							—	.38***	.10	.12
Student characteristics								—	.07	.13
Studying behaviors									—	−.00
Psychological Research Methods Survey										—

Note. ATR = attitudes toward research; PUR = perceived utility of research; PAR = perceived ability in statistics; APA = perceived ability to write in American Psychological Association style; Scientist = attitudes toward the scientific process; PAS = perceived ability in statistics; Confidence (Conf.) = confidence in research skills; StuCh. = student characteristics; SBeh. = studying behavior; PRMS = Psychological Research Methods Survey.

*p < .05. **p < .01. ***p < .001.

Student academic characteristics showed consistent positive and strong associations with a number of measures of attitudes toward RM (both perceived ability and for research subscales of the Sizemore & Lewandowski, 2009 measure), toward writing, the scientific method, and psychology as a science, and in general confidence in the ability to do RM. Attitudes toward writing, the scientific method, and psychology as a science also predicted confidence toward RM. Only one measure, views of psychology as a science, significantly correlated with RM knowledge.

Finally, we tested whether course design was associated with attitudes toward RM. We conducted a multivariate analysis of covariance (MANCOVA) on seven measures of RM attitudes and knowledge (attitudes toward RM, perceived utility of RM, perceived ability with RM, attitudes toward writing and the scientific method, confidence in RM, and RM knowledge), with course design (require student to conduct their own research or not) as the between-subjects factor, and ACT scores as a covariate. The MANCOVA showed no statistically significant differences between groups.

Discussion

Our study provides a detailed picture of both faculty and student attitudes toward the RM course in psychology. While some of our findings, such as those relating to challenges with teaching the course, are consistent with past research (e.g., Ciarocco et al., 2017), we take a nuanced look into hitherto unexplored terrain. In particular, we document some critical course design components, illustrate the ubiquity of important concepts, and unearth key associations between student attitudes, knowledge, and behaviors. Results can help instructors modify their course requirements and potentially highlight areas not previously focused on.

Key Design Elements

The absolute majority of instructors in our sample have students collect their own data and run their own research, whether experimental or correlational in nature. Likewise, most instructors have their students write a complete APA-style paper. Instructors who do these time-consuming activities can take solace in that many other instructors do so as well. Having to write APA-style papers had a strong association with self-reported learning, statistically correlating with a number of student measures including confidence in RM knowledge, and attitudes toward the scientific process. Having to take quizzes and read peer-reviewed research articles likewise correlated with a number of student measures. Conducting research in particular reflects a previously demonstrated best practice (e.g., Roberts & Allen, 2013) although the mere fact of doing so did not lead to higher levels of knowledge or more positive attitudes toward RM in this study. Analyses comparing students who completed their own research and those (albeit a smaller number) who did not were not significant.

The high number of classes having students conduct research suggests a belief in the utility of hands-on skills and is supported by past work. It is clear that RM can be effectively taught via class exercises that engage students as researchers, who systematically measure then analyze behavior, their own or of each other (Neumann et al., 2013). The use of such data helps them understand key methodological and statistical concepts and endorses their use in future classes. In one direct test, compared to students in a canned condition, students who conducted an experiment displayed significantly greater knowledge of the methodological and statistical issues addressed in class and were more confident regarding their ability to use this knowledge appropriately in the future (Allen & Baughman, 2016).

Does RM Have a Common Core?

In Health Psychology and Introductory Psychology, the cumulative research highlights the variety and inconsistency of topics, leading to the conclusion that students in different institutions, sometimes even within the same institution, may not be exposed to the same content (Homa et al., 2013; Panjwani et al., 2017). In contrast, our survey findings suggest there is high consistency in what is seen as important in the RM course. Of note is that the importance of major factors did not vary across training background of the instructors. Training in one area of psychology did not predict rating any area as more important than the other at the factor level.

Perhaps not surprisingly, rank was a significant factor but only in terms of statistics. Instructors of different rank did not vary in their ratings of basic knowledge, design, skills, or secondary topics but did vary in how important they saw statistics. Higher ranked instructors saw statistics as more important. Given that how important a faculty member sees a topic will reflect in how they approach it in class, this finding has important implications for faculty development as it suggests differential emphasis on statistics across instructors.

The ratings of importance of all major topics reflect content coverage in major textbooks and suggest how instructors may divide up class time. Going beyond the rating of importance listings which provides readers with a guide toward general instructor views on the topics, our open-ended questions provide particular food for thought. For example, one respondent noted a major challenge is “students lack-of critical thinking and poor writing along with lot of whining and low enthusiasm for independent work—they want to do minimal work and just be told what to do. Also low TA support for grading and teaching labs.” While the challenges (e.g., low student interest and time needed for coverage) resonate with past work (e.g., Sizemore & Lewandowski, 2009), we also highlight some successful strategies (e.g., application exercises and group projects).

Of note is our list of topics instructors feel are absent. Reflecting the contemporary gestalt of the field, instructors saw the replication crisis and evaluating news and research as being two areas lacking in RM textbooks. For example, one respondent said, “I would like more about the replicability crisis and concerns about methods and statistics in the field. A more realistic rather than idealistic representation of the field would be helpful.” Another commented, “I focus on developing the knowledge and skills needed to make evidence-based decisions about real world questions they face in the present and might face in the future (e.g., should group work be banned, should parents vaccinate their children).” Our closed and open-ended results can provide the instructor with valuable information to aid course planning and topic selection.

Our results also reflect a growing focus on preparing students for the workforce rather than solely preparing them for graduate school. Students and instructors do not perceive the RM course in the same way (Strohmetz et al., 2018), and student perceptions are valuable (Shaw et al., 2019). Instructors may tend to focus more on the role of the course as a preparation for graduate school ignoring the reality that most of the students who major in psychology do not go on for further study in the field. We saw evidence for some instructors recognizing the mismatch between student and instructor in our open-ended answers. For example, one respondent noted “I’m changing my opinion on how important it is that students need to be able to write a research report after realizing that the vast majority of students taking this course do not plan to go into a career in research.”

Predicting Student Learning

Our deep dive into student attitudes and knowledge highlights some important correlations together with some surprising findings. It is important to note the negative relationship between student ACT score and different course components. Our results suggest that students with lower ACT scores may learn more from presentations, group work, and class participation. Conversely, this suggests that perhaps those with higher ACT, hence higher cognitive ability, may prefer to work in more traditional ways (i.e., readings, independent work) While each of these three components are seen as pedagogically sound, paying attention to individual differences in ability may help the instructor better structure pedagogical design. For example, the instructor may want to use matched assignment in creating groups ensuring that ability levels within each group vary.

It is notable than only one of our student attitude measures, views of psychology as a science, predicted knowledge of RM. This association suggests that students who view psychology as a science are more likely to work harder in the course or may conversely be those who have a higher knowledge about the scientific process. The amount of time studied was not a significant predictor of RM knowledge nor did study behavior predict confidence or attitudes toward the RM and science.

Why is RM challenging to teach and to learn? In the future, research needs to examine more closely where students get stuck. One reason may be the presence of bottlenecks, points of a course where the learning of a significant number of students is disrupted (Middendorf & Pace, 2004). Little work addresses bottleneck concepts in psychology. Gurung and Landrum (2013) found that scales of measurement and selecting the appropriate statistic are bottleneck concepts, but this work was done in the context of an introductory psychology class. Some instructors have begun to identify difficult concepts in RM and are testing interventions such as screencasts and videos to alleviate the difficulties (Cross, 2018), but more work is needed. Future research needs to examine which of the important concepts taught in RM are potential bottlenecks. Faculty can correspondingly design pedagogies to capture them.

Limitations

Two critical limitations of the current research design are its correlational nature and our measurement of perceptions of usefulness and liking of features versus actual amounts of the same. For example, while it is tempting to speculate about how course elements drive learning, we acknowledge that the student measures may primarily reflect how more positive attitudes about RM and one’s perceived competencies are linked. Similarly, when discussing class resources, a favorable rating of one class element had a positive association with all other course elements—not necessarily showing that one element is better than any others. Our findings reveal a pattern of individual differences where course elements of RM are either consistently liked or consistently disliked. In addition, the relationship between student attitudes toward science and skills may lead them to favorably or unfavorably evaluate all elements of the course. Our findings are inconclusive as to whether elements of the course influence student attitudes and perceived skill or whether those with stronger perceived skills simply value the class more. It is possible that those who are more confident in their skills develop a halo effect toward all aspects of their RM course and rate the elements as more valuable than their peers who lack this confidence. Our research has created a basis for future, causal research looking at a potential relationship between the presence or absence of specific course components and learning outcomes rather than attitudes toward these components.

Despite our best efforts, our sample size was not substantial. While the spread of ranks, gender, and other demographics suggests we attained a diverse sample of participants, our external validity is constrained. Not all the instructors in our survey passed on the survey to their students, and an unequal number of students volunteered across universities. Combined with the often small size of RM classes, we were not able to examine differences in learning between classes or map instructor differences onto student learning differences. Research comparing students in classes with different designs may allow for the identification of better practices in teaching the class.

In general, our instructor ratings of important topics in RM set the stage for instructors to examine their coverage and the amount of time allocated in a class to the main areas. Our identification of areas needing more coverage and less coverage, together with our listing of some successful approaches (from instructors), and helpful aids (from students), should provide RM instructors with key information to enhance their teaching of this important course.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Addison

W. E.

Stowell

J. R.

Reab

M. D.

(2015). Attributes of introductory psychology and statistics teachers: Findings from comments on RateMyProfessors.com. Scholarship of Teaching and Learning in Psychology, 1, 229–234. http://dx.doi.org/10.1037/stl0000034

Allen

P. J.

Baughman

F. D.

(2016). Active learning in research methods classes is associated with higher knowledge and confidence, though not evaluations or satisfaction. Frontiers in Psychology, 7, Article 279. https://doi.org/10.3389/fpsyg.2016.00279

American Psychological Association. (2012). Guidelines for preparing high school psychology teachers: Course-based and standards-based approaches. https://www.apa.org/education/k12/teaching-guidelines.pdf

American Psychological Association. (2013). APA guidelines for the undergraduate psychology major: Version 2.0. Retrieved from http://www.apa.org/ed/precollege/undergrad/index.aspx

Amsel

Allen

Bauer

(2014, August). Psychological literacy: Its nature, acquisition, and application. Presented at the 6th International Conference on Psychology Education, Flagstaff, AZ.

Amsel

Baird

Ashley

(2011). Misconceptions and conceptual change in undergraduate students’ understanding of psychology as a science. Psychology Learning and Teaching, 10, 3–10. https://doi.org/10.2304/plat.2011.10.1.3

Ball

C. T.

Pelco

L. E.

(2006). Teaching research methods to undergraduate psychology students using an active cooperative learning approach. International Journal of Teaching and Learning in Higher Education, 17, 147–154.

Ciarocco

N. J.

Lewandowski

G. W.

Van Volkom

(2013). The impact of a multifaceted approach to teaching research methods on students’ attitudes. Teaching of Psychology, 40, 20–23. https://doi.org/10.1177/0098628312465859

Ciarocco

N. J.

Strohmetz

D. B.

Lewandowski

G. W.

Jr (2017). What’s the point? Faculty perceptions of research methods courses. Scholarship of Teaching and Learning in Psychology, 3, 116–131. https://doi.org/http://dx.doi.org/10.1037/stl0000085

10.

Cranney

Dunn

D. S.

(Eds.). (2011). Educating the psychologically literate citizen: Global perspectives. Oxford University Press. http://dx.doi.org/10.1093/acprof:oso/9780199794942.001.0001

11.

Cross

(2018). Just in time teaching: The effects of video explanations on student performance. Presentation at the Society for the Teaching of Psychology’s Annual Conference on Teaching, Phoenix, AZ.

12.

Dunn

D. S.

Brewer

C. L.

Cautin

R. L.

Gurung

R. A. R.

Keith

K. D.

McGregor

L. N.

Nida

S. A.

Puccio

Voigt

M. J.

(2010). The undergraduate psychology curriculum: Call for a core. In Halpern

D. F.

(Ed.), Undergraduate education in psychology: A blueprint for the future of the discipline (pp. 47–61, Chapter xvi, 227 Pages) American Psychological Association. http://dx.doi.org/10.1037/12063-003

13.

Earley

M. A.

(2014). A synthesis of the literature on research methods education. Teaching in Higher Education, 19, 242–253. http://dx.doi.org/10.1080/13562517.2013.860105

14.

Field

A. P.

(2010). Non-sadistical methods for teaching statistics. In Upton

Trapp

(Eds.), Teaching psychology in higher education (pp. 134–163). Wiley-Blackwell.

15.

Friedrich

(1996). Assessing students’ perceptions of psychology as a science: Validation of a self-report measure. Teaching of Psychology, 23, 6–13. https://doi.org/10.1207/s15328023top2301_1

16.

Goode

C. T.

Lamoreaux

Atchison

K. J.

Jeffress

E. C.

Lynch

H. L.

Sheehan

(2018). Quantitative skills, critical thinking, and writing mechanics in blended versus face-to-face versions of a research methods and statistics course. Teaching of Psychology, 45, 124–131. https://doi.org/http://dx.doi.org/10.1177/0098628318762873

17.

Gurung

R. A. R.

Hackathorn

Enns

Frantz

Cacioppo

J. T.

Loop

Freeman

J. E.

(2016). Strengthening introductory psychology: A new model for teaching the introductory course. American Psychologist, 71, 112–124. https://doi.org/http://dx.doi.org/10.1037/a0040012

18.

Gurung

R. A. R.

Landrum

R. E.

(2013). Bottleneck concepts in psychology: Exploratory first steps. Psychology Learning & Teaching, 12, 236–245. http://dx.doi.org/10.2304/plat.2013.12.3.236

19.

Gurung

R. A. R.

Weidert

Jeske

A. S.

(2010). A closer look at how students study (and if it matters). Journal of the Scholarship of Teaching and Learning, 10, 28–33.

20.

Homa

Hackathorn

Brown

C. M.

Garczynski

Solomon

E. D.

Tennial

Sanborn

U. A.

Gurung

R. A. R.

(2013). An analysis of learning objectives and content coverage in introductory psychology syllabi. Teaching of Psychology, 40, 169–174. http://dx.doi.org/10.1177/0098628313487456

21.

Manning

Zachar

Ray

LoBello

(2006). Research methods courses and the scientist and practitioner interests of psychology majors. Teaching of Psychology, 33, 194–196.

22.

McGovern

T. V.

Corey

Cranney

Dixon

W. E.

Jr Holmes

J. D.

Kuebli

J. E.

Ritchey

K. A.

Smith

R. A.

Walker

S. J.

(2010). Psychologically literate citizens. In Halpern

D. F.

(Ed.), Undergraduate education in psychology: A blueprint for the future of the discipline (pp. 9–27, Chapter xvi, 227 Pages) American Psychological Association. http://dx.doi.org/10.1037/12063-001

23.

Middendorf

Pace

(2004). Decoding the disciplines: A model for helping students learn disciplinary ways of thinking. In Pace

Middendorf

(Eds.), Decoding the disciplines: Helping students learn disciplinary ways of thinking (pp. 1–12). Jpssey-Bass.

24.

Murtonen

(2005). University students’ research orientations: Do negative attitudes exist toward quantitative methods? Scandinavian Journal of Educational Research, 49, 263–280. https://doi.org/10.1080/00313830500109568

25.

Murtonen

(2015). University students’ understanding of the concepts empirical, theoretical, qualitative and quantitative research. Teaching in Higher Education, 20, 684–698. https://doi.org/10.1080/13562517.2015.1072152

26.

Murtonen

Olkinuora

Tynjala

Lehtinen

(2008). “Do I need research skills in working life?” University students’ motivation and difficulties in quantitative methods courses. Higher Education, 56, 599–612. https://doi.org/10.1007/s10734-008-9113-9

27.

Neumann

Hood

Neumann

M. M.

(2013). Using real-life data when teaching statistics: Student perceptions of this strategy in an introductory statistics course. Statistics Education Research Journal, 12, 59–70.

28.

Norcross

J. C.

Hailstorks

Aiken

L. S.

Pfund

R. A.

Stamm

K. E.

Christidis

(2016). Undergraduate study in psychology: Curriculum and assessment. American Psychologist, 71, 89–101. https://doi.org/10.1037/a0040095

29.

Panjwani

A. A.

Gurung

R. A. R.

Revenson

T. A.

(2017). The teaching of undergraduate health psychology: A national survey. Teaching of Psychology, 44, 268–273. http://dx.doi.org/10.1177/0098628317712786

30.

Rajecki

D. W.

Appleby

Williams

C. C.

Johnson

Jeschke

M. P.

(2005). Statistics can wait: Career plans activity and course preferences of American psychology undergraduates. Psychology Learning & Teaching, 4, 83–89. http://dx.doi.org/10.2304/plat.2004.4.2.83

31.

Roberts

L. D.

(2016). Editorial: Research methods pedagogy: Engaging psychology students in research methods and statistics. Frontiers in Psychology, 7, 3.

32.

Roberts

L. D.

Allen

P. J.

(2012). Student perspectives on the value of research participation. In McCarthy

Dickson

K. L.

Cranney

Trapp

Karandashev

(Eds.), Teaching psychology around the world (Vol. 3, pp. 198–211). Cambridge Scholars.

33.

Roberts

L. D.

Allen

P. J.

(2013). A brief measure of student perceptions of the educational value of research participation. Australian Journal of Psychology, 65, 22–29. https://doi.org/10.1111/ajpy.12007

34.

Rottinghaus

P. J.

Gaffey

A. R.

Borgen

F. H.

Ralston

C. A.

(2006). Diverse pathways of psychology majors: Vocational interests, self-efficacy, and intentions. The Career Development Quarterly, 55, 85–93. https://doi.org/10.1002/j.2161-0045.2006.tb00007.x

35.

Shaw

T. J.

Yang

Nash

T. R.

Pigg

R. M.

Grim

J. M.

(2019). Knowing is half the battle: Assessments of both student perception and performance are necessary to successfully evaluate curricular transformation. PLoS One, https://doi.org/10.1371/journal.pone.0210030

36.

Sizemore

O. J.

Lewandowski

G. W.

(2009). Learning might not equal liking: Research methods course changes knowledge but not attitudes. Teaching of Psychology, 36, 90–95. https://doi.org/10.1080/00986280902739727

37.

Stoloff

McCarthy

Keller

Varfolomeeva

Lynch

Makara

Simmons

Smiley

(2010). The undergraduate psychology major: An examination of structure and sequence. Teaching of Psychology, 37, 4–15. https://doi.org/10.1080/00986280903426274

38.

Strohmetz

D. B.

Ciarocco

N. J.

Lewandowski

G. W.

Jr (2018). Getting on the same page: Aligning student and instructor perception to facilitate learning in methods classes. Presentation at the Society for the Teaching of Psychology’s Annual Conference on Teaching, Phoenix, AZ.

39.

Sümer

(2016). Rapid growth of psychology programs in Turkey: Undergraduate curriculum and structural challenges. Teaching of Psychology, 43, 63–69. http://dx.doi.org/10.1177/0098628315620886

40.

Vittengl

J. R.

Bosley

C. Y.

Brescia

S. A.

Eckardt

E. A.

Neidig

J. M.

Shelver

K. S.

Sapenoff

L. A.

(2004). Why are some undergraduates more (and others less) interested in psychological research? Teaching of Psychology, 31, 91–97. http://dx.doi.org/10.1207/s15328023top3102_3