Potential Potholes: Predicting Challenges and Learning Outcomes in Research Methods in Psychology Courses

Abstract

Identifying impediments to learning can help both instructors and students. In this study students (N = 222) from across the nation listed concepts from research methods they found the most challenging—“potholes” or impediments to learning, and also rated the difficulty of various concepts from the class. We also measured student attitudes and perceptions of the course. Our results showed validity is one of the hardest concepts for students to understand in RM. Not surprisingly, within a list of concepts in research methods, students tend to be tripped up by terms of similar grouping. Perceptions of a research methods course was strongly related to the students’ attitudes and canonical correlation analysis revealed several important findings from our data set. For example, we found that when students perceive more value, expectancy for success, cognitive strategies, and resources management strategies, they are less likely to be challenged by the concepts of samples and variables and threats to internal validity. Our findings provide a clear map of student potholes in research methods courses and suggest ways to change student attitudes about the same.

Keywords

research methods psychology curriculum undergraduate training

Research methods (RM) in psychology, either as a dedicated course or embedded in introductory psychology courses, is required of all psychology majors (Roberts, 2016). The American Psychological Association (2013) highlights that one of the five learning goals for undergraduate psychology majors is “scientific inquiry and critical thinking,” and that “students completing foundation-levels course should learn basic skills and concepts in interpreting behavior, studying research, and applying research design principles to draw conclusions about psychological phenomena” (p. 15). RM instructors who are often seasoned researchers certainly understand this important learning goal and are often excited about teaching RM, a course they know undergraduate students can benefit from taking (Gurung & Stoa, 2020). However, student perceptions toward RM are often negative, characterized by heightened anxiety due to perceived difficulty of RM (Harlow et al., 2009), low enjoyment in conducting research (Vittengl et al., 2004), and devaluation of research and statistical skills (Ciarocco et al., 2013; Sizemore & Lewandowski, 2009). Challenging concepts in RM courses, student attitudes toward RM, and perception of skills (e.g., experimental designs, statistics) learned from RM courses may explain the disconnect between instructor and student attitudes to RM.

RM is a unique course compared to other psychology courses such as clinical psychology and personality psychology. Concepts in RM, such as different types of validity and research design, are abstract and may be hard to relate to, because students often do not have hands-on research experience and cannot see the research application in the real world (Ciarocco et al., 2013; Vittengl et al., 2004). Challenges in teaching RM stem from students’ inability to (a) think through proper hypotheses, randomization, control conditions, double-blind testing, counterbalancing, power, sample size, experimenter effects, and demand characteristics for experimental studies, and (b) implement controls for non-experimental studies (McKelvie & Standing, 2018). Students may perceive certain topics to be “potholes,” impediments on their road to learning the material. No empirical research to date has examined specific RM concepts that students perceive challenging. What are students’ potholes? Understanding the challenging concepts in RM from a student perspective, as well as measuring levels of perceived difficulty, can inform effective course design which can help students tackle these challenges successfully.

In addition to challenges in teaching and learning RM, student attitudes and course design may influence learning outcomes in RM and thus explain the disconnect between instructors’ and students’ expectations. Instructors typically assume that RM promotes scientific and critical thinking, yet these learning outcomes might be lost on students with negative attitudes toward statistics and research (Ciarocco et al., 2013; Sizemore & Lewandowski, 2009). Specifically, Sizemore and Lewandowski (2009) found that after completing a RM in psychology course, students of different abilities all perceived research and statistics as less useful than they did before the course. Unexpectedly, the students who perceived a higher ability in RM had even worse attitudes toward research and statistics.

Additional research has now delved into how course design can potentially contribute to the decline in student attitudes toward RM. Several experimental and quasi-experimental studies indicated that the implementation of pedagogical tools, such as applied projects (Harlow et al., 2009), replication projects (McKelvie & Standing, 2018), and multifaceted active learning activities (Ciarocco et al., 2013), could enhance student attitudes and motivation toward RM. Although these studies revealed a close link between student attitudes and course design, it is unclear as to whether student attitudes and course design both play a role in student perceptions of challenging concepts and learning outcomes (e.g., levels of learning and enjoyability).

Different institutions and instructors have different requirements regarding research design (e.g., allow only experimental or descriptive and correlational as well), data collection (e.g., having students collect data or provide data), and data analysis (e.g., statistical software, Gurung & Stoa, 2020). Because there are unique challenges and student attitudes associated with data collection and certain research designs (McKelvie & Standing, 2018; Roberts, 2016), investigating these factors can inform better teaching practices in RM for tackling undergraduate psychology students’ roadblocks on the journey to achieve the “scientific injury and critical thinking” learning goal (APA, 2013).

Our study sought to uncover student potholes to learning in RM with an aim to address where and why students get tripped up, as well as how we may give them a hand. Specifically, we sought to answer the following research questions:

What are the potholes (challenging concepts) in RM courses from student perspective?

How do student attitudes (e.g., goals and value beliefs) and course design relate to perceived difficulty of potholes in RM courses?

What is the association between potholes, student attitudes, and learning outcomes in RM courses?

Method

Participants

We collected data nationwide from U.S. undergraduate students enrolled in a RM course in psychology at their corresponding universities. Our sample consisted of 222 students, broken down into 56 females, 17 males, one other, and 148 who did not respond to this question. Students ranged in age from 18 to 53 (M = 21.48; SD = 5.33). The largest percentage of the students in this sample who reported ethnicity were either White (23.0%) or Asian (6.8%). Participants’ previous high school GPA ranged from 2.00 to 4.89 (M = 3.67; SD = .50). We did not collect data on where the participants were enrolled in school.

Materials

To gather participant perceptions of the potholes or most challenging concepts in RM, we asked participants to write down up to five concepts that they found challenging. Participants then rated the difficulty of 63 research methods concepts (e.g., confound, effect size) using a scale of 1 (not at all) to 5 (a great deal). These concepts were derived from two past studies of difficult terms in RM (Gurung & Landrum, 2013; Gurung & Stoa, 2020). Specifically, Gurung and Stoa (2020) asked instructors to rate the importance of teaching 25 topics (e.g., sources of information, ethics, APA style writing, confounding variables). Gurung and Landrum (2013) had students rate the difficulty of and their confidence in their own knowledge of 107 terms relating to introductory psychology RM and learning. We removed overlaps to select the final set of terms relating to RM. We list all terms in Table 1.

Table 1.

Example Statements and Reliability of the Motivated Strategies for Learning Questionnaire Subscales.

Scale	M	SD	Reliability
MOTIVATION SCALES
Value Component: Intrinsic Goal Orientation	5.04	0.98	.77
In a class like this, I prefer course material that really challenges me so I can learn new things.
Value Component: Extrinsic Goal Orientation	5.60	1.09	.77
Getting a good grade in this class is the most satisfying thing for me right now.
Value Component: Task Value	5.68	0.88	.88
I think I will be able to use what I learn in this course in other courses.
ALL VALUE COMPONENTS	5.47	0.74	.86
Expectancy Component: Control of Learning Beliefs	5.27	0.96	.68
If I study in appropriate ways, then I will be able to learn the material in this course.
Expectancy Component: Self-Efficacy for Learning and Performance	5.20	1.04	.90
I believe I will receive an excellent grade in this class.
ALL EXPECTANCY COMPONENTS	5.22	0.88	.88
Affective Component: Test Anxiety	4.70	1.34	.84
When I take the test I think about how poorly I am doing compared with other students.
LEARNING STRATEGY SCALES
Cognitive and Metacognitive Strategies: Rehearsal	5.07	1.01	.60
When I study for this class, I practice saying the material to myself over and over.
Cognitive and Metacognitive Strategies: Elaboration	5.05	0.82	.73
When I study for this class, I pull together information from different sources, such as lectures, readings, and discussions.
Cognitive and Metacognitive Strategies: Organization	5.00	1.13	.69
When I study the readings for this course, I outline the material to help me organize my thoughts.
Cognitive and Metacognitive Strategies: Critical Thinking	4.34	1.05	.77
I often find myself questioning things I hear or read in this course to decide if I find them convincing.
Cognitive and Metacognitive Strategies: Metacognitive Self-Regulation	4.71	0.70	.73
When reading for this course, I make up questions to help focus my reading.
ALL COGNITIVE AND METACOGNITIVE STRATEGIES	4.80	0.68	.90
Resource Management Strategies: Time and Study Environment	5.21	0.99	.84
I usually study in a place where I can concentrate on my coursework.
Resource Management Strategies: Effort Regulation	5.11	0.70	−.46
I work hard to do well in this class even if I don’t like what we are doing.
Resource Management Strategies: Peer Learning	3.87	1.32	.64
When studying for this course, I often try to explain the material to a classmate or a friend.
Resource Management Strategies: Help Seeking	4.13	1.09	.53
I ask the instructor to clarify concepts I don’t understand well.
ALL RESOURCE MANAGEMENT STRATEGIES	4.75	0.65	.75

We used the Motivated Strategies for Learning Questionnaire (MSLQ, Pintrich et al., 1991) to assess student attitudes. This survey consists of 81 items, broken into 31 items assessing student goals and value beliefs of a course, skill to succeed in a course, and anxiety about tests, 31 items examining student use of different cognitive and metacognitive strategies, and 19 items addressing student management of different resources. In total, there are five major subscales broken down further into 15 different subscales. Participants completed this self-report on a scale of 1 (not at all like me) to 7 (very like me). We list the subscales, example statements, and means in Table 1.

We asked students to select one or more choices regarding how their course was conducted: (a) students conduct their own research (experimental design required), (b) students conduct their own research (correlational design required), (c) students use data provided (original research not conducted by students), and (d) no data analyzed by students. We also asked what statistics program participants used.

Finally, we measured general demographics and perceived learning outcomes. We asked how much the participants felt they learned from the course, how enjoyable the course was, and how difficult the course was, using a scale ranging from 1 (none at all, extremely unenjoyable, or extremely easy) to 5 (a great deal, extremely enjoyable, or extremely difficult). We also asked students to list their recalled high school GPA and ACT scores, as well as their best guess at their final grade in the course.

Procedure

After the study was approved by the university’s Institutional Review Board, we used electronic mail invitations to instructors who had signed up on the Hub for Intro Psych and Pedagogical Research (HIPPR), and invited reviewers for SAGE Publishing who had given their consent, as well as other known instructors of RM in psychology across the U.S. Instructors were asked to pass the survey on to their students in their RM courses. All surveys were online and created using Qualtrics software. Participants first read a consent form, then wrote in the concepts they found challenging, rated the difficulty of concepts, completed the MSLQ, questions on course design, and general demographics. The entire survey took approximately 15 minutes. There was no reward for participation in the study but instructors passing on the survey could allow students to gain course research credit for participation. We did not assess whether rewards were provided.

Results

Determining Student Potholes

We asked students to give us up to five concepts that they felt challenged by in RM. Students wrote in a total of 225 items. After eliminating duplicates, 100 unique terms remained. Students mentioned types of validity most often, peaking at 21% (48 of the write-ins). This includes construct validity, which by itself was one of the most frequent replies, reaching 5.3% of all responses. Students also commonly noted the following concepts: quasi-experiments (5.8%), general knowledge of statistics (4.8%), and operational definition (4%). To visualize this data, we created a word cloud of all terms, with the larger words being the terms that occurred more often (Figure 1).

Figure 1.

Word Cloud of Challenging Terms.

When given a list of terms from RM, students rated quasi-experiments (M = 2.60; SD = 1.02) and operational definition (M = 2.47; SD = 1.07) in the top five most difficult concepts. However, semantic differential formats (M = 2.74; SD = 1.25) and Cronbach’s α (M = 2.73; SD = 1.15) now topped the list. The top five difficult concepts also included parsimony (M = 2.40; SD = 1.15). The means and standard deviations of all 63 difficult terms can be found in Table 2.

Table 2.

Means of Term Difficulty Ratings.

Term	M	SD
Semantic Differential Formats	2.74	1.26
Cronbach’s α	2.73	1.16
Quasi-experiment	2.60	1.03
Operational definition	2.47	1.07
Parsimony	2.40	1.16
Directionality or third-variable problem	2.36	1.11
Belmont report	2.34	1.14
Effect size	2.33	1.20
Between groups design	2.33	1.07
Margin of error	2.28	1.06
Systematic variability	2.25	1.00
Confound	2.25	1.16
Covariance	2.24	1.13
Manipulation checks	2.21	1.02
Empiricism	2.21	1.10
Critical thinking	2.20	1.22
Extraneous variable	2.19	1.10
Response sets	2.19	1.16
Within groups design	2.18	1.09
Regression to the mean	2.18	1.07
Demand characteristics	2.15	1.06
Correlation coefficient	2.12	1.14
Counter balancing	2.08	1.00
Instrumentation threats	2.08	1.06
Maturation	2.07	1.09
Selection effect	2.05	0.98
APA ethical principles	2.03	1.04
Deception (two types)	2.02	0.96
Data Fabrication vs. Falsification	2.02	1.03
Attrition	2.02	1.17
Carryover, practice, and order effects	2.02	1.10
Temporal precedence	2.02	1.06
History threat	1.99	1.16
Subject variables	1.99	1.06
Ceiling and floor effects	1.97	0.91
Interval scale	1.89	1.00
Null hypothesis	1.89	1.11
Correlation	1.88	1.00
Confirmation bias	1.86	1.16
Experiment	1.84	0.77
Ratio scale	1.83	0.99
Confederate	1.81	1.10
Ethics	1.80	1.10
Single blind procedure	1.78	0.95
Double blind procedure	1.77	1.08
Hypothesis	1.76	1.11
Qualitative approach	1.74	0.97
Case study	1.71	0.99
Random sampling	1.71	1.00
Quantitative approach	1.67	0.91
Scientific method	1.67	1.01
Double blind study	1.66	1.04
Independent variable	1.63	1.05
Dependent variable	1.59	1.02
Random assignment	1.58	0.98
Outliers	1.57	0.99
Debriefing	1.55	1.00
Scatterplot	1.48	0.90
Population	1.46	0.92
Sample	1.44	0.88
Informed consent	1.44	0.92
Placebo	1.41	0.92
Confidentiality	1.41	0.92

Because we expected many of these terms to be related thematically, we conducted a principal components analysis with varimax rotation to determine if there were smaller sets of factors in the 63 topics, acknowledging that our small sample size might limit the interpretations of any findings. With all 63 topics, the Kaiser-Meyer-Olkin measure was slightly above the commonly acceptable threshold of .50 (KMO = .604), suggesting widespread correlations within our data which might not be a problem for a factor analysis, but should be interpreted with caution (Cerny & Kaiser, 1977). The Bartlett’s Test of Sphericity, which tests the overall significance of all the correlations within the correlation matrix, was significant, χ² (1953) = 5144.44, p < .000, indicating that the variables were related, and that meaningful factors might emerged from our data. We performed an initial analysis with the criterion set at eigenvalue > 1 and examined the rotated factor matrix using varimax rotation. Fifteen factors had eigenvalues over Kaiser’s criterion of 1 and in combination explained 82.1% of the variance. The scree plot revealed a justification to retain three, four, or five factors. We retained five factors as they were deemed thematically appropriate. These five factors explained 57.2% of the variance.

Given our small sample we only used the results of the principle components analysis as a starting point to inductively label the five factors for further analysis: (a) Factor 1—items related to Samples and Variables (SV) (terms included random assignment, population, sample, dependent variable, random sampling, independent variable, scatterplot, quantitative approach, scientific method, outliers, subject variables, and confounds, dependent variable, independent variable, scatterplot), (b) Factor 2—items related to Ethics and Theory Data Cycle (ET)(terms include debriefing, informed consent, confidentiality, double blind procedure, ethics, hypothesis, correlation, experiment, critical thinking, case study, and confirmation bias), (c) Factor 3—items related to Threats to Internal Validity (Threats)(terms include attrition, regression to the mean, instrumentation threats, maturation, history threat, demand characteristics, quasi-experiment), (d) Factor 4—items related to Design Confounds (terms include systematic variability, selection effect, operational definition, extraneous variable, and directionality or third-variable problem), and (e) Factor 5—items related to Scale Measurements (terms include semantic differential formats, Cronbach’s α, response sets, margin of error, and correlation coefficient).

We combined items within each factor into an aggregate score for further analyses. Each factor showed high internal reliability: SV Cronbach’s α = .95; ET Cronbach’s α = .93; Threats Cronbach’s α = .91; Design Cofounds Cronbach’s α = .84; and Scale Measurements Cronbach’s α = .83. Bivariate correlations showed that factors were all moderately correlated (r(91) ranged from .26 to .59, p < .05) with each other. This factor analysis was largely exploratory, and these findings should be interpreted with caution only serving as a guide until tested with a larger sample size. The complete factor loading with terms can be found here: https://osf.io/87hr6/?view_only=c0dda905c05a4022bdc3e56f641b9fbf.

Predicting Potholes in Research Methods

We first tested whether the perceived difficulty of the five factors were related to student attitude, ability, and course design. To test if attitudes for learning associated with perceived challenge in RM, we conducted a bivariate correlation using our five factors (SV, ET, Threats, Design Confounds, and Scale Measures) and the five large components of the MSLQ. Nearly all MSLQ components were negatively correlated with our factors. As the student attitude factor increased, the student tended to find the concepts of those factors less challenging. No significant correlations were found between ET and any of the MSLQ components, nor Affective Components and MSLQ. Complete correlations can be found in Table 3.

Table 3.

Correlations of MSLQ Subscales and Factor Loadings.

	SV	ET	Threats	Design Confounds	Scale Measure
Value Components	−.29*	−.15	−.25*	−.24*	−.13
Expectancy Components	−.33**	−.17	−.45**	−.34**	−.10
Affective Components	.24	−.18	.15	.02	−.00
Cognitive and Metacognitive Strategies	−.24*	−.05	−.34*	−.14	−.05
Resource Management Strategies	−.21	−.10	−.36**	−.16	.17

Notes: SV = Samples and Variables, ET = Ethics and Theory Data Cycle, Threats = Threats to Internal Validity; * p < .05; ** p < .01.

Next, we tested if course design correlated with perceived understanding of difficult terms. We conducted a multivariate analysis of variance (MANOVA) with our created factors (SV, ET, Threats, Design Confounds, and Scale Measurements) as the dependent variable and course design as the independent variable. Course design was categorized as one of three levels: Having taken a course where the student conducted their own research (either experimental or correlational), a course where data was provided for analysis, or a course where no data analysis was required. No significant differences were found.

Finally, to test the associations between ability and these factors, we performed bivariate correlations between the factors that we created and self-reported high school GPA and ACT scores. None of the factors were significantly correlated with ACT scores or GPA.

Testing Associations Between Potholes, Attitudes, and Course Perceptions

First, we tested if potholes related to perceptions of the RM course. We performed correlational analyses between the five created factors and enjoyment, difficulty, and amount learned. There was a negative correlation between threats and amount learned (r(73) = −.24, p = .039), and design confounds and amount learned (r(73) = −.38, p = .001), indicating an association between understanding these topics and overall learning in the course. An ANOVA with anticipated final grade as the independent variable and the five challenge factors as the dependent variables revealed a significant finding between SV, F(2, 72) = 3.69, p = .030, Threats, F(2, 72) = 10.95, p < .001, and Design Confounds, F(2, 72) = 3.96, p = .023. A Tukey post hoc test revealed significant differences in the difficulty of Threats; specifically between C (2.91 ± .92) and A(1.82 ± .61, p < .000) and B (2.29 ± .63) and A(1.82 ± .61, p = .012) and Design Confounds between grades C (2.94 ± .86) and A (2.06 ± .76, p = .020). While we did not have any students report anticipating an D or F in their research methods course, our findings were nonetheless significant: The students who had a better understanding of the terms in those groups tended to feel more confident in achieving a higher grade in the course.

To examine how the five major components of the MSLQ were related to course perceptions, we conducted three simple linear regressions for each student perception of the course (difficulty, learning, enjoyability). We entered the five MSLQ subscales as a single block for this analysis. The model significantly explained 11% of the variance in difficulty, yet only variable expectancy components significantly predicted perceived difficulty (β = −.429, p = .018). Similarly, we conducted a linear regression model to predict learning from the course with the same subscales. The model significantly explained 21.0% of the variance in learning, whereas only value components significantly predicted perception of learning (β = .654 p < .001). There were no significant findings regarding the MSLQ and enjoyability. Correlations between the MSLQ scales and perceptions of the course are displayed in Table 4 (correlations of 15 subscales are available on the OSF site).

Table 4.

Correlations of MSLQ Subscales and Perceptions of Course.

	Learning	Enjoyability	Difficulty
Value Components	.44**	−.10	−.08
Expectancy Components	.26*	.04	−.29*
Affective Components	−.04	.05	.14
Cognitive/Metacognitive Strategies	.19	−.07	−.08
Resource Management Strategies	.16	−00	.00

Notes: * p < .05; ** p < .01.

To evaluate the multivariate shared relationship between potholes, attitudes, and course perceptions, we conducted a canonical correlation analysis using the five MSLQ large components, the five pothole factors, and learning, enjoyability, and difficulty variables. The analysis yielded five functions with squared canonical correlations ( $R_{c}^{2}$ ) of .470, .277, .160, .065, and .013 for each successive function. Collectively, the full model across all functions was statistically significant using the Wilks’s λ = .296 criterion, F(40, 264.33) = 2.12, p < .000. Because Wilks’s λ represents the variance unexplained by the model, 1 − λ yields the full model effect size in an r² metric. Thus, for the set of five canonical functions, the r² type effect size was .686, which indicates that the full model explained a substantial portion, about 68.6%, of the variance shared between the variable sets. The dimension reduction analysis allows the researcher to test the hierarchal arrangement of functions for statistical significance. As noted, the full model (Functions 1–5) was statistically significant, but the other functions (2–5, Wilks’s λ = .560, F(28, 221.36) = 1.38, p = .106, 3–5, Wilks’s λ = .775, F(18, 175.85) = .775, p = .555, 4–5, Wilks’s λ = .923, F(10, 126) = .515, p = .877, and 5 alone, Wilks’s λ = .987, F(4, 64.00) = .206, p = .934, were not.

Given the effects for each function, only the first three functions were considered noteworthy in the context of this study (47.0%, 27.7%, and 16.0% of shared variance, respectively), even though only function one was statistically significant. The last two functions only explained 6.5% and 1.3%, respectively, of the remaining variance in the variable sets after the extraction of the prior functions.

Table 5 presents the standardized canonical function coefficients, structure coefficients, and squared structure coefficients for each variable in Function 1, Function 2, and Function 3. Looking at the Function 1 coefficients, one sees that relevant criterion variables were primarily threats to internal validity, samples and variables, and design confounds. This conclusion was supported by the squared structure coefficients. These variables also tended to have the larger canonical function coefficients. Furthermore, the structure coefficients of threats and samples and variables had the same sign, indicating that they were positively related. Design confounds were inversely related to the other factors.

Table 5.

Canonical Solution for MSLQ predicting Learning Outcomes for Function 1, 2, and 3.

	Function 1			Function 2			Function 3
Variable	Coef	r_s	$r_{s}^{2}$ (%)	Coef	r_s	$r_{s}^{2}$ (%)	Coef	r_s	$r_{s}^{2}$ (%)	h₂
Learning	−.246	−.339	11.4	.878	.709	50.3	−.253	−.418	17.5	79.2
Enjoyable	−.003	−.032	1.0	−.371	−.447	19.9	−.126	−.083	.69	3.6
Difficult	.387	.339	11.4	.094	.120	1.4	.409	.253	6.4	19.2
Samples and Variables	.676	.454	20.6	−.374	−.114	1.3	−.382	.330	10.9	32.8
Ethics and Theory Data Cycle	−.263	.111	1.2	−.429	.133	1.8	.964	.634	40.2	43.2
Threats to Internal Validity	.987	.732	53.6	.356	.220	4.8	−.540	−.005	0.0	58.4
Design Confounds	−.425	.393	15.4	.389	.015	.02	.628	.497	24.7	40.1
Scale Measurements	−.553	.121	1.5	−.340	−.048	.23	−.253	.079	.62	2.4
$R_{c}^{2}$	47.0	27.7	16.0
Value Components	.230	−.544	29.6	1.433	.666	44.3	−.318	−.330	10.9	84.6
Expectancy Components	−.759	−.914	83.5	−.790	.069	0.47	−.827	−.386	14.9	98.8
Affective Components	.227	.408	16.6	−.458	−.178	.03	−.759	−.524	27.4	44.0
Cognitive and Metacognitive Strategies	−.373	−.700	49.0	.051	.168	.03	.807	.201	4.1	53.1
Resource Management Strategies	−.147	−.523	27.4	−.534	−.019	0.03	.105	.153	2.3	29.7

Note: Structure coefficients (r_s) greater than |.45| are bolded. Communality coefficients (h₂) greater than 45% are bolded. Coef = standardized canonical function coefficient; r_s = structure coefficient; $r_{s}^{2}$ = squared structure coefficient; h₂ = communality coefficient.

Regarding the predictor variable set in Function 1, expectancy components and cognitive and metacognitive strategies were the primary contributors to the predictor synthetic variable, with a secondary contribution by value components. Because the structure coefficient for expectancy and cognitive and metacognitive strategies was negative, these were positively related to samples and variables and threats to internal validity.

Discussion

As far as we are aware, no other research has collected a self-report of challenging terms for students in research methods. While previous research has asked students to rate the difficulty of such terms (e.g. Gurung & Landrum, 2013), this study is the first to ask students to come up with these terms on their own. To further predict challenges, we examined how student attitudes played a role in learning and the effect of course design. Results may help instructors focus on certain topics that appear to be more difficult for students to grasp, as well as strategize ways to strengthen attitudes toward learning.

While instructors may seem disheartened that students found so many terms difficult, the mere fact that they could recall a term and rate challenge of terms indicated some level of learning. Terms that the students found challenging seemed to be related, whether terms be categorized as samples and variables, ethics and theory data cycle, threats to internal validity, design confounds, or scale measurements. Combining the list of terms and write-ins, it seems that validity is one of the hardest concepts for students to understand in RM. This should be of great concern, considering the importance of validity to comprehending quality research. Whether a student goes on to conduct research or be a consumer of research, a strong understanding of the definition, types, and threats of validity is essential. Our findings suggest better practices for teaching and learning the types and threats to validity are needed.

When given a list of concepts in research methods, students tend to be tripped up by terms of similar grouping. We found those groups to be samples and variables, ethics and theory data cycle, threats to internal validity, design confounds, and scale measurement. These factors indicate a need for instruction for the terms within a group—particularly if a student has trouble with one term in the group, they are likely to have issues with another. This could be potentially beneficial in recognizing where students may struggle even if they themselves do not recognize it. Table 2 discloses each of the measured terms from most difficult to least difficult. These terms can now be given more attention in the classroom if need be.

Perceptions of a RM course was strongly related to the students’ attitudes. Attitudes were significantly correlated with a better perception of understanding of the terms in this course. Unsurprisingly, and aligning with previous research (Gurung & Stoa, 2020; Hartley & Hartley, 1976), attitudes toward the course, self-efficacy, and the belief that the student has the resources to succeed predicts less perceived challenges in RM. Our findings suggest that perhaps instructors should clarify the value of taking a research methods course to the average students’ everyday life. Doctoral and masters programs involve training in the research process, and instructors may be conditioned to believe that the key reason for students to take research methods is to understand the research process; however, if instructors are reminded that future preparation is a purpose of taking a research methods course, it is then considered the top reason (Ciarocco et al., 2017). While instructors may have high hopes that every student will go into statistics or research, students are more likely to enter a role where they will not conduct research every day. Nonetheless, it is important to know how to consume research. By connecting the dots of the content in a research methods course to everyday life, future research may find that the value component of attitude increases.

The canonical correlation analysis revealed several important findings from our data set. The first function suggests that when students perceive more value, expectancy for success, cognitive strategies, and resources management strategies, they are less likely to be challenged by the concepts of samples and variables and threats to internal validity. The second function suggests that students who perceive more value also believe they are learning more in the course, whereas the third function found that students who have greater affective components find ethics and theory data cycles less difficult.

Our findings suggest that the ideal RM student would have strong intrinsic goal orientation, task value, control of learning beliefs, and self-efficacy for learning and performance. Unfortunately, this is not how all students come to class. To improve these characteristics, instructors may focus on student autonomy and the ability to learn and apply concepts to everyday life, as establish strong inquiry skills (King, 1995). For example, exposing undergraduate students to research practices after taking a RM course (i.e. as a teaching assistant or research assistant) increases confidence, mastery of subject matter, and mentorship to other students (Thieman et al., 2009). As most students do not go on to graduate school, it is important to describe what employable skills they can garnered from this course. Instructors may do this by listing specific skills practiced in the syllabus, talking about these marketable skills on the first day of class, listing the skills practiced on each assignment, and talking about what skills were practiced on the last day of class, linking these skills directly to things they did in class as well as encouraging students to add these skills to their resume (Strohmetz et al., 2018). Focusing on integration of course material and study skills into other courses and daily life could strengthen the attitudes of students, as well as develop habits of inquiry.

Overall, student attitudes explained medium to large amount of variance in learning outcomes in RM. Predicting variables of understanding topics in research methods included high expectancy and cognitive and metacognitive strategies. Students who believe they have the necessary abilities to succeed, as well as implement study strategies such as peer learning, help seeking, finding a productive study environment, time management, and effort regulation skills tend to understand concepts in the course. For instructors, this means that it may be worth spending some time discussing studying strategies with students.

While not the primary motivation of this study, our findings regarding ACT and high school GPA are nonetheless notable. Neither ACT scores or high school GPA predicted difficulty or enjoyability of a RM course. While it has been found that high school GPA is more useful than admission test scores (such as the ACT) in predicting college GPA (Amelang, 1976; Sawyer, 2013), neither GPA or ACT score predicted success in a RM course. Several factors may have played into this, particularly that our sample tended to be high achievers, believing to receive either an A, B, or C in their RM course. A larger variety of grades would have been ideal for further analysis.

While we were still able to collect data from a spread of different course designs from different universities, our data seems to be limited to those who are either high achievers or perceive themselves to be high achievers in the course of research methods. Variability across instruction, including instructor personality and characteristics, methods, resources, and point systems could create confounding variables. Moreover, the cross-sectional nature of our study cannot be interpreted as causal relationships. While we think that attitudes influence perceived challenges and learning outcomes, it is plausible that the opposite direction occurs.

Future research should investigate the path that creates motivation for students in research methods. Knowing that the course adds value to their everyday life and feeling that they have the skill set to not only survive, but thrive, in a research methods course could make the difference in the amount that the student learns. Research should also further examine the topics considered to be bottlenecks to research methods—those concepts that disrupt learning in a significant number of students (Middendorf & Pace, 2004). More longitudinal and intervention research is needed to understand the causal relationships that predict learning outcomes in RM and the effectiveness of various teaching strategies and course designs. In the case of research methods and determined from the primary analysis of this study, validity may perhaps be a starting point. Knowing bottlenecks is just the beginning of improving the pedagogy of research methods.

In general, students who find the course to have purpose and have higher sense of self-efficacy tend to do better in the course. Identifying concepts that students find difficult as well as what motivates students to succeed should provide RM instructors with a base of information to enhance the focus of instruction and prevent students from getting tripped up in this course.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Amelang

(1976). Concurrent validity of ability and personality tests for several criteria of college-success. Psychologie in Erziehung Und Unterricht, 23(5), 259–272.

American Psychological Association. (2013). APA guidelines for the undergraduate psychology major: Version 2.0. http://www.apa.org/ed/precollege/undergrad/index.aspx

Cerny

B. A.

Kaiser

H. F.

(1977). A study of a measure of sampling adequacy for factor analytic correlation matrices. Multivariate Behavioral Research, 12(1), 43–47. https://doi.org/10.1207/s15327906mbr1201_3

Ciarocco

N. J.

Lewandowski

G. W.

Van Volkom

(2013). The impact of a multifaceted approach to teaching research methods on students’ attitudes. Teaching of Psychology, 40, 20–25. https://doi.org/10.1177/0098628312465859

Ciarocco

N. J.

Strohmetz

D. B.

Lewandowski

G. W.

Jr (2017). What’s the point? Faculty perceptions of research methods courses. Scholarship of Teaching and Learning in Psychology, 3(2), 116–131. https://doi.org/10.1037/stl0000085

Gurung

R. A. R.

Landrum

R. E.

(2013). Bottleneck concepts in psychology: Exploratory first steps. Psychology Learning & Teaching, 12(3), 236–245. https://dx-doi-org.web.bisu.edu.cn/10.2304/plat.2013.12.3.236

Gurung

R. A. R.

Stoa

. (2020). A national survey of teaching and learning research methods: Important concepts faculty and student perspectives. Teaching of Psychology, 47(2), 111–120.

Harlow

L. L.

Burkholder

G. J.

Jennifer

(2009). Evaluating attitudes, skill, and performance in a learning-enhanced quantitative methods course: A structural modeling approach. Structural Equation Modeling, 9, 37–41.

Hartley

A. A.

Hartley

J. T.

(1976). Predicting performance in the basic research methods course in psychology. Educational and Psychological Measurement, 36(2), 449–452. https://dx-doi-org.web.bisu.edu.cn/10.1177/001316447603600226

10.

King

(1995). Inquiring minds really do want to know: Using questioning to teach critical thinking. Teaching of Psychology, 22(1), 13–17. https://dx-doi-org.web.bisu.edu.cn/10.1207/s15328023top2201_5

11.

McKelvie

Standing

L. G.

(2018). Teaching psychology research methodology across the curriculum to promote undergraduate publication: An eight-course structure and two helpful practices. Frontiers in Psychology, 9, 1–4. https://doi.org/10.3389/fpsyg.2018.02295

12.

Middendorf

Pace

(2004). Decoding the disciplines: A model for helping students learn disciplinary ways of thinking. New Directions for Teaching and Learning, 2004(98), 1–12. https://doi.org/10.1002/tl.142

13.

Pintrich

P. R.

Smith

D. A. F.

Garcia

McKeachie

W. J.

(1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). https://files.eric.ed.gov/fulltext/ED338122.pdf

14.

Roberts

L. D.

(2016). Editorial: Research methods pedagogy: Engaging psychology students in research methods and statistics. Frontiers in Psychology, 7(1430), 1–3. https://doi.org/10.3389/fpsyg.2016.01430

15.

Sawyer

(2013). Beyond correlations: Usefulness of high school GPA and test scores in making college admissions decisions. Applied Measurement in Education, 26(2), 89–112. https://dx-doi-org.web.bisu.edu.cn/10.1080/08957347.2013.765433

16.

Sizemore

O. J.

Lewandowski

G. W.

(2009). Learning might not equal liking: Research methods course changes knowledge but not attitudes. Teaching of Psychology, 36, 90–95. https://doi.org/10.1080/00986280902739727

17.

Strohmetz

D. B.

Ciarocco

N. J.

Lewandowski

G. W.

(2018, October). Getting on the same page: Aligning student and instructor perceptions facilitate learning in methods class. Presented at the 2018 Annual Conference on Teaching, Phoenix, AZ, United States.

18.

Thieman

T. J.

Clary

E. G.

Olson

A. M.

Dauner

R. C.

Ring

E. E

. (2009). Introducing students to psychological research: General psychology as a laboratory course. Teaching of Psychology, 36(3), 160–168.

19.

Vittengl

J. R.

Bosley

C. Y.

Brescia

S. A.

Eckardt

E. A.

Neidig

J. M.

Shelver

K. S.

Sapenoff

L. A.

(2004). Why are some undergraduates more (and others less) interested in psychological research? Teaching of Psychology, 31, 91–97. https://doi.org/10.1207/s15328023top3102_3