Hi,I Want to Talk to You About Your Progress: A Large Course Intervention for At-Risk College Students

Abstract

Performance during the first year of college and in introductory courses has been widely identified as critical to college students’ retention and success. Accordingly, interventions to assist beginning college students in gateway courses have gained increased attention in higher education. This study tested such an intervention using learning analytics and early performance feedback in large introductory courses, with comparison between course sections not exposed to the intervention. Distinct from other research, the present study replicated the efficacy of the intervention across 2 years, in multiple course subjects, and with different instructors at a 4-year Hispanic-serving institution. Importantly, findings also showed that high-risk college students especially benefitted from the intervention in terms of both achievement and persistence. Based on these findings, recommendations are made to higher education administrators about the potential gains to students when they are provided with early, consistent, and individualized feedback in all courses, made feasible by current technology.

Keywords

college student achievement college student success college persistence at-risk college students intervention study

Increasing the number of college graduates necessitates moving students beyond the high-stakes first year of college, during which they must contend with large-enrollment introductory-level courses (McElwee, 2013; Twigg, 2005). Between the first and second year of college, institutional dropout rates are impacted heavily by failure rates in introductory courses (Twigg, 2005). Typical DFW (drop, failure, and withdrawal) rates for introductory courses at Research I universities are around 15%, and range from about 22% to 45% at comprehensive universities (Twigg, 2005). Underserved student populations, including low-income students and students of color, are at particular risk of underperforming in these gateway courses (Twigg, 2005; Flanders, 2015).

In response to this problem, a variety of interventions have been employed for improving student learning and performance in large introductory courses across disciplines. Recent research including meta-analyses suggest that interventions in gateway courses result in increased passing rates and student performance, particularly among underserved or at-risk students (Valentine et al., 2011; Wibrowski, Matthews, & Kitsantas, 2016). A current trend in student success interventions derives from learning analytics, which refers to the collection and analysis of student performance data to assess progress and make predictions (Jayaprakash, Moody, Lauría, Regan, & Baron, 2014; Mattingly, Rice, & Berge, 2012; Picciano, 2012). While learning analytics are used by some higher education institutions to inform policy and decision-making, data have also been used at the course level to help instructors identify students in large classes who are at risk of failing or withdrawing from the course (Bevitt, Baldwin, & Calvert, 2010; Tampke, 2013; Wagner, Sasser, & DiBiase, 2002). Faculty utilizing early alert systems, made possible by developing technologies, can quickly find out which students performed poorly on initial course assignments or exams and reach out to those students to modify their learning and performance as early as possible during the academic term (Jayaprakash et al., 2014; Tampke, 2013).

Another broad category of interventions pertains to active learning, which aims to actively engage students in the learning process, in contrast with passive learning in traditional lecture courses (see Prince, 2004, for a review). Some research shows that students fail lecture-based courses at a higher rate than comparable courses that involve some element of active learning (Freeman et al., 2014). Examples of active learning interventions include learning communities, peer-assisted learning, use of response systems (i.e., clickers), among others (Freeman et al., 2014).

Research Objectives

The purpose of this study is to examine the impact of a coordinated strategy for decreasing withdrawals and improving grades in introductory large-enrollment courses. The strategy involves learning analytics and some elements of active learning. Similar to other research employing learning analytics, this study used formative data to provide immediate feedback to instructors and students, with the aim of improving students’ performance before the end of the term. The present study also drew upon active learning research by seeking to reduce the detachment between learners and instructors in large classroom environments where a sense of anonymity and disengagement among students can develop. Research on higher education teaching and learning emphasizes the importance of enhancing instructor–student communication, using early warning systems to identify students needing additional support, and providing constructive feedback to students on how to improve performance (Habley & McClanahan, 2004; Kuh, Kinzie, Buckley, Bridges, & Hayek, 2006; Twigg, 2005). The intervention designed for the present study incorporates these research-based practices.

The present study tested an intervention where instructional technology was utilized to (a) identify students in large courses who performed poorly on assessments early in the term and (b) allow instructors to easily send a large number of individual messages to students based on this information, which communicated the instructor’s concern and offered advice. The technology employed in the study consisted of the web-based course management system subscribed to by the university, in which instructors’ online gradebook was integrated with a commercial software program.

Importantly, the present study offers a specific focus on undergraduate at-risk students. Risk status in higher education is typically determined by prior academic performance, insufficient preparation for college-level work, poor performance in the first term, or number of hours employed (Schreiner, Noel, Anderson, & Cantwell, 2011; Valentine et al., 2011). These students may be at risk for having to take remedial courses, failing or withdrawing from courses, or leaving college before graduating either voluntarily or due to academic probation (Valentine et al., 2011). Research on improving student success in higher education has highlighted the need for identifying such students upon enrollment, tracking their progress, and supporting the retention of at-risk students (Habley & McClanahan, 2004; Kuh et al., 2006; Twigg, 2005). This research sought to explore the extent to which at-risk students would particularly benefit from the study’s intervention.

Further, this research takes into consideration other factors besides risk status that may impact college student performance, such as study skills, self-efficacy, academic motivation and expectations, and demographic or background characteristics. A comprehensive review of these factors is beyond the scope of this article, but an abundance of research in higher education points to the necessity of accounting for these variables in predicting student achievement and retention (Kuh et al., 2006; Zepke & Leach, 2010). Although this research will include such variables in the models to be tested, it should be noted that they are secondary to the main objectives of the research.

Justification for the Research

The present study is unique among other studies of college student success and retention in several ways. First, while the majority of similar studies are cross sectional and discipline specific, this investigation spans 2 years and involves several courses with different instructors in various departments. Second, other studies typically measure course-taking outcomes in terms of either achievement (grades) or retention (course withdrawal), whereas this study assessed both types of outcomes for the intervention and control groups. Finally, the understudied context of the present study is a Hispanic-serving institution, defined in Title V of the Higher Education Act under the U.S. Department of Education as an institution of higher education consisting of at least 25% Hispanic students with at least half of those students qualifying as low income. Thus, this research may inform instructors and leaders in higher education about how to support and retain beginning college students from a growing U.S. demographic group through a replicable course-based cross-disciplinary intervention.

Methods

Research Context

As part of a larger institutional initiative aimed at increasing student success at this public 4-year university, the present work was conducted with support from the provost’s office over a period of 2 years. In Year 1 and Year 2 of the study, the research design and methods were consistent, but each year used a different sample of participants, as well as a different set of instructors and courses. The second year of the study was intended as a replication of the first year of the study to test the efficacy of the intervention in a subsequent academic year with another sample of undergraduate students. In both years of the study, instructors teaching large undergraduate introductory-level courses (with enrollment of at least 100 students per section) were recruited on a volunteer basis with a monetary incentive for participation. A requirement for participation was that instructors needed to teach multiple sections of a course, so that sections of the same course could be assigned to the intervention or control group. All courses were delivered face-to-face on campus. Instructors were informed that they would be testing a new intervention for increasing student achievement and retention in large courses.

Overview of the Intervention

The intervention, aimed at increasing student success in large introductory courses, is a combination of (a) early identification of students’ poor performance and (b) immediate feedback to individual students from the instructor. Instructors participating in the study were already using a web-based platform (Blackboard Learn™) available to university instructors to manage their gradebooks, but they were largely unfamiliar with software (Starfish Zoom-In™) within the system for delivering the intervention. Prior to the start of the semester in Year 1 and in Year 2 of the study, instructors received training on how to use the software to administer the intervention once classes began.

The first component of the intervention is early identification of students’ poor performance. The instructor designated an assessment (i.e., quiz, assignment, exam, etc.) administered early in the term as the indicator of students’ initial performance in the course. Scores on the assessment were recorded in the instructor’s online gradebook. The Starfish Zoom-In™ software provides a menu preloaded with the assessment scores where the instructor can assign cutoff values to categorize scores as poor, ok, or good. The instructor may then generate a list of students filtered by a particular score category.

The second component of the intervention is providing immediate feedback to students regarding their performance in the course. For students whose assessment scores were categorized as ok or good, instructors composed a message expressing encouragement to continue their good work and to seek help from the instructor or teaching assistants when needed. In comparison, students whose scores were categorized as poor received a message from the instructor expressing concern about their initial performance in the course. The message reminded students about the stakes involved in not passing the course and offered strategies from the instructor on how to improve their performance (see Appendix for a sample message). Instructors composed messages specific to their course, with adherence to the intervention’s aim to provide timely information to students on the results of their assessment and feedback corresponding to their level of performance.

Students received the message within their Blackboard™ course inbox. The Zoom-In™ software facilitated the process of sending a large quantity of messages for students at different score categories. Importantly, the messages do not reveal that other students received the same message, thereby protecting the confidentiality of students’ scores and allowing for the appearance of individual attention. Also, students received a message from their instructor with performance feedback immediately following each course assessment for the entire term.

Risk Status

As noted earlier, the current investigation also intends to explore student success among at-risk college students. We sought to determine whether at-risk students in particular would benefit from the intervention. In the context of the present study, the university’s institutional research office calculates a risk probability for an entering student’s likelihood of degree completion based on several factors, including full- versus part-time status, high school class rank percentile, number of hours employed, SAT math score, and the university’s math and science placement test score. Access to the risk status scores for undergraduates participating in the study was requested from the institutional research office and the institutional review board.

Research Design

Each year of the investigation used a quasi-experimental design in order to compare outcomes for students in the intervention group to those for students in the control group. The intervention group consisted of those class sections where instructors employed the intervention described earlier, while the control group consisted of class sections where instructors did not apply the intervention, but taught courses equivalently in all other respects. All instructors involved in the study taught two sections of a given course, so one of the instructor’s sections was assigned to the intervention group and the other section to the control group. (However, in Year 1, two instructors each taught three course sections, so two sections were assigned to the intervention group and one section to the control group). While instructors did not randomly assign courses to intervention or control groups, all class sections consisted of undergraduate students in required introductory-level courses. Also, all students were administered a survey to assess demographic, background, cognitive, and attitudinal factors that could impact their course performance and retention.

Student Survey

To assess factors found to predict college student achievement and persistence in previous research, as well as to use these factors as control variables in testing the impact of the intervention, a survey was administered to all students at the start of the course in Year 1 and Year 2 of the study. Responses to survey items were based on a 5-point Likert scale, with the exception of questions regarding demographic information. Participants’ mean scores on each subscale within the survey were calculated for use in subsequent regression analyses of the study data.

One subscale included in the survey assessed study skills and preparedness (adopted from Bajwa, Gujjar, Shaheen, & Ramzan, 2011). Students were asked to respond to a series of items on a scale from not at all (1) to almost always (5): I turn in assignments on time, I come to class prepared, having completed the reading, I sit where I can see/hear what is going on in lectures, and I take good notes during class. In Year 1, the four items factored together using confirmatory factor analysis and showed acceptable reliability (α = .70). Year 2 of the study replicated the confirmatory factor analysis with a Cronbach’s alpha of .63.

Another subscale was included in the survey to assess students’ self-efficacy (adopted from Bandura, 1989, as cited in Zimmerman, Bandura, & Martinez-Pons, 1992). Participants were asked to indicate how well they could do the following on a scale from not well at all (1) to very well (5): study when there are other interesting things to do, take class notes, plan your schoolwork, and arrange a place to study without distractions. In Year 1, these four items factored together using confirmatory factor analysis and showed acceptable reliability (α = .72). Year 2 of the study replicated the confirmatory factor analysis with a Cronbach’s alpha of .77.

The survey administered in Year 2 contained some items not included in the Year 1 survey. One set of items adapted from the Patterns of Adaptive Learning Scale (Midgley, Arunkumar, & Urdan, 1996) assessed self-handicapping, a coping strategy for anticipating failure which uses excuses for avoiding challenges (Nurmi, Aunola, Salmela-Aro, & Lindroos, 2003). Participants reviewed a list of excuses students might use for not doing well in class, including: look for reasons to keep from studying (not feeling well, having to help their parents, taking care of a brother or sister, etc.), purposely do not try hard in class, and put off doing class work until the last minute. Participants were asked how true it is that they tend to use each excuse, from not at all (1) to very true (5). The latter two of the three items factored together using confirmatory factor analysis and showed good reliability (α = .81).

Another set of items included in the Year 2 survey assessed grit, as defined by Duckworth and Quinn (2009) as trait-level perseverance for long-term goals (see items on the short-form scale, Grit-S). Also, to assess students’ sense of belonging to campus culture, the following items were adopted from Nuñez (2009) and Museus and Maramba (2011): My family values are left behind by going to college, I feel troubled because my home life and my school life are like two different worlds, and I can talk to my family about my struggles and concerns at school. Participants indicated their level of agreement with each statement on a scale from strongly disagree (1) to strongly agree (5).

The survey administered in Year 1 and Year 2 included other items regarding academic motivation and goals, study strategies, grade expectations, class attendance, course load and repeated courses, employment, family encouragement, and caregiving roles. Student background or demographic items included classification, major, race or ethnicity, parents’ education levels, region of high school attended, comfort with languages spoken, and wait time on the international border. These latter variables pertain to the dynamic bi-national culture of this university situated alongside the U.S.-Mexico border.

Procedure

Students completed the survey in class at the first meeting of the semester and were informed that their individual responses would not be shared with instructors or affect their course grades, and that data would be analyzed at the group level by the researchers, in accordance with the university’s institutional review board guidelines. Within the first 3 weeks of the semester, instructors recorded students’ grades on the first course assessment (i.e., quiz, exam, project) in their online gradebook and categorized students’ performance as good, ok, or poor. As described earlier, instructors sent messages providing performance feedback to students who were in class sections receiving the intervention. Students who were in class sections for the control group did not receive any such messages from instructors but completed the same assessments and other course requirements as their counterparts in the intervention group.

Variables

Separate data sets were maintained for Year 1 and Year 2 of the study. However, variables were coded and analyzed in the same way for each year of the project. The primary independent variable was a student’s inclusion in the intervention group (coded as 1) or in the control group (coded as 0). A second independent variable was risk status. As mentioned earlier, the institutional research office estimates each undergraduate student’s likelihood of degree completion in terms of an individual risk status score. The scores are ranked as low, medium, and high risk, which we coded as 1, 2, and 3, respectively. Variables assessed by the survey (e.g., study skills, self-efficacy, demographics, etc.) were included in analyses as factors that may be predictive of student performance and persistence but were included primarily to control for individual differences that could impact the effects of the intervention. Categorical items on the survey were coded as dummy variables for analyses.

Two dependent variables were used to measure student success in the course: academic performance, operationalized as the official course letter grade and persistence, operationalized as course withdrawal versus completion. Instructors assigned one of five traditional course grades (A, B, C, D, and F) without pluses and minuses. These grades were transformed into numerical values (0–4) with higher values indicating better performance. The persistence variable was coded as 1 if the student withdrew from the course and 0 if the course was completed.

Participants

Year 1

In Year 1 of the study, five volunteer instructors taught a total of eight class sections of introductory-level courses in history, political science, psychology, and chemistry. Four and a half class sections were exposed to the intervention, while the other three and a half class sections did not receive the intervention and thus served as the control group. (The half-sections were due to one of the instructors subdividing a class section). Undergraduates (N = 1,787) participated in the study through their enrollment in the intervention group (n = 867) or the control group (n = 920) class sections. Among students in the sample, 43% were enrolled in history courses, 25% in psychology courses, 17% in political science courses, and 15% in chemistry courses. Female students comprised 46% of the sample. Given that the university is a Hispanic-serving institution, a large majority of participants in the sample identified their race or ethnicity as Hispanic (78.3%). The remaining sample of students identified as White, non-Hispanic (6.7%); Black, non-Hispanic (2.6%); Asian American (0.8%); Native American (0.5%); two or more ethnic or racial groups (5.3%); or as international students (5.3%, predominantly Mexican nationals). A majority of participants were freshmen (58.5%), followed by sophomores (31.6%), juniors (8.0%), and seniors (1.9%). Nearly a third of the students in the sample reported that neither of their parents attended college (29.7%). In terms of the risk probabilities calculated by the institution, 31% of the students in our sample were categorized as low risk, 37% as medium risk, and 32% as high risk.

Year 2

In Year 2 of the study at the same institution, four volunteer instructors taught a total of eight class sections of introductory-level courses in history, biology, and business. Four of these class sections were exposed to the intervention, while the other four comprised the control group. Of the undergraduates (N = 795) that participated, 491 enrolled in class sections assigned to the intervention group and 304 enrolled in sections assigned to the control group. In this sample, 51% of students were enrolled in history courses, 31% in biology courses, and 18% in business courses. Female students comprised 56% of the sample. A large majority of students in the sample identified as Hispanic (82.8%), while the remaining students identified as White, non-Hispanic (7.4%); Black, non-Hispanic (2.6%); Asian American (0.8%); Native American (0.1%); two or more ethnic or racial groups (0.8%); or as international students (4.9%, predominantly Mexican nationals). A majority of participants were freshmen (58.2%), followed by sophomores (20.5%), juniors (13.5%), and seniors (7.7%). About a quarter of the students in the sample reported that neither of their parents attended college (25.6%). Within this sample of students, 37% of students were categorized as low risk, 44% as medium risk, and 19% as high risk.

Results

Overview of Data Analysis

Data analysis was conducted for each year of the study separately, as the second year was designed as a replication of the first year to retest the efficacy of the intervention. In addition, a pooled sample would have introduced other methodological variation across years of the study (including some differences in questionnaire items, broader recruitment of instructors in the second year, and only one instructor involved in both years of the study). Thus, to be consistent with the purpose and methodology of the study, data are presented separately for each year of the study.

For Year 1 and Year 2 of the study, a series of regression analyses were conducted to test whether students in the intervention group would show higher performance (as evidenced by final course grades) and greater persistence (in terms of the number of course withdrawals), relative to students in the control group. The risk status variable was included in the regression models to investigate the extent to which outcomes might differ for students in the intervention group classified as high risk. Predictor variables from the survey were also included in the models. Due to the distinct categorical nature of the dependent variables, we used an ordinal regression model (ordered logistic) for the equations using grades as the outcome variable, and a binary regression model (logistic) for equations using withdrawals as the outcome variable. Results from the regression models for academic performance (grades) and for persistence (withdrawals) are presented separately for each year of the study. Based on the findings from the regression models, we conducted additional analyses to examine the degree of impact that the intervention had on grades and on withdrawals. The analyses used the regression model estimations to compute the change in probability for each outcome when participants were in the intervention group relative to the control group. Findings from those analyses are presented in graphs showing the percent change in probability. All analyses were conducted using Stata 14.2.

Year 1

Given that participants in the history courses comprised about half of the sample in Year 1, supplemental analyses involving only the history participants were conducted. Notation will be made regarding the extent to which findings for the full sample correspond to those for the history sample

Academic performance

Table 1 presents the regression results for final course grades, including the variable coefficients and standard errors for the predictors of student grades in the full sample and in the history subsample. The independent variable of primary interest—intervention—was found to be a significant predictor of final grades for both the full sample (p = .001) and the history subsample (p = .009). In other words, students in the intervention group were more likely to have higher final grades than students in the control group. This result offers support for our expectation that the intervention would positively impact student performance in large introductory courses.

Table 1.

Ordered Logistic Regression Predicting Final Course Grade in All Courses and in the History Course in Year 1.

	All courses (N = 1,057)		History only (n = 426)
Predictor	B	SE B	B	SE B
Intervention group	0.42***	0.13	0.53**	0.20
Risk status	−0.81***	0.08	−0.69***	0.13
Self-efficacy	−0.21	0.11	−0.07	0.18
Study skills	0.45**	0.15	0.54*	0.25
Grade expectations	0.55***	0.14	0.40	0.22
Class attendance	0.17	0.09	0.30*	0.15
Number of times course taken	−0.35*	0.16	−0.64	0.37
Number of courses in semester	0.08	0.07	−0.17	0.12
If materials difficult, change study habits	−0.14*	0.07	−0.02	0.12
Work hard even if dislike class material	0.29***	0.09	0.13	0.15
Seek help from academic advisor	−0.24***	0.06	−0.38***	0.10
Family encouragement for college degree	0.23*	0.09	0.36*	0.15
Parent college attendance	0.06	0.04	−0.05	0.07
Hours or week employed	−0.22***	0.06	−0.11	0.09
Caregiver	0.13	0.17	0.32	0.30
Age	0.03	0.02	−0.02	0.03
Classification	0.37***	0.10	0.41*	0.18
Humanities major	0.72***	0.13	–	–
Hispanic	−0.10	0.24	0.12	0.41
Native American	−2.38***	0.74	−1.19	1.19
African American (non-Hispanic)	−1.13**	0.43	−1.63*	0.67
Asian American	0.05	0.67	1.83	1.08
Mixed ethnicity	−0.89**	0.34	−0.65	0.57
International-Mexican	−0.23	0.40	0.64	0.70
International-other	0.61	0.92	15.40	881
Spanish most comfortable	−0.07	0.21	0.04	0.35
Spanish and English most comfortable	−0.08	0.14	0.52*	0.25
Other language most comfortable	0.09	0.88	−1.66	1.74
Wait time at border	−0.13	0.07	−0.31**	0.11
State X high school	0.90*	0.42	1.60*	0.62
State Y high school	0.85	1.09	1.15	1.55
Other state high school	0.50	0.32	0.95	0.51
Local high school	0.44	0.33	−0.36	0.55
Mexico high school	0.40	0.78	−1.10	1.40
Other international high school	1.34	0.79	1.98	1.35
College of business	−0.40	0.26	−0.24	0.41
College of education	−0.42	0.36	0.74	0.54
College of engineering	−0.42	0.36	0.88*	0.44
College of health sciences	−0.20	0.28	−0.13	0.44
College of liberal arts	0.14	0.23	0.66*	0.34
College of nursing	−0.15	0.26	0.38	0.41
College of science	−0.32	0.24	0.66	0.38
τ¹	2.27	1.01	0.48	1.67
τ²	3.00	1.01	0.93	1.67
τ³	4.36	1.01	2.45	1.67
τ⁴	5.94	1.02	4.47	1.68
χ² (df)	398.52 (42)***		152.16 (41)**
log likelihood	−1,324.49		−460.44

Note. Sample size reflects missing listwise values from nonresponses to individual survey items.

*p < .05, **p < .01, ***p < .001.

We conducted additional analyses to determine the amount of impact that the intervention had on grades. Figure 1 plots the probability change using the full sample model estimations found in Table 1. Each bar represents the change in the probability of earning a certain final course grade for students that received the intervention in comparison to those that did not. For example, the likelihood of earning a C, D, or F dropped −4.95%, −1.78%, and −2.10%, respectively, for students in the intervention group in comparison to those in the control group. The probability of earning a B essentially did not change (−0.21%), while the likelihood of earning an A increased by 9.03%.

Figure 1.

Impact of the intervention on final course grade in Year 1, in terms of the change in probability of receiving a specific grade for the intervention group relative to the control group.

Findings for the history subsample were similar to those for the full sample: Interventions improved the likelihood of earning an A (13.23%) and lowered the likelihood of earning a C (−5.83%), D (−0.90%), or F (−2.07%). Somewhat different from the finding for all courses, the likelihood of earning a B dropped (−4.44%) in the history course.

To test the notion that high-risk students would especially benefit from the intervention, we examined the degree to which the intervention impacted students in each of the risk status categories. Figure 2 displays the change in probability of earning specific grades by risk category. The values were calculated using the full sample model found in Table 1. The pattern of findings is consistent with findings for the overall sample, but the values reveal differences among the risk groups. The high-risk group had the largest decrease in Ds (−2.73%) and Fs (−4.43%), as well as the greatest increase in Bs (4.74%). Although the percent change in earning an A is lowest for the high-risk group (5.62%), we need to consider that the baseline probability of earning an A was lower for this group than for the medium- and low-risk groups. These findings show that, relative to the other groups, high-risk students benefitted most from the intervention.

Figure 2.

Impact of the intervention on final course grade by risk status in Year 1, in terms of the change in probability of receiving a specific grade for the intervention group relative to the control group.

A broader way to view the effects of the intervention on student performance is the overall pass rates. An analysis of the probability change in final course grade was conducted using grades recoded as pass or fail: Grades A, B, or C were recoded as 1, and Grades D or F were recoded as 0. Results showed that students in the intervention group were 3.87% more likely to pass than those in the control group, or an estimated 61 students passed the courses due to the intervention. The impact was greatest among the high-risk students, who were 7.16% more likely to pass with the intervention, or an estimated 36 students. The other groups also improved their likelihood of passing by 4.04% for the medium-risk group (an estimated 23 medium-risk students) and 1.98% for the low-risk group (an estimated 10 low-risk students).

We were also interested in which variables apart from the intervention or control variable accounted for students’ performance in large classes. The findings in Table 1 for both the overall and history samples show, expectedly, that the risk status variable was negatively associated with grades, as students classified higher in risk had lower grades. For both samples, study skills, family encouragement, and classification were positively associated with final grades, such that higher levels of each predictor were related to higher grades. Also, in both samples: Students who graduated from an in-state high school located in another city received higher grades than students who graduated from a high school within the city; and students who self-identified as Black (non-Hispanic) received lower grades than students who identified as White (non-Hispanic). In addition, enrollment in the history course, as opposed to the other courses in the study, was associated with receiving higher final grades. Unexpectedly, students who were more likely to agree with the statement “I will seek academic help from an advisor this semester” received lower grades.

Certain variables assessed by the survey that significantly predicted final grades for the overall sample, but not for the history subsample, include the following. Final grades were positively associated with grade expectations and agreement with the statement: “I work hard to do well in a class even if I don’t like what we are doing.” The number of hours students intended to work for pay and the number of times the course had been repeated were negatively related to final grades. Students who identified themselves as Native American or as mixed race or ethnicity received lower grades than students who identified as White (non-Hispanic). Also, unexpectedly, lower grades related to agreement with the statement: “If course materials are difficult to understand, I change the way I study.”

Other variables significantly predicted final grades for the history subsample but not for the overall sample. Class attendance (in terms of greater agreement with “I attend class regularly even when attendance is not required”) was positively related to final grades. Students majoring in engineering and in liberal arts received higher grades than students who were undecided. Students who reported feeling comfortable speaking both English and Spanish received higher grades than students comfortable speaking only English. Students who reported frequently missing class or arriving late to class due to long wait times at the international border received lower grades.

Persistence

In addition to students’ grades, the other dependent variable examined was persistence, in terms of withdrawal from the course. Table 2 presents the regression results for withdrawals, including the coefficients and standard errors for the predictors of persistence in the overall sample. In line with our expectations, the intervention was negatively related to withdrawals (p = .049; one-tailed test), such that students in the intervention group were less likely to drop a course than students in the control group. Overall, approximately 6% of students dropped a course at some point during the semester. Of the students in the intervention group, approximately 3.9% dropped a course, compared with approximately 5.7% of students in the control group. (Given that the total number of withdrawals was relatively few, a separate analysis for the history subsample was not conducted).

Table 2.

Logistic Regression Predicting Course Withdrawal in Year 1 (N = 1,165).

Predictor	B	SE B
Intervention group	−0.53*	0.32
Risk status	0.46*	0.20
Grade expectations	−0.63**	0.27
Parent college attendance	−0.17*	0.10
Hours or week employed	0.33**	0.13
Caregiver	0.82**	0.34
constant	−1.24	1.40
χ² (df)	31.10 (6)***
log likelihood	−174.99

Note. Sample size reflects missing listwise values from nonresponses to individual survey items. Tests are one tailed.

*p < .05, **p < .01, ***p < .001.

Additional analyses were conducted to determine how much the intervention reduced course withdrawals. Figure 3 shows the probability change based on the model estimations in Table 2. Exposure to the intervention accounted for a 1.5% decrease in withdrawals. The reduction was greatest among the high-risk group (−2.3%), although withdrawals also decreased among the medium-risk (−1.5%) and low-risk (−1%) groups. In other terms, an estimated 23 students overall did not drop a course due to exposure to the intervention, including an estimated 12 students in the high-risk group, 9 students in the medium-risk group, and 5 students in the low-risk group. These findings for course withdrawals parallel the findings for students’ grades, which indicate that high-risk students in particular benefitted from the intervention.

Figure 3.

Impact of the intervention on course withdrawal by risk status in Year 1, in terms of the change in probability of dropping a course for the intervention group relative to the control group.

As shown in Table 2, there were other factors apart from the intervention that accounted for students’ persistence. As suggested earlier, the risk status variable was positively related to withdrawals, such that higher risk status was associated with increased likelihood of dropping a course. Reflecting the findings for students’ grades, course withdrawals were associated with grade expectations (higher expectations were related to lower likelihood of withdrawal), and employment (the intent to work more hours was related to higher withdrawal). Other variables that had not predicted grades were found to significantly predict persistence. Students whose parents have college experience were less likely to drop a course, and students who serve in the role of a primary caregiver were more likely to withdraw from a course.

Year 2

Academic performance

Table 3 presents the regression results for final course grades in Year 2 of the study. As in Year 1, the intervention was found to be a significant predictor of final grades (p < .001), such that students in the intervention group were more likely to have higher final grades than students in the control group. This finding contributed additional evidence that the intervention positively impacted student performance.

Table 3.

Ordered Logistic Regression Predicting Final Course Grade in Year 2 (N = 493).

Predictor	B	SE B
Intervention group	0.64***	0.18
Risk status	−0.46***	0.13
Self-efficacy	0.27	0.21
Study skills	−0.28	0.15
Grade expectations	0.76***	0.26
Class attendance	0.19	0.15
Number of times course taken	−0.45	0.28
Number of courses in semester	0.08*	0.05
If materials difficult, change study habits	−0.19*	0.10
Work hard even if dislike class material	0.19	0.12
Grit	0.02	0.10
Self-handicapping	−0.16*	0.08
Belonging to campus culture	−0.17	0.15
Family encouragement for college degree	−0.001	0.18
High school friends also enrolled	0.01	0.07
High school friends talk about college, career	0.09	0.15
Parent college attendance	0.09	0.06
Hours or week employed	−0.08	0.09
Classification	0.18	0.11
Humanities major	0.72***	0.13
White non-Hispanic	−0.96	0.97
Hispanic	−0.70	0.87
African American (non-Hispanic)	−1.33	1.04
Asian American	−0.19	1.42
Mixed ethnicity	−0.96	0.94
International-Mexican	−0.22	0.95
Wait time at border	0.06	0.09
Local high school	−1.09***	0.33
τ¹	−0.33	1.97
τ²	0.58	1.97
τ³	1.80	1.98
τ⁴	3.35	1.98
χ² (degrees of freedom)	128.64 (27)***
log likelihood	−670.15

Note. Sample size reflects missing listwise values from nonresponses to individual survey items.

*p < .05, ***p < .001.

In conducting similar analyses in Year 2 as in Year 1, we analyzed the degree of impact of the intervention on students’ grades. The likelihood of earning a C, D, or F dropped −6.56%, −4.32%, and −4.22%, respectively, for students in the intervention group relative to those in the control group. The likelihood of earning a B increased by 3.08%, and the likelihood of earning an A increased by 12.02% (see Figure 4).

Figure 4.

Impact of the intervention on final course grade in Year 2, in terms of the change in probability of receiving a specific grade for the intervention group relative to the control group.

Also, in Year 2 as in Year 1, we examined the degree to which the intervention impacted students’ risk categories (see Figure 5). The high-risk group had the largest decrease in Ds (−5.26%) and Fs (−6.73%), as well as the greatest increase in Bs (7.26%). Again, these findings show that high-risk students benefitted most from the intervention.

Figure 5.

Impact of the intervention on final course grade by risk status in Year 2, in terms of the change in probability of receiving a specific grade for the intervention group relative to the control group.

The effects of the intervention on overall pass rates were analyzed. Students in the intervention group were 7.51% more likely to pass than those in the control group, or an estimated 64 students passed due to the intervention. Replicating the Year 1 results, the impact was greatest among the high-risk students, who were 11.47% more likely to pass in the intervention group than in the control group; this equates to an estimated 19 students. The other groups also improved their likelihood of passing by 8.31% for the medium-risk group and 5.56% for the low-risk group, or an estimated 31 medium-risk and 18 low-risk students.

In Year 2, variables assessed by the survey were examined in terms of their relationship to students’ grades (see Table 3). As in Year 1, the risk status variable was negatively associated with grades, such that students classified higher in risk had lower grades. Also consistent with Year 1: Final grades were positively related to grade expectations; students who graduated from local high schools had lower grades relative to students from other high schools; and lower grades were associated with greater agreement with the statement: “If course materials are difficult to understand, I change the way I study.” Only for students in the Year 2 sample, higher grades were associated with majoring in the humanities and taking a higher number of courses during the semester. The items included on the Year 2 survey pertaining to self-handicapping showed that higher levels of self-handicapping were related to lower grades, consistent with definitions of the construct. Some variables related to grades in Year 1 did not yield significant associations in Year 2, including study skills, family encouragement, number of hours employed, number of times the course had been repeated, class attendance, wait time at the international border, classification, and ethnicity. Finally, subscales on grit and sense of belonging to campus culture, included with the Year 2 survey, were not significantly related to grades or persistence.

Persistence

In Year 2, we again analyzed persistence as an outcome, in terms of course withdrawals. Consistent with findings from the first year of the study, the intervention was negatively related to withdrawals (p = .024; one-tailed test), such that students in the intervention group were less likely to drop a course than students in the control group (see Table 4). Approximately 6.7% of students dropped a course at some point during the semester. Of the students in the intervention group, approximately 5.6% dropped a course, compared with approximately 8.5% of students in the control group.

Table 4.

Logistic Regression Predicting Course Withdrawal in Year 2 (N = 795).

Predictor	B	SE B
Intervention group	−0.60*	0.30
Risk status	0.17	0.20
Grade expectations	−0.47	0.31
Parent college attendance	−0.08	0.09
Hours or week employed	0.26*	0.14
Caregiver	0.27	0.18
constant	−2.55*	1.54
χ² (degrees of freedom)	14.28 (6)***
log likelihood	−176.86

Note. Sample size reflects missing listwise values from nonresponses to individual survey items. Tests are one tailed.

*p < .05, ***p < .001.

Overall, exposure to the intervention accounted for a 3.2% decrease in withdrawals, or an estimated 27 students did not drop a course due to exposure to the intervention (see Figure 6). The reduction was greatest among the high-risk group (3.9% or an estimated 12 students), although withdrawals also decreased among the medium-risk (3.3% or an estimated 9 students) and low-risk (2.9% or an estimated 6 students) groups. These results underscore the importance of the intervention for high-risk students.

Figure 6.

Impact of the intervention on course withdrawal by risk status in Year 2, in terms of the change in probability of dropping a course for the intervention group relative to the control group.

As in Year 1, we examined the relationship between persistence and factors other than the intervention (see Table 4). In Year 2, there was not a direct significant relationship between risk status and withdrawals, as there had been the previous year, although the withdrawals for students at each risk category in the intervention group followed the same pattern found in Year 1. Also, the only variable that was directly related to withdrawals was employment, where working more hours was related to higher course withdrawal.

Discussion

This study offers compelling support for the efficacy of an intervention aimed at improving college student success in large introductory courses. Drawing upon current research on learning analytics and active learning, the intervention was designed as a combination of early identification of students’ poor performance and immediate feedback to individual students from the instructor. Educational software facilitated the process of sorting students’ performance data and sending a large quantity of tailored messages in a way that approximated individual attention in high enrollment courses.

The study offers several important findings on how to improve postsecondary outcomes for students, especially for Hispanic students and students identified as at risk. First, the findings of this research showed that students who were exposed to the intervention were more likely to earn passing grades, higher grades, and less likely to withdraw from the course, relative to students not exposed to the intervention. When students enrolled in control group courses where the intervention was not employed to identify and respond to students’ early performance, students fared less well in terms of grades and withdrawals. These findings were consistent across 2 years of the study, involving different courses, instructors, and samples of participants. Given the variation within each year and across both years of the study, the effects of the intervention on students’ achievement and persistence are notable.

Second, the intervention was particularly effective for students classified by the institution as high-risk but was also effective for medium-risk and low-risk students. Students’ risk of not completing a degree was calculated by the institution based on a combination of factors involving prior performance (i.e., test scores and class rank) and current status (i.e., enrollment and employment hours). If the intervention was effective in boosting the performance and persistence of high-risk students, this suggests that instructor-based actions can have significant effects even for students with academic and situational challenges.

Lastly, the intervention appeared to be a more consistent influence on student achievement and persistence across the 2 years of the study than others factors assessed by the survey. While in Year 1, factors such as study skills, family encouragement, and classification were significantly related to students’ grades, these factors did not predict grades in Year 2, while other factors did, including self-handicapping and the number of courses taken in the semester. However, there were a few factors associated with grades across both years, including grade expectations, majoring in humanities, and location of the high school attended. Similarly, certain factors predicted course withdrawal in Year 1 (including grade expectations, parents’ education level, and being a caregiver) but did not predict withdrawal in Year 2. One factor, employment hours, was associated with course withdrawal in both years. Thus, given that the intervention predicted both grades and course withdrawals in each year of the study, the implementation of a strategy to boost student performance (vs. no such implementation) appears to have unique and consistent effects beyond a variety of factors.

Beyond the scope of this study is an investigation of the exact mechanisms driving the efficacy of the intervention. The intervention was developed with an understanding of the positive impacts of responding to students’ early performance data, based on established student achievement literature. Yet, precisely why the intervention was helpful to students in different courses with different instructors is not known from the findings of this study. It is conceivable that students in the intervention group were motivated to improve their performance following the instructor’s early and direct contact through the individual messages, which suggested strategies for studying and coursework. Students may have perceived that the distance between themselves and the instructor was reduced in a large course after receiving the messages. Students may have gained the confidence to persist in the course when they received instructor feedback that conveyed concern and encouragement. The present study calls for more research focused on discovering the underlying causal effects of the intervention.

The results of the study may suggest that this type of intervention in large gateway courses is especially important for students at the beginning of their college careers. Students who may be struggling with the transition to college—especially those from underrepresented groups or identified as high risk—and who perform poorly on initial assessments could make significant achievement gains if they are provided with early, consistent, and individualized feedback in all of their first-year courses. Higher education administrators could facilitate the coordination of instructor-based interventions by providing incentives to faculty who integrate interventions in their courses, and by educating college and department leaders about the benefits to students who receive such interventions.

Higher education administrators may be particularly interested in the long-term effects of interventions providing formative feedback to students. If students were exposed to similar interventions in all of their courses over consecutive semesters, would this impact cumulative grade point averages and graduation rates? The findings of this study imply that individual students and higher education institutions may mutually benefit from investing in strategies to scaffold students’ achievement in postsecondary learning environments.

Appendix

Example of an instructor’s message to a student following a course assessment.

Hi,

I want to talk to you about your first exam. Our records indicate that you did not pass. Please remember that in order to pass the exam (and the course) you need score a C or better. The syllabus outlines what points translate to which letter grades. In short, you need to score 70 or better to pass the exams.

I want you to succeed in the course. The course is set up for your success, but only you can succeed. No one will do this for you. However there are few ways we can help you achieve this. We can meet, one-on-one, to discuss any problems you may be facing.

Alternatively, you can go through the following checklist and see where you can improve:

Do all of the readings before the class meetings. Remember reading the text is more than just looking at the words. Ask yourself the following question before you begin: “What will I learn in this chapter?” Ask yourself similar questions before each chapter’s section: “What will I learn in this section?” The chapter title and introduction will help you with the first question. The section titles will help you with the second question. After you read each section, ask yourself this question: “What do I know now that I did not know before I read the section?” If you cannot answer this question, you will have to re-read the section! Write down the answers to these questions and bring them to class. Also, write down questions regarding items you just do not understand. Bring these questions to class. If you do not get the answers during class, let me know and I will help you!

Review your notes before we begin each class. This includes both your class and reading notes. Ask yourself, “What did I learn in the last class?” and “What do we still need to cover?” The PowerPoint slides posted on Black Board and the readings should help you to answer the second question.

Write down the quiz questions and answers. You will see them again on the exams! If you did not answer the question(s) correctly, find out why.

Spend ample time reviewing everything for the exams. If there are any questions that have not been addressed, simply ask. This can be done one-on-one or through email.

During class, your attention needs to be focused. If friends are distracting you, sit somewhere else.

I want to leave you with one final word about studying: Studying is a personal habit. I will not fool you in believing there is one and only one way to study. However one thing I do know is that you cannot study course materials AND do something else. This is just impossible. Therefore I suggest finding the correct environment where you can focus and try the items suggested above.

Please let me know if there is any other way I can help.

Best,

Footnotes

Acknowledgments

The authors thank Nazanin Heydarian for review and feedback on earlier versions of the manuscript, Amanda DeGraff and Rachel L. Boren for coordination of the studies’ procedures, and all the university faculty led by Charles Ambler who contributed to large course initiatives.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was funded in part by the University of Texas System Initiative on Transforming Undergraduate Education.

Author Biographies

Penelope Espinoza, PhD, is an associate professor of educational leadership at The University of Texas at El Paso. She conducts interdisciplinary research in social psychology and education that deals with issues of equity and achievement in higher education and in pre-college educational settings. She is especially interested in applied research aimed at improving educational practice and policy for underrepresented and disadvantaged students.

Gaspare M. Genna, PhD, is professor of political science at The University of Texas at El Paso. His primary research interest is in explaining regional and international economic and political cooperation. His work uses economic and psychological models of inter-group behavior. He is also a longtime experimenter of high impact teaching techniques in large class settings.

References

Bajwa

Gujjar

A. A.

Shaheen

Ramzan

(2011). A comparative study of the study habits of the students from formal and non-formal systems of education in Pakistan. International Journal of Business and Social Science, 2(14), 175–186.

Bevitt

Baldwin

Calvert

(2010). Intervening early: Attendance and performance monitoring as a trigger for first year support in the biosciences. Bioscience Education, 15(4), 1–14. Retrieved from https://eprint.ncl.ac.uk/pub_details2.aspx?pub_id=156088

Duckworth

A. L.

Quinn

P. D.

(2009). Development and validation of the Short Grit Scale (Grit-S). Journal of Personality Assessment, 9(2), 166–174. doi:10.1080/00223890802634290

Flanders

G. R.

(2015). The effect of gateway course completion on freshman college student retention. Journal of College Student Retention, 19(1), 2–24. doi:10.1177/1521025115611396

Freeman

Eddy

S. L.

McDonough

Smith

M. K.

Okoroafor

Jordt

Wenderoth

M. P.

(2014). Active learning increases student performance in science, engineering, and mathematics. PNAS, 111(23), 8410–8415. doi:10.1073/pnas.1319030111

Habley

W. R.

McClanahan

(2004). What works in student retention? Four-year public colleges. Retrieved from the ACT website: https://www.act.org/content/dam/act/unsecured/documents/Retention-FourYearPublic.pdf

Jayaprakash

S. M.

Moody

E. W.

Lauría

E. J. M.

Regan

J. R.

Baron

J. D.

(2014). Early alert of academically at-risk students: An open source analytics initiative. Journal of Learning Analytics, 1(1), 6–47. doi:10.18608/jla.2014.11.3

Kuh

G. D.

Kinzie

Buckley

J. A.

Bridges

B. K.

Hayek

J. C.

(2006). What matters to student success: A review of the literature. Washington, DC: National Postsecondary Education Cooperative. Retrieved from https://nces.ed.gov/npec/pdf/kuh_team_report.pdf

Mattingly

K. D.

Rice

M. C.

Berge

Z. L.

(2012). Learning analytics as a tool for closing the assessment loop in higher education. Knowledge Management & E-Learning: An International Journal, 4(3), 236–247. Retrieved from http://www.kmel-journal.org/ojs/index.php/online-publication/article/view/196/148

10.

McElwee

R. O.

(2013, April). Teaching and advising first-year students. Observer, 26. Retrieved from https://www.psychologicalscience.org/observer/teaching-and-advising-first-year-students

11.

Midgley

Arunkumar

Urdan

T. C.

(1996). “If I don’t do well tomorrow, there’s a reason”: Predicting adolescents’ use of academic self-handicapping strategies. Journal of Educational Psychology, 88(3), 423–434.

12.

Museus

S. D.

Maramba

D. C.

(2011). The impact of culture on Filipino American students’ sense of belonging. The Review of Higher Education, 34(2), 231–258. doi:10.1353/rhe.2010.0022

13.

Nuñez

A.-M.

(2009). Latino students’ transitions to college: A social and intercultural capital perspective. Harvard Educational Review, 79(1), 22–48.

14.

Nurmi

J.-E.

Aunola

Salmela-Aro

Lindroos

(2003). The role of success expectation and task-avoidance in academic performance and satisfaction: Three studies on antecedents, consequences, and correlates. Contemporary Educational Psychology, 28, 59–90.

15.

Picciano

A. G.

(2012). The evolution of big data and learning analytics in American higher education. Journal of Asynchronous Learning Networks, 16(3), 9–20. Retrieved from https://files.eric.ed.gov/fulltext/EJ982669.pdf

16.

Prince

(2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231. doi:10.1002/j.2168-9830.2004.tb00809.x

17.

Schreiner

L. A.

Noel

Anderson

E. C.

Cantwell

(2011). The impact of faculty and staff on high-risk college student persistence. Journal of College Student Development, 52(3), 321–338.

18.

Tampke

D. R.

(2013). Developing, implementing, and assessing an early alert system. Journal of College Student Retention, 14(4), 523–532. doi:10.2190/CS.14.4.e

19.

Twigg

C. A.

(2005). Increasing success for underserved students: Redesigning introductory courses. Retrieved from The National Center for Academic Transformation website: http://www.thencat.org/Monographs/IncSuccess.htm

20.

Valentine

J. C.

Hirschy

A. S.

Bremer

C. D.

Novillo

Castellano

Banister

(2011). Keeping at-risk students in school: A systematic review of college retention programs. Educational Evaluation and Policy Analysis, 33(2), 214–234. doi:10.3102/0162373711398126

21.

Wagner

E. P.

Sasser

DiBiase

W. J.

(2002). Predicting students at risk in general chemistry using pre-semester assessments and demographic information. Journal of Chemical Education Research, 79(6), 749–755. doi:10.1021/ed079p749

22.

Wibrowski

C. R.

Matthews

W. K.

Kitsantas

(2016). The role of a skills learning support program on first-generation college students’ self-regulation, motivation, and academic achievement: A longitudinal study. Journal of College Student Retention, 19(3), 317–332. doi:10.1177/1521025116629152

23.

Zepke

Leach

(2010). Improving student engagement: Ten proposals for action. Active Learning in Higher Education, 11(3), 167–177. doi:10.1177/1469787410379680

24.

Zimmerman

B. J.

Bandura

Martinez-Pons

(1992). Self-motivation for academic attainment: The role of self-efficacy beliefs and personal goal setting. American Educational Research Journal, 29(3), 663–676.