Words of Warning: A Randomized Study of the Impact of Assorted Warning Letters on Academic Probation Students

Abstract

Objective: Superiority experiments supply a critical lens to probe higher education interventions because they can dismantle the collective effect of a program and identify the relative contribution of the core components. However, such lines of inquiry are generally absent in higher education literature. We draw on experimental data to probe the dominant mechanisms of a simple but promising academic probation intervention. Methods: We used a 2 × 2 factorial experiment designed to unpack the effects of an academic probation intervention and identify the dominant mechanism. Within this framework, 500 academic probation students were assigned to one of four treatment conditions or a control condition. Treatment groups were assigned to receive notification of academic probation status by either certified or regular mail (factor A) that contained either a standard or embellished letter (factor B). Results: We found that students who received embellished warning letters significantly improved subsequent academic performance. The impact on performance was 0.35 grade points when notification sent by regular mail and amplified to 0.48 grade points when sent via certified mail. The routine notification communication message did not differ from the treatment as usual method regardless of mail delivery method. Conclusions: These findings suggest the availability of low-cost, scalable interventions are available and can be identified to positively impact at-risk students’ academic achievement.

Keywords

academic probation college student achievement randomized experiment experiment education intervention student outcomes

Introduction

Despite the universal implementation of academic probation policies within higher education, relatively few empirical studies have been conducted on the impact of such programs on student success. Rarer are evaluations that have used rigorous research designs to evaluate the prospect of alternative, intervention methods that can both provide a positive impact on student outcomes as well as be scaled across postsecondary education.

One notable exception is a recent study by Yeaton and Moss (2020). These researchers used a randomized experiment, and two quasi-experimental research designs to study a novel warning letter intervention. They found that a strongly worded notification letter, sent by certified mail, significantly increased probation students’ subsequent semester academic performance. The promise of this study’s findings is buttressed by the inexpensive financial and human capital implementation costs of the intervention, relative to the measured impact. Even with the favorable results, questions lingered about whether the positive impact was due to the letter content, the delivery method, or a specific combination of both conditions. The current experiment investigates the comparative and coordinated contribution of these components in an effort to identify, refine, and scale the potential value of the intervention.

A Case for Investigating a Refined Treatment

The vast majority of rigorous research on academic probation within higher education focuses on the effect of existing institutional policies on academic performance (Albert & Wozny, 2019; Fletcher & Tokmouline, 2017; Lindo et al., 2010; Moss & Yeaton, 2015) or the impact of broadly implemented, complex, high-priced interventions (Angrist et al., 2009; Weiss et al., 2011) on academic performance, and drop-out risk. Take, for instance, the details of the interventions used in the two experimental studies (Angrist et al., 2009; Weiss et al., 2011).

Both experimental studies assigned academic probation students to treatment conditions that entailed multi-faceted interventions. In Weiss et al. (2011), the intervention comprised of three components. First, students were enrolled in a three credit College Success course taught by a college counselor. Two-thirds of the course involved lectures on topics such as study skills and setting personal goals. The other one-third of the course was a workshop that allowed students to practice the lecture content. The second component expected students to visit a Success Center five times, where they would receive additional support in mathematics and English. The third aspect of the intervention included extra academic advising during and outside of the course.

The interventions used in the Angrist et al. (2009) study also included various components. One condition, a financial incentive program, required students enroll full-time and depended upon the combination of students’ fall and winter term GPAs to award merit scholarships that ranged between $1,000 and $5,000. A second condition, the service program, involved numerous activities such as subject-specific peer-advising, email communication, and study groups. For the third condition, students were offered both the incentive and service programs but were not mandated to use either.

Both Weiss et al. (2011) and Angrist et al. (2009) reported a number of positive impacts on student outcomes related to the interventions but neither attempted to understand precisely what aspects of the treatment drove differences in outcomes. Shadish et al. (2002) refer to research that systematically analyzes treatment components treatment as a “dismantling study.” Studies that employ this approach strive to ascertain which intervention features produced the greatest impact or locate components that may be combined and attributed to a change. The process often incorporates isolating unique components or the combined contribution of several components to deconstruct and identify the most essential aspects of the intervention that drive measured impact. Though studies that intentionally dismantle the collective impact of a program into component-specific contributions are common in psychology (e.g., identification of effective treatment components), the approach has been overlooked in the education literature on academic probation (Britton et al., 2018).

Notification Methods and Messages Matter

Conventional practice for colleges and universities is to formally notify underperforming students of placement on academic probation with a letter. Notification letters are typically sent by U.S. mail, with some institutions opting for even less expensive email notifications. Though research is limited, these “standard” delivery methods have been found to be ineffective mechanisms to impact students’ subsequent achievement (Moss & Yeaton, 2015). Experimental research has investigated whether alternative contact methods, such as sending notification by certified mail, can influence recipients’ actions. Studies have found that notification letters sent via certified mail produced higher survey participation rates and also some emerging evidence found that certified letters positively impacted probation students’ academic performance (del Valle et al., 1997; Yeaton & Moss, 2020).

The nature of the communication sent is as important as the delivery method used to warn underperforming students. Institutions must choose where to place emphasis in the message communicated to students. Such message framing tactics are often used as methods to impact decision making and alter behavior (Tversky & Kahneman, 1981). Those messages that convey costs (loss-framed) associated with continued low academic performance (e.g., academic dismissal or further sanctions) may be construed differently than messages that emphasize the benefits (gain-framed) associated with academic performance improvement (e.g., graduation or career development).

The effectiveness of message framing has been found to vary across contexts. In a meta-analysis of 189 effect sizes across 94 health studies, Gallagher and Updegraff (2012) found that gain-framed messages were more often to be positively associated with prevention behaviors than were loss-frame messages. However, within higher education, loss-framed messages had a greater effect on career-related student behaviors than gain-framed messages (Tansley et al., 2007), but message framing had no influence on developmental education students’ placement exam performance or participation (Moss et al., 2019). With these mixed results, a cautious approach would entail a notification message of probationary status that includes both loss- and gain-framed content.

Research Questions

The current study was designed to clarify the impact and contribution of several aspects of a novel intervention for students on academic probation. We explored whether the method of notification about being placed on probation, as well as the message content, influenced probation students’ academic performance. We attempted to untangle the particular contributions of the intervention components by exploring the following research questions:

What was the specific contribution of the delivery mode of the letter? Holding the content of the letter constant, did the nature of letter delivery method (i.e., regular or certified mail) produce differential impacts on subsequent student achievement gains when compared to students who received the email notification letter?

What was the specific contribution of the letter content? Holding the delivery mode constant, did the embellished letter content, sent by regular or certified mail, improve subsequent term student achievement relative to those probation students who received the typical email notification letter?

Methods

Data for this study were drawn from academic probation students enrolled at a large, suburban, community college located in a Midwestern area of the United States. The study’s time frame encompassed two subsequent academic terms, fall (September through December) and winter (January through May). In the fall term, slightly more than half of all students at the college were female (57.8%), about one-fifth were first-time college students (19.1%), the average age was 26.9 (s = 10.3), and approximately 71% were enrolled part-time. More than half of students were classified as White (55.7%), followed by African American (27.6%), with the remaining student body identified as another race. Study protocols outlined below were reviewed and approved by the college’s senior academic administrators.

The academic sanction program at the college consisted of a commonly applied progressive framework. The first step involved students being warned, via an emailed letter, of unsatisfactory academic performance. In the subsequently enrolled semester, students were given the opportunity to improve their academic performance, however, if performance did not improve, students were moved into the second step of sanction. This second step required students meet with an academic guidance counselor before being allowed to register for courses. If unsatisfactory academic performance persisted, the third step of sanction was academic dismissal, with an opportunity to appeal for re-matriculation after 1-year. Our study focused on the first step of the sanction program, the warning phase.

Qualities of the Study Sample and Selection Process

As previously noted, students were placed on academic probation if they unsuccessfully met the college’s predetermined threshold that demonstrates satisfactory academic performance. At this college, a term GPA of 2.0 or higher was used to maintain good academic standing and meant avoidance of academic probation. At the end of the fall term, 4,092 students failed to meet this threshold and transitioned into the first step of the academic sanction program, that is, academic probation (see Figure 1). From this group, students were eligible for this study if they possessed characteristics that mirrored the sample from the foundational study (i.e., Yeaton & Moss, 2020).

Figure 1.

Flow diagram of sample selection and random assignment to treatment condition.

To replicate our sample characteristics, students were excluded if they were enrolled in less than four credits in fall term or had a fall GPA less than 1.0 or greater than 2.0. Ineligibility also occurred if a student was not preregistered for at least four credits in the winter term. Ultimately, 1,246 students were qualified for potential inclusion in the investigation. From this group, 500 students were randomly selected to be part of the study.

Description of the Experimental Conditions

To address our research questions, we conducted a randomized experiment and drew upon a 2 × 2 factorial design augmented with a control condition (Byar et al., 1993; Shadish et al., 2002). The first experimental factor probed variation in the mode of letter delivery (A), whereas the second experimental factor (B) probed variation in the letter content. The control condition employed the standard approach that sent an email using the conventional language.

Mode of delivery for the intervention groups consisted of two possibilities. Students either received notice by regular U.S. mail (A1) or by U.S. Postal Service certified letter (A2). All envelopes and letters were sent on official college letterhead. The certified letters were hand delivered by a U.S. mail carrier and required a household member to sign a prepaid return signature card. This card was returned to one of the researchers to verify receipt.

Two different types of notification letter were sent. The first letter condition contained content identical to the notification typically sent to probation students, hereafter referred to as the treatment as usual condition (TAU) letter (B1). The letter content was brief and contained basic information about the student being placed on probationary status, the need to earn a higher GPA the following term to avoid progressing through the sanction program, and a suggestion to take on-campus workshops or meet with academic counselors.

The second letter variant also provided students with a notification about being placed on academic probation, but it also contained extra information and was designed to have greater impact on student’s subsequent academic performance. We refer to letters with this content as embellished letters (B2).

Within the embellished letter, a figure was added that clearly displayed possible progression through the academic sanction program, eventually ending in dismissal from the college. The letter also clearly highlighted available resources students might find useful to avoid progression through the program. The overall tone of the letter was direct, and it provided concrete examples of how poor academic performance may impact various areas of students’ lives and encouraged self-reflection about possible solutions. The text also suggested that students post the letter in a visible location as a reminder during the term they were on probation.

The control condition consisted of students who received notification by the typical process implemented at the college. Under this condition, students received notice by email with letter content identical to the TAU letter. All letters were sent the next business day after final grades for the fall term were posted.

The particular combination of factors and control group meant students could be randomly assigned to one of five factorially combined experimental conditions. Using conventional research notation, the design can be described as:

\begin{array}{l} R X_{A1 + B1} O_{2} \\ R X_{A1 + B2} O_{2} \\ R X_{A2 + B1} O_{2} \\ R X_{A2 + B2} O_{2} \\ R O_{2} \end{array}

Here, each student was randomly assigned to a condition (signified by R), represented by each row. In turn, students were exposed to a treatment condition (X) with the subscript of X specifying the specific combination of components (i.e., A1 + B1, A1 + B2, A2 + B1, or A2 + B2), and the impact of those treatments was assessed on the change in fall to winter (O₂) semester GPA. The bottom row denotes students assigned to the conventional notification method, the TAU condition.

Design and Assignment Details

Using the effect size from Yeaton and Moss (2020) as guidance, we anticipated our warning notifications would have a moderate effect on the order of d = 0.35 (Cohen’s d scale). Although this prior research considered a simple two-group comparison between warning and no warning notification conditions, in the current study’s design, we assumed that such effects would extend to comparisons between each of the five conditions used in this study. With an anticipated moderate effect size of d = 0.35, our sample of 500 provided a power level of 0.70 or higher for each of the pairwise contrasts. When factoring in potential attrition, at a rate of about 20%, the power was reduced to about 0.60 or higher for these contrasts with an alpha level of 0.05.

Each student from the random sample was assigned to one of the five experimental conditions using a simple random assignment allocation rule. Here, every student had an equal probability of assignment to any of the five conditions. After randomization, 96 students were assigned to TAU, 95 to regular mail and TAU letter (A1 + B1), 96 to regular mail and new letter (A1 + B2), 107 to certified mail and TAU letter (A2 + B1), and 106 to certified mail and new letter (A2 + B2).

Following random assignment, we assessed for balance across several demographic characteristics (see Table 1). We found no differences across experimental conditions for gender, race, academic intent, first-time student status, average age, average number of fall credits, or average fall GPA. With archival data from the college student information system, we also explored if any of the sample had previously been placed on academic probation and been sent notification. No sample students had been on academic probation during the 14 semesters leading up the study time period.

Table 1.

Tests for Balance Across Student Characteristics by Warning Notification Procedure.

Characteristic	Treatment condition					Test of imbalance
	TAU (n = 96)	Regular mail, TAU letter (n = 95)	Regular mail, new letter^a (n = 96)	Certified mail, TAU letter (n = 107)	Certified mail, new letter (n = 106)
	% (n)	% (n)	% (n)	% (n)	% (n)
Gender
Male	34.4 (63)	47.4 (45)	47.9 (46)	46.7 (50)	34.9 (37)	7.95 (χ²)
Female	65.6 (33)	52.6 (50)	52.1 (50)	53.3 (57)	65.1 (69)
Race
White	53.1 (51)	47.4 (45)	44.8 (43)	49.5 (53)	49.1 (52)	3.16 (χ²)
African American	37.5 (36)	35.8 (34)	40.6 (39)	37.4 (40)	37.7 (40)
Other	9.4 (9)	16.8 (16)	14.6 (14)	13.1 (14)	13.2 (14)
Academic Intent
Earn credential	32.3 (31)	32.6 (21)	27.1 (26)	31.8 (34)	34.0 (36)	1.25 (χ²)
Other (transfer, undecided, gain skills)	67.7 (65)	67.4 (64)	72.9 (70)	68.2 (73)	66.0 (70)
First time student
Yes	29.2 (28)	47.4 (45)	36.5 (35)	36.4 (39)	41.5 (44)	7.46 (χ²)
No	70.8 (68)	52.6 (50)	63.5 (61)	63.6 (68)	58.5 (62)
Mean fall GPA (SD)	1.56 (.29)	1.52 (.30)	1.52 (.28)	1.54 (.29)	1.55 (.30)	0.40 (F)
Mean fall credits (SD)	8.74 (2.4)	9.38 (2.6)	9.06 (2.6)	9.20 (2.9)	8.98 (2.5)	0.88 (F)
Mean age (SD)	24.9 (9.3)	23.1 (7.3)	24.4 (8.7)	23.5 (7.8)	23.0 (7.2)	1.01 (F)

Note. No statistically significant differences are observed in any of the tests of imbalance shown. TAU = treatment as usual.

New letter refers to the embellished letter.

After assignment to treatment condition, 61 students were lost to attrition (e.g., transferred to another institution, dropped out, or withdrew from classes prior to the end of the winter term). Thus, 88% of students assigned to a condition were retained and used in the analysis (79 in the TAU group, 83 in the regular mail and TAU letter group, 86 in the regular mail and new letter group, 95 in the certified mail and TAU letter group, and 96 in the certified mail and new letter group). We investigated the potential impact of differential attrition by testing for balance across the study covariates. For those students who remained, there were no differences between experimental conditions among any of the covariates tested for the initial sample. Consequently, attrition was unlikely to undermine the internal validity of the findings.

Outcomes of Interest and Analytic Approach

To assess the impact of the treatment conditions on student success, we assessed student GPA using two different scales. The primary outcome to gauge impact was winter semester GPA measured on a scale of 0.0 to 4.0. Semester GPA allowed us to determine the magnitude of impact for each experimental condition. We also considered a dichotomized indicator of whether students winter GPA was at or above 2.0 as our secondary outcome.

We used an ANCOVA model to assess if there were differences between winter GPAs for each group while adjusting for differences in fall GPA. Our analytic model was

Y_{i} = β_{0} + β_{1} Z_{1 i} + β_{2} Z_{2 i} + β_{3} Z_{3 i} + β_{4} Z_{4 i} + x β {+ e}_{i}

where

Y_i = winter semester GPA

β₀ = intercept coefficient capturing the conditional average GPA for students who received an email notification with the TAU content

β₁ = effect of using standard mail relative to TAU email

β₂ = effect of using standard mail with embellished content relative to email with TAU content

β₃ = effect of using certified mail with TAU content relative to email with TAU content

β₄ = effect of using certified mail with embellished content relative to email with TAU content

β = coefficients for control variables

Z_1i = indicator for group that received TAU content via standard mail

Z_2i = indicator for group that received embellished content via standard mail

Z_3i = indicator for group that received TAU content via certified mail

Z_4i = indicator for group that received embellished content via certified mail

x = vector of covariates (age, fall GPA, gender, academic intent, race, first time student status)

e_i = residual

To answer the research questions, we unpacked the individual and combined effects of delivery mode and letter content by drawing on a series of contrasts among conditions. In our analyses, we first considered the effects that owe to changes in the mode of letter delivery. Specifically, we used β₁ to estimate the contribution of adopting standard mail relative to the TAU email when holding letter constant to the TAU content. We used β₃ to track the contribution of adopting certified mail relative to email when holding constant letter content to the TAU content. We also contrasted the relative gains associated with adopting certified mailings over standard mailings using the difference between β₃ and β₁ (β₃ − β₁).

We next considered the effects that owe to changes in the content of the letter by examining the differences among conditions that modulate the content (TAU vs. embellished) while holding the delivery mode constant at standard or certified mailings. We contrasted β₂ and β₁ to index the effect of adopting embellished content while holding constant the delivery mode to standard mail. We estimated the effect of adopting embellished content while holding constant the delivery mode to certified mail using the difference between β₄ and β₃.

Last, we examined the combined effects generated by simultaneously switching the mode and delivery methods. We used β₂ to estimate the combined effects of switching from an email with TAU content to a standard mailing with embellished content. We used β₄ to estimate the combined effects of switching from an email with TAU content to a certified mailing with embellished content.

Our categorical analysis examined the extent to which exposure to one of the conditions improved the probability with which students’ GPAs would exceed the probationary status threshold. Using the same covariates detailed above, we drew on a logistic regression model

ln(\frac{P (Y_{i} = 1)}{1 - P (Y_i = 1)}) = β_{0} + β_{1} Z_{1 i} + β_{2} Z_{2 i} + β_{3} Z_{3 i} + β_{4} Z_{4 i} + x β

where

P(Y_i = 1) captures the probability that a student’s winter semester GPA will exceed the probationary threshold GPA (2.0)

β₀ = intercept coefficient capturing the conditional log-odds that a student’s winter GPA will exceed the probationary threshold GPA (2.0) when exposed to an email notification with the TAU content

β₁ = effect of using standard mail relative to email

β₂ = effect of using standard mail with embellished content relative to email with TAU content

β₃ = effect of using certified mail with old content relative to email with TAU content

β₄ = effect of using certified mail with embellished content relative to email with TAU content

β = coefficients for control variables

Z_1i = indicator for group that received standard mail with TAU content

Z_2i = indicator for group that received standard mail with embellished content

Z_3i = indicator for group that received certified mail with TAU content

Z_4i = indicator for group that received certified mail with embellished content

x = vector of covariates (age, fall GPA, gender, academic intent, race, and first time student status)

Once again, we draw on the same contrasts among conditions outlined above to address our research questions.

Results

The results of our analyses for the first set of research questions are outlined in Table 2. When we held constant the letter content to the TAU content, switching from email to either regular or certified mail increased subsequent term GPA but only to a very limited degree (see Table 2). Our results indicated that adopting regular mail increased the subsequent GPA by 0.10 (β₁; standard error 0.18), while adopting certified mail increased the subsequent GPA by 0.08 (β₃; standard error 0.17). In both instances, however, our analyses suggested that the new modes of delivery did not produce effects that were statistically discernible from the standard email.

Table 2.

Effect Estimates by Experimental Condition.

Condition	GPA		Passed
Condition	Estimate [95% CI]	p-Value	Estimate [95% CI]	p-Value
TAU letter + regular mail	0.10 [−0.26, 0.45]	.589	−0.17 [−0.81, 0.47]	.607
TAU letter + certified mail	0.08 [−0.26, 0.43]	.624	−0.08 [−0.69, 0.54]	.810
Embellished letter + regular mail	0.35 [0.01, 0.70]	.047	0.22 [−0.41, 0.85]	.498
Embellished letter + certified mail	0.48 [0.13, 0.82]	.006	0.27 [−0.34, 0.89]	.388

Note. Covariates include age, first time college student, race, academic intent, and gender. TAU = treatment as usual.

A different picture emerged when we considered letter content. Our results suggested that content embellishment increased subsequent term GPA to a larger degree than delivery mode (Table 2). Switching to embellished content while holding delivery constant to regular mail produced a 0.25 gain (β₂ − β₁; standard error 0.18) when compared with the TAU content. Changing to embellished content while holding delivery constant to certified mail produced a 0.40 gain (β₄ − β₃; standard error 0.16) when compared with the TAU content. In both instances, the results suggested that content changes alone produced moderate effects; when using certified mail there was a clear benefit from swapping out the TAU content in favor of the embellished content. However, there was some outstanding uncertainty as to whether adopting the embellished content was statistically discernible from the TAU content when using regular mail.

The combined effects generated by simultaneously switching the mode and delivery methods demonstrated positive and statistically significant effects. Adopting regular mail with embellished content improved students’ subsequent GPA by 0.35 (β₂; standard error = 0.18) relative to an email with TAU content. That is, for probation students who received an embellished letter by regular mail, next semester achievement significantly increased by 0.35 grade points. Certified mail with embellished content also improved students’ subsequent GPA by 0.48 grade points (β₄; standard error = 0.17) relative to an email with TAU content, as shown in Figure 2.

Figure 2.

Adjusted estimates of letter content and delivery method on subsequent term grade point average.

We also considered the impact of the two letter variations and two mode of delivery combinations on a binary measure of students having or not having a subsequent grade point average that would move them off of academic probation. None of the four intervention conditions significantly increased the probability of students earning a 2.0 or better in the subsequent term after adjusting for the model covariates and when compared to the control condition (Table 2).

However, as with the continuous measure of impact, the embellished letter groups exhibited the best outcomes. Students from the embellished letter treatment groups that received notification by either regular or certified mail had the highest percentage of probation students earn a winter grade point at or above 2.0 (57.0% and 57.3%, respectively). Those groups that received the TAU letter did not fare as well. The TAU letter group that received notification by regular mail only had 45.8% earn a GPA at or above 2.0, whereas the TAU letter plus certified mail group had 50.5% above this threshold. Therefore, despite the lack of statistical significance, the overall pattern of better performance among students who received the embellished letter versus the TAU letter content was similar to the results when GPA was considered as the outcome.

Discussion

To our knowledge, this is the first higher education randomized study to prospectively dismantle the components of an intervention used for academic probation students in order to isolate which elements promote academic success. We aimed to determine if the nature of information sent or the method of notification about probationary status influenced students’ academic performance and found that the notification message used to inform students of probationary status had an effect. The embellished letter had the greatest positive impact on students’ next semester grade point average when compared to the typical method of notification. The embellished letter improved grades by 0.35 points when sent via regular mail. However, when the embellished letter was by certified mail the impact was bolstered to 0.48 grade points. Regardless of type of delivery method used, the typical notification message did not have an effect.

The embellished letter groups were also the only groups that earned winter grade point averages above 2.0; the threshold needed to cancel probationary status. The control group (M = 1.73, s = 0.98) and the two treatment groups that received the TAU letter content all had similar average winter term grade points below 2.0 (TAU letter + regular mail, M = 1.73, s = 1.24 and TAU letter + certified mail, M = 1.79, s = 1.11). Both groups that received the embellished letter had average winter grade points above 2.0. Of all treatment groups, students who received the embellished letter by certified mail had the highest average winter grade point (M = 2.14, s = 1.29), followed by students who received the embellished letter by means of regular mail (M = 2.04, s = 1.13).

Despite these optimistic findings, we did not observe the same academic benefit when we considered the probability of students earning a next semester grade point average that would move them off of academic probation (i.e., a GPA ≥ 2.0). In our inferential model, none of the combinations of notification method and content significantly improved the probability of moving off probation when contrasted to the typical method. Yet, we did find that both of the embellished letter groups had about 57% of probation students, the highest level among all groups, earn at or above the cutoff grade point average.

These somewhat contradictory findings raise questions as to why there was a discrepancy between the continuous and categorical achievement measures. One possible explanation for this null finding is that studies with the extant sample size will typically yield much lower statistical power to detect the treatment effects for dichotomous outcomes relative to quantitative outcomes. Thus, the cruder achievement measure fell short on the precision produced by a continuous valued outcome. A second potential reason for the binary outcome, no effect finding is that even though the embellished letter showed an increase of one-third to one-half grade points, the magnitude of the improvement was not sufficient enough to push the overall grade point above 2.0. To explore this possibility, we analyzed the distribution of fall grades for the overall sample, as well as across experimental conditions.

In this supplementary analysis, we found that for all groups nearly half (45.8%) of the probationary students would need to improve their next semester grade point average by a half-grade point to move above the non-probation threshold. In fact, about one-quarter of students (23.3%) in the experimental conditions would have needed to raise their GPA by a three-quarters to a full grade point. Consequently, if any of the lowest fall GPA students had increased performance by the largest average effect (i.e., 0.48 for the embellished, plus certified letter group), they would have demonstrated significant improvement in grade point but would not have shed their probationary status.

Our findings provide strong evidence that the nature of communication sent to academically at-risk students matters. Academic benefit was largest among the two groups that received the embellished letter. This letter contained content that encouraged a more personal connection to the events that led to probationary status, provided details about potential adverse outcomes, as well as provided specific direction to resources available to mitigate poor performance. All of these factors have been found to be important for motivating and informing probation students (Rodriguez, 2019). In contrast, the typical letter contained mainly a warning of probationary status and listed a phone number to the academic counseling office.

Overall, our findings closely align with those from the previous study by Yeaton and Moss (2020) that informed this study. These researchers found that the embellished letter sent by certified mail increased subsequent grades by 0.36 points, whereas we found an increase of 0.48 grade points after adjusting for covariates. However, unlike our study, this foundational study did not attempt to identify the most important aspects of the intervention protocol.

In both studies, effects found were sizeable for such a modest intervention. Other research has also observed rather modest effects with sending certified letters instead of regular mail or altering message content (e.g., del Valle et al., 1997; Gallagher & Updegraff, 2012), so perhaps the current findings are not unusual. Nevertheless, upon closer examination of the 95% confidence intervals associated with the two significant effect estimates in our study, it is possible that studies which use our same methods and analytic procedures, as well as our particular combination of intervention methods, might observe a negligible impact on student achievement. Future research might try to replicate across other contexts, with different populations, and larger samples to explore consistency of our conclusions. Additional research might also consider a condition that our study did not include—sending the embellished letter by email. Inclusion of this group to the current study conditions would have framed our study as a full factorial design and provided an opportunity to explore all possible patterns among factors.

Previous literature on academic probation students suffers from a similar shortcoming. Earlier intervention efforts include complex and expensive programs. For example, Weiss et al. (2011) evaluated two iterations of the Opening Doors program, which consisted of cash stipends, academic counseling sessions, a college success course, and support service center visits. Angrist et al.’s (2009) study of the STAR program included students assigned to either a multifaceted support program, a conditional, “strings attached” financial incentive program, or a group that combined both elements. Regardless of their findings, neither study attempted to uncover precisely what program components were most essential and necessary to positively impact students.

Finding and implementing effective intervention components is essential to assist academically underperforming students succeed. Weak intervention strategies may result in no academic benefit (Moss & Yeaton, 2015; Yeaton & Moss, 2020), higher rates of attrition (Fletcher & Tokmouline, 2017; Sneyers & De Witte, 2018), or even reduced self-confidence associated with probationary status classification (Rodriguez, 2019). Complex and expensive interventions may result in unnecessary expenditures and create obstacles that discourage at-risk students’ educational pursuits. Therefore, future research should build upon the current study and explore possible within program mechanisms that promote student educational gains without significant institutional or student costs (Albert & Wozny, 2019).

We are unaware of any research that systematically analyzes the content of probation letters. In the same spirit that this study approached understanding the subcomponents of the intervention, there is an opportunity to experiment with different elements of letter content reminiscent of recent research on developmental education (Moss et al., 2019). Randomly assigning probationary students to different letter content may lead to even better outcomes and result in messages tailored to specific student populations (Tversky & Kahneman, 1981). For example, more experienced underperforming students may benefit from being encouraged to set goals more than first-year probation students (Bowman et al., 2020). Uncovering such nuances can help steer at-risk students to resources most likely to promote academic success, which in turn can guide institutional policies and practices.

This study found that when program elements can be compartmentalized, carefully planned experimental research can be used to dismantle the effects of the intervention components. The elimination of unnecessary components creates fiscal savings and conserves human capital. Equally important, academic probation students would not be exposed to intervention components that provide no benefit and hamper progress. The reasonable cost of sending a modified notification letter by regular or certified mail is a step in this direction. Ideally, institutions will continuously seek to identify efficiencies and remove barriers to at-risk students’ success.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Brian G. Moss

Author Biographies

Brian G. Moss conducts research focused on methods used to enhance causal inference. More specifically, his research centers on the use of experimental and quasi-experimental research designs to study at-risk education and health populations.

Ben Kelcey is a professor in the College of Education Criminal Justice, & Human Services at the University of Cincinnati. His research interests include the development of measurement and quantitative research methods to understand effective teaching and teachers.

References

Albert

Wozny

(2019, September 24). The effect of academic probation on academic outcomes: Evidence from the US Air Force Academy [Conference session]. 4th IZA Workshop on the Economics of Education Bonn, Germany. http://conference.iza.org/conference_files/Education_2019/wozny_n26108.pdf

Angrist

Lang

Oreopoulos

(2009). Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics, 1, 136–163. https://doi.org/10.1257/app.1.1.136

Bowman

N. A.

Jang

Kivlighan

D. M.

Schneider

(2020). The impact of a goal-setting intervention for engineering students on academic probation. Research in Higher Education, 61(1), 142–166. https://doi.org/10.1007/s11162-019-09555-x

Britton

W. B.

Davis

J. H.

Loucks

E. B.

Peterson

Cullen

B. H.

Reuter

. . . Lindahl

J. R.

(2018). Dismantling mindfulness-based cognitive therapy: Creation and validation of 8-week focused attention and open monitoring interventions within a 3-armed randomized controlled trial. Behaviour Research and Therapy, 101, 92–107. https://doi.org/10.1016/j.brat.2017.09.010

Byar

D. P.

Herzberg

A. M.

Tan

W. Y.

(1993). Incomplete factorial designs for randomized clinical trials. Statistics in Medicine, 12(17), 1629–1641. https://doi.org/10.1002/sim.4780121708

del Valle

M. L.

Morgenstern

Rogstad

T. L.

Albright

Vickrey

B. G

. (1997). A randomized trial of the impact of certified mail on response rate to a physician survey, and a cost-effectiveness analysis. Evaluation & the Health Professions, 20(4), 389–406. https://doi.org/10.1177/016327879702000402

Fletcher

Tokmouline

(2017). The effects of academic probation on academic success: Regression discontinuity evidence from four Texas universities (Discussion paper No. 11232). IZA Institute of Labor Economics. Retrieved February, 19, 2018.

Gallagher

K. M.

Updegraff

J. A.

(2012). Health message framing effects on attitudes, intentions, and behavior: A meta-analytic review. Annals of Behavioral Medicine, 43(1), 101–116. https://doi.org/10.1007/s12160-011-9308-7

Lindo

J. M.

Sanders

N. J.

Oreopoulos

(2010). Ability, gender, and performance standards: Evidence from academic probation. American Economic Journal: Applied Economics, 2, 95–117. https://doi.org/10.1257/app.2.2.95

10.

Moss

B. G.

Bahr

P. R.

Arsenault

Oster

(2019). Knowing is half the battle, or is it? A randomized experiment of the impact of supplemental notification letters on placement exam participation, preparation and performance. Research in Higher Education. 60(6), 737–759. https://doi.org/10.1007/s11162-018-9536-9

11.

Moss

B. G.

Yeaton

W. H.

(2015). Failed warnings: Evaluating the impact of academic probation warning letters on student achievement. Evaluation Review, 39(5), 501–524. https://doi.org/10.1177/0193841X15610192

12.

Rodriguez

(2019). “I think of probation and I think of prison.” The lived experiences of community college students on academic probation: A phenomenological study [Doctoral dissertation]. California State University.

13.

Shadish

W. R.

Cook

T. D.

Campbell

T. D.

(2002). Experimental and quasi-experimental design for generalized causal inference. Houghton Mifflin.

14.

Sneyers

De Witte

(2018). Interventions in higher education and their effect on student success: A meta-analysis. Educational Review, 70(2), 208–228. https://doi.org/10.1080/00131911.2017.1300874

15.

Tansley

D. P.

Jome

L. M.

Haase

R. F.

Martens

M. P.

(2007). The effects of message framing on college students’ career decision making. Journal of Career Assessment, 15(3), 301–316. https://doi.org/10.1177/1069072707301204

16.

Tversky

Kahneman

(1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458. https://doi.org/10.1126/science.7455683

17.

Weiss

Brock

Sommo

Rudd

Turner

(2011). Serving community college students on probation: Four-year findings from Chaffey College’s opening doors program. MDRC.

18.

Yeaton

W. H.

Moss

B. G.

(2020). A multiple-design, experimental strategy: Academic probation warning letter’s impact on student achievement. The Journal of Experimental Education, 88(1), 123–144. https://doi.org/10.1080/00220973.2018.1469110