Abstract
This article reports findings from an impact study of a 2-year postsecondary academic program offered in state prisons. Outcomes examined for participants during their 1st year of participation include performance on a standardized test of critical thinking skills, credit acquisition, achievement motivation, educational aspirations, personal development, and institutional (prison) climate. A cluster randomized design was used in which prisons in six states were randomly assigned to implement the Correctional Education Association College of the Air (CEA/COA) program or control programming. Analyses show that students in the CEA/COA program had average critical thinking scores that were about three fourths of a point lower, as measured by the Collegiate Assessment of Academic Proficiency (effect size = −.14), and acquired on average approximately two fewer credits (effect size = −.43) than students from control sites after 1 year. There were no significant group differences associated with educational aspirations, achievement motivation, personal development, and institutional climate measures.
Keywords
Introduction
The population of inmates in the United States has more than tripled over the past 25 years, resulting in more than 1 in 100 adults behind bars in an American jail or prison on any given day (Coley & Barton, 2006; Pew Center on the States, 2008). Contributing to this high rate of incarceration is the fact that over two thirds of prisoners are expected to be rearrested and over half are expected to be reincarcerated within 3 years of release (Langan & Levin, 2002). Not surprisingly, recidivism rates are lower for released inmates who are employed after their release (Solomon, Visher, La Vigne, & Osborne, 2006; Winterfield, Coggeshall, Burke-Storer, Correa, & Tidd, 2009); however, formerly incarcerated people often lack the resources to secure employment because of deficits in educational attainment and achievement (Harlow, 2003; Travis, Solomon, & Waul, 2001). Correctional education programs are designed to provide inmates with these resources by developing the knowledge and skills needed for success after release, focusing on instruction that leads to high school, career and technical, and postsecondary education credentials.
Community colleges have long been active partners in correctional education programs, providing a variety of education programs and serving as the primary providers of postsecondary education programming (Erisman & Contardo, 2005). Prisons have tended to partner with community colleges because of their cost-effectiveness, location, status as accredited institutions, and willingness to provide services, and these partnerships have been promoted at the federal level (U.S. Department of Education, 2009). Partnerships with prisons are consistent with the historical missions of many community colleges to provide open access and to serve traditionally underserved populations and students from diverse backgrounds (Green, 2006; Shannon & Smith, 2006).
The relationship between participation in prison educational programs and participant outcomes such as reduced rates of recidivism, postrelease employment and education, and other public cost savings has been well documented (Batiuk, McKeever, & Wilcox, 2005; Bazos & Hausman, 2004; Coley & Barton, 2006; Erisman & Contardo, 2005; Fine et al., 2001; Gaes, 2008; MTC Institute, 2003). Postsecondary correctional education programs have been shown to be especially promising for achieving these outcomes. For example, Chappell’s (2004) review of research found that prisoners who had participated in postsecondary education while incarcerated had rates of recidivism that were 46% lower, on average, than for those who had not taken college classes. In addition to reduced recidivism, postsecondary program participation has been associated with other postrelease outcomes, such as higher rates of employment and increased earnings (Batiuk et al., 2005; Contardo & Tolbert, 2008; Duguid, Hawkey, & Knights, 1998; Lichtenberger & Onyewu, 2005; Steurer, Smith, & Tracy, 2001; Wilson, Gallagher, & MacKenzie, 2000; Winterfield, Coggeshall, Burke-Storer, Correa, & Tidd, 2009). Participation in postsecondary programs while incarcerated has also been linked to prerelease outcomes for inmates and the prison environment, including changes in inmate behavior and attitudes and improved conditions in correctional facilities, including reduced disciplinary infractions, improved relationships among inmates and correctional staff, development of positive peer role models, and enhanced inmate self-esteem (Fine et al., 2001; Taylor, 1992; Winterfield et al., 2009). Although most U.S. prisons offer a range of vocational and academic education programming, national surveys suggest that postsecondary programs are available at fewer than 40% of correctional facilities and that only about 6% of the eligible inmate population participates (Erisman & Contardo, 2005; Gorgol & Sponsler, 2011; Stephan, 2008).
The Correctional Education Association College of the Air (CEA/COA) program was developed in response to an emerging body of research about positive impacts associated with participation in postsecondary correctional education programs and demand for low-cost and accessible programming for incarcerated students. The program is designed to facilitate access, persistence, and degree completion for this high-needs population by providing a widely available, low-cost, and flexible model for postsecondary academic program delivery. This article presents methods and results from a study of the impact of 1st-year participation in the CEA/COA program relative to students in other types of college programming. Based on data collected during a 3-year period in 43 prisons in six states, CEA/COA participant outcomes were contrasted with outcomes for students who participated in other types of postsecondary education programming.
The CEA/COA Program
The CEA/COA program is provided through a partnership between the Correctional Education Association (CEA), a nonprofit association serving educators who provide services to students in correctional settings, and Milwaukee Area Technical College (MATC), a 2-year community-based technical college serving more than 56,000 students per year. During the 2010-2011 academic year, the CEA/COA program served students in over 50 institutions in 13 U.S. states. The program provides courses leading to an associate of arts degree for students in prison, and was designed to include a series of courses equivalent to the general education/liberal arts and science requirements for freshmen and sophomores enrolled in a typical 4-year bachelor’s degree program. All courses offer credit toward an Associate of Arts degree, with the exception of two developmental courses that were added to the curriculum in the fall of 2009 and that focused on algebra and skills to foster success in college. These courses were added by MATC after feedback from sites implementing the CEA/COA program suggested that many students who participated during the 2008-2009 academic year were not well prepared to succeed in credit-bearing courses. Completion of the full 66-credit sequence is expected to prepare students to enter a 4-year degree program with junior standing. The program is designed to:
build skills and knowledge in the methods of idea analysis and synthesis necessary to develop and apply critical thinking techniques in the study of communications, humanities, fine arts, and social sciences;
provide students with a well-rounded academic background including written and verbal communication skills; broad-based, problem-solving abilities; critical thinking; and knowledge of diverse cultures;
provide a foundation for future studies of law, education, business, social sciences, history, English, communications, or humanities; and
offer an intermediate goal prior to completing a 4-year degree and prepare students to enter or advance in the workplace (MATC, 2009).
The CEA/COA program uses a distance-learning approach that combines the viewing of prerecorded course lessons with readings and supplemental materials from texts, study guides, and CD-ROMs, assignments, and assessments. Supplementary materials include a video introduction by the course instructor with information about assignments, grading, and expectations, a handbook for students, and study guides. Written assignments and multiple-choice examinations are included in the course curricula, and all assignments and examinations are sent to the course instructor for grading and written feedback. A site coordinator at each prison acts as a liaison between students and the course instructor for delivery of course materials, assignments, and examinations via U.S. mail, e-mail, and fax. The site coordinator also serves as a vehicle through which students can contact the instructor with questions.
CEA/COA courses are based on MATC College of the Air program courses that were originally developed for delivery to nonincarcerated students via public television, videotape, and audiocassette. Courses were adapted to meet the requirements of correctional settings to ensure that the content, language, and supplemental resources adhered to typical correctional facility policies. The CEA/COA program was first offered in the fall of 2006 using a satellite broadcast network known as the Corrections Learning Network (CLN) to deliver prerecorded lessons. In 2007, the administration of CLN was taken over by the CEA and the satellite-based network was renamed the Transforming Lives Network (TLN). During the 2008-2009 academic year, prerecorded CEA/COA lessons were delivered via the TLN satellite broadcast network. Because TLN discontinued operation in summer 2009, CEA/COA programming was delivered to prisons via prerecorded courses on DVD beginning in the fall of 2009.
The CEA/COA program is designed to allow students who transfer to another prison to continue their participation if their destination institution offers the program. If released from prison during their participation in the CEA/COA program, students may complete courses in which they are enrolled with no additional cost. After release, students may continue to enroll in future semesters by taking online courses. Students are eligible to continue in the program at the reduced rate of US$325 per 3-credit course plus the cost of books and supplies. Completed course credits may also be transferred to other colleges and universities.
Research Questions and Design
The goal of the study was to assess the impact of the CEA/COA program relative to other types of college programs and to support program development by examining the following primary question: To what extent does the CEA/COA program improve participant academic achievement and progress toward postsecondary academic degrees? Secondary research questions examined program impact on participant achievement motivation, educational aspirations, personal development, and perceptions of institutional climate. Analyses were also conducted to assess the extent to which measures of program implementation and student engagement were related to participant outcomes.
Participant outcomes included pretest and posttest measures of academic achievement, college credit acquisition, achievement motivation, educational aspirations, personal development, and perceptions of institutional climate. A cluster randomized trial design was used in which prisons were randomly assigned to either implement CEA/COA programming or to implement other forms of postsecondary academic programming that would normally be offered (i.e., a “business-as-usual” condition).
Sample
The study sample included 43 prisons in six states, 23 of which were randomly assigned to implement the CEA/COA program. Data were collected from three cohorts of participants, recruited in the fall of the 2008-2009, 2009-2010, and 2010-2011 academic years.
Sample of Prisons
Between 2006 and 2008, states were approached to participate in the study with the goal of meeting the following criteria: (a) geographic variation, (b) a minimum of four prisons per state with a high concentration of offenders who were eligible to participate in postsecondary programming, (c) the institutional capacity to expand postsecondary programming, and (d) willingness among state educational administrators to implement treatment or control programming based on random assignment. Five states met these criteria and were recruited to participate in the study during the 2008-2009 academic year. Two additional states were recruited in early 2009 for the 2009-2010 academic year, one of which was excluded from the study after random assignment of prisons because no CEA/COA programming was implemented. 1
Once states were recruited, the research team worked with state educational administrators to identify prisons for participation in the study. The criteria for prison participation included the following: (a) a high concentration of inmates eligible to receive federal grant funds for postsecondary education, (b) the infrastructure to provide postsecondary academic instruction, (c) no prior delivery of the CEA/COA program, (d) willingness to comply with random assignment procedures, and (e) recruitment of a minimum number of students beginning postsecondary academic programming who received grant funding for their participation.
Thirty-eight prisons in five states were recruited for the 2008-2009 academic year. Five additional prisons were recruited for the 2009-2010 academic year, two of which were in a sixth state that was recruited to begin that year. The final sample included 43 prisons in six states, as follows: Iowa (6 prisons), Massachusetts (8 prisons), Nevada (8 prisons), Oklahoma (11 prisons), South Carolina (8 prisons), and Wisconsin (2 prisons). All prisons that agreed to participate indicated their commitment to implementing intervention or control programming based on random assignment and to the associated research requirements, with the recognition that the CEA/COA program would be available to all prisons at the conclusion of the study.
While some prisons discontinued their implementation of postsecondary programming or their participation in the study, all 43 prisons recruited for the study implemented programs and participated in the study for a minimum of 1 year, allowing for examination of the 1st-year impact on participants across all prisons in the sample.
Random Assignment
A prerandomization blocked design was used in which random assignment to treatment versus control status occurred at the prison level within states. Blocking prisons by state was used to control for state-level variation on the outcome measures, improve the precision of impact estimates, and increase the statistical power associated with impact analyses. A computerized random number generator was used to randomly assign prisons to the CEA/COA and control conditions within each state. Thirty-eight prisons in five states were randomly assigned to participate beginning in the 2008-2009 academic year. Two states had an odd number of prisons, and random assignment resulted in one additional prison assigned to implement the CEA/COA program in each of these states, resulting in a sample of 38 prisons, 20 of which were assigned to implement the CEA/COA program. Five additional prisons were randomly assigned to participate beginning in the 2009-2010 academic year, two of which were assigned to implement the CEA/COA program. Two prisons in a newly recruited state were randomly assigned using the procedures described above, such that one was assigned to the CEA/COA condition and one was assigned to the control condition. The remaining three prisons were in two states that participated in the study during the prior year. In one state, random assignment was used to assign one of two newly recruited prisons to the CEA/COA condition. In the other state, one new prison was recruited and random assignment was used to assign it to implement control programming. The final sample included 43 prisons, 23 of which were assigned to the CEA/COA condition.
Case Study Prisons
Three prisons in each state were randomly selected to be case study sites for on-site qualitative data collection (two CEA/COA prisons and one control prison). 2 Case study prisons were visited during the fall and spring of each year to collect additional information about program implementation and perceptions of outcomes. Data collection included focus groups with randomly selected students, interviews with site coordinators, observation of courses, and document analysis.
Sample of Students
Data were collected from three cohorts of students, recruited in the fall of the 2008-2009, 2009-2010, and 2010-2011 academic years. Designated site coordinators at each prison recruited students to participate. To maximize the potential for inmate completion of CEA/COA programming and to ensure comparable participants in CEA/COA and control prisons, criteria for study participation were designed to reflect criteria for participation in a federal grant program that funded the costs of tuition, books, and supplies for most postsecondary education students within the study’s sample of prisons. During the 2008-2009 academic year, the program was known as the Grants to States for Workplace and Community Transition Training for Incarcerated Youth Offenders (IYO) program. In addition to the requirement that student tuition costs be paid with external grant funding, criteria for participation in the study during the 2008-2009 academic year also required that students be between 18 and 25 years old, have a release date between 1 and 5 years, and have a high school diploma or equivalent. With the passage of the Higher Education Act in August 2008, the IYO program was renamed the Grants to States for Workplace and Community Transition Training for Incarcerated Individuals Program (IIP). New eligibility criteria for IIP participation were defined and went into effect in July 2009. In response to this policy change, criteria for student participation in the study were revised beginning in the fall of 2009. Specifically, the criteria for participation in the study during the 2009-2010 and 2010-2011 academic years required that students be between 18 and 35 years old, have a release date between 1 and 7 years, have a high school diploma or equivalent, and have no convictions for a criminal offense against a victim who is a minor, a sexually violent offense, or murder.
Analytic Sample
Student data (collected during their 1st year of CEA/COA or control programming) were aggregated across the three cohorts to maximize statistical power for impact analyses. The cross-cohort sample included 1,534 students (728 treatment and 806 control). An analytic sample for impact analyses was created to exclude students who did not meet study eligibility requirements and who left the study for reasons exogenous to the intervention. Specifically, students who were not in their 1st year of participation in CEA/COA or control programming (5%), failed to meet age eligibility requirements (6%), lacked a high school diploma or equivalent (2%), or left the study because they were transferred to another prison, were released, or were paroled prior to posttest data collection (19%) were excluded. The final analytic sample included 1,088 students (512 treatment and 576 control).
Study Participants Relative to All Inmates at Sampled Prisons
Individual inmate-level data were collected for all incarcerated students in the analytic sample, but were not available for all inmates in study prisons. For example, age was available for each student in the analytic sample, while information about the age of all inmates at sampled prisons was available based only on administrator surveys that solicit the proportion of students within several age groups. To give a sense of the ways in which characteristics of sampled students reflected characteristics of all inmates in sampled prisons, descriptive data about inmate age, gender, and race/ethnicity are compared below for (a) the analytic sample of 1,088 students, based on survey data provide by incarcerated students, and (b) all inmates from the sample of 43 prisons, based on survey data provided by prison administrators.
Age
The study required that students be between the ages of 18 and 35, and the average age of incarcerated students in the analytic sample was 25.9 years (SD = 4.2). Across study prisons, the average percentage of the inmate population that was 35 or younger was 33.4% (SD = 9.4). Therefore, the age group of incarcerated students in the analytic sample corresponded to roughly one third of the inmate population at sampled prisons.
Gender
The analytic sample of incarcerated students was 76.4% male, slightly lower than the proportion of sampled prisons that served only male inmates (86.0%). The actual proportion of male inmates at sampled prisons was not available.
Race/ethnicity
The distribution of incarcerated students in the analytic sample by race/ethnicity was White or Caucasian (45.8%), Black or African American (36.5%), American Indian or Alaskan Native (5.0%), Latino or Hispanic (10.8%), and Asian or Pacific Islander (2.0%). Across study prisons, the average percentage of the inmate population in each race/ethnicity group was similar: White or Caucasian (48.4%), Black or African American (34.8%), American Indian or Alaskan Native (3.8%), Latino or Hispanic (10.7%), and Asian or Pacific Islander (0.9%).
Characteristics of CEA/COA and Control Prisons
Analyses were also conducted to compare baseline characteristics of prisons, site coordinators, and students at CEA/COA and control prisons. First, characteristics of prisons at baseline were compared (at the beginning of the 2008-2009 academic year for the initial sample of 38 prisons and at the beginning of 2009-2010 academic year for the five prisons that joined the study in the 2nd year). One prison was missing these baseline data. Prisons were compared on 15 different characteristics, such as size; percent of inmates in minimum, medium, and maximum custody; percentage of prisons offering a variety of adult, postsecondary, or vocational education; and the number of full- and part-time educational and postsecondary academic staff. None of these comparisons yielded statistically significant differences. Characteristics of site coordinators in CEA/COA and control prisons were also compared, including race/ethnicity, gender, education level, education certification (adult or postsecondary), years of experience teaching adults, and years of experience teaching in correctional facilities. None of these comparisons yielded statistically significant differences. Lastly, students were compared on the preintervention characteristics of race/ethnicity, gender, age, acquisition of a high school diploma or equivalent in high school or alternative school, number of college credits earned prior to participating in the study, and parenthood status. These analyses controlled for the clustering of students within prisons. Analyses yielded one statistically significant difference indicating that, on average, students in control prisons had acquired more college credits than their counterparts at CEA/COA sites prior to the intervention (control group, M = 6.25; CEA/COA group, M = 3.37; p = .02).
Response Rates
Response rates for the two student data collection instruments, the student survey and the Collegiate Assessment of Academic Proficiency (CAAP), are presented in Table 1, based on the analytic sample. A student was considered to have responded to an instrument if he or she provided at least one valid response. All students responded to the pretest survey and nearly all completed the pretest CAAP. Response rates for the posttest instruments were approximately 70%, with slightly higher response rates among control students.
Student Survey and CAAP Response Rates.
Note. CAAP = Collegiate Assessment of Academic Proficiency.
Attrition
Attrition rates were calculated separately for each posttest outcome variable and are presented in Table 2. Outcome and other measures are described in the next section. Attrition for one measure (educational aspirations) exceeded the attrition standard of the What Works Clearinghouse (2011), indicating the need to establish baseline equivalency on the pretest measure of that outcome and include pretest covariates to control for any baseline differences.
Attrition Rates for Posttest Outcomes.
Note. CAAP = Collegiate Assessment of Academic Proficiency.
Group Equivalence Analyses
Pretest equivalence was tested for each outcome, comparing treatment and control students in the benchmark impact analysis sample, which included students for whom pre- and posttest data were imputed (see the “Treatment of Missing Data” section), and accounting for the variation between students within prisons. Results of the pretest comparisons yielded a statistically significant difference for credit acquisition only. Prior to participation in the study, CEA/COA group students had acquired 2.89 fewer credits, on average, than students in the control group (p = .02). Effect sizes for the pretest differences were also calculated to represent the magnitude of the difference. Effect sizes for the pretest differences ranged from .00 for personal development to −.19 for credits, and none exceeded the What Works Clearinghouse (2011) threshold of .25.
Data Collection and Measures
Data were collected from pretest and posttest administrations of the CAAP Critical Thinking Test, student surveys, and administrative records maintained by site coordinators. Data collection took place during fall and spring on-site visits to each prison in the sample by the research team. Composite variables were created by averaging responses to individual survey items when at least half of the items had valid responses. Each measure is described as follows.
Outcome Measures
Confirmatory and exploratory outcomes were measured using pretest and posttest data from the CAAP, student survey, and administrative records. Several measures, described next, were employed.
Academic achievement
The Critical Thinking Test scale score from the CAAP was used to assess academic achievement. The CAAP is an assessment program that is designed to assess and evaluate postsecondary general education programs, focusing on skills typically acquired in the first two years of college. The Critical Thinking Test was selected as the CAAP module that would be most sensitive to the impact of diverse postsecondary academic correctional education programs. The test includes 32 multiple-choice items that measure students’ skills in clarifying, analyzing, evaluating, and extending arguments. Different forms of the test were administered at pretest and posttest. The test has a Kuder–Richardson Formula 20 reliability coefficient of .85 (ACT, 2007).
Credit acquisition
The number of undergraduate credits completed during the academic year was used as a measure of progress toward completing a postsecondary degree. This information was gathered from administrative records maintained by the site coordinator. For the small number of students for whom these data were missing (fewer than 5%), reports of completed credits from the student survey were used.
Achievement motivation
The study employed an adaptation of a measure developed by Plucker and Quaglia (1998) for high school students, which focuses on generalized motivation for achievement (i.e., setting high expectations for achievement and exhibiting attitudes and behaviors that support them, including problem solving, responsibility, and effort). The seven-item measure used a 4-category agreement scale (1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree) and had an alpha reliability of .94. Sample items included the following: “I have high expectations for myself” and “I put forth the necessary effort to reach a goal.”
Educational aspirations
The student survey assessed the likelihood that students would complete future postsecondary education. A continuous variable was developed using an approach adapted from Wolf et al. (2008), with the following values: complete current semester = 13, continue college classes in the future = 14, complete a 2-year associate’s degree = 15, complete a 4-year bachelor’s degree = 17, and obtain a master’s degree or other higher degree = 19. Years of expected future schooling were computed based on the highest level of education identified by the student as one that he or she “probably will” or “definitely will” complete.
Personal development
A series of items was developed to reflect short-term outcomes (e.g., changes in inmate behavior and attitudes, improved relationships with other inmates and correctional staff, and enhanced inmate self-esteem) that are relevant to success while in prison and on release and that have been associated with participation in postsecondary correctional education programs (Fine et al., 2001; Taylor, 1992; Winterfield et al., 2009). The 11-item measure was adapted from Owens and Volkwein (2002), used the 4-category agreement scale described earlier, and had an alpha reliability of .89. Sample items included the following: “I try to be a positive role model for other prisoners” and “I spend time thinking about how to best prepare myself for a career.”
Institutional climate
A measure of institutional climate was developed for the student survey to assess the quality of relationships among inmates and between inmates and institutional staff. The eight-item measure used the four-category agreement scale described earlier and had an alpha reliability of .89. Sample items included the following: “I work hard to cooperate with other inmates” and “I strive to improve my communication with institutional staff.”
Student Characteristic Measures
Measures of student characteristics, collected using the student survey and administrative records, included race/ethnicity, gender, whether the student earned a high school diploma in a traditional or alternative high school, parenthood status, and age at baseline.
Program Implementation Measures
Measures of generic implementation fidelity (i.e., relevant to the implementation of both CEA/COA and control programs) were collected using the posttest student survey. These measures are described next.
Course instructor support
Student perceptions of the extent to which their course instructor or instructors provided support that aligned with indicators of best practice were measured using the student survey. Items were adapted from Owens and Volkwein (2002), and additional items were developed to measure other factors identified as being related to program success in correctional settings (Rose & Voss, 2003; Shobe, 2003). The 19-item measure used the four-category agreement scale described earlier and had an alpha reliability of .97. Sample items included the following: “My course instructors are available to answer questions” and “My course instructors provide feedback, such as grades and comments, on my work in a timely manner.”
Education staff support
A parallel measure of education staff support was created based on the course instructor support measure described earlier. This 19-item measure also used the same four-category agreement scale and had an alpha reliability of .98. Items were identical to those previously described, but asked about support from “prison education staff” rather than “my course instructor.”
Peer support
The measure of peer support was adapted from Owens and Volkwein (2002). The six-item measure used the same four-category agreement scale and had an alpha reliability of .81. Sample items included the following: “I get support from other students when I need it” and “I share my experiences and/or knowledge with other students.”
Resources that support participation
Items from surveys of resources that incarcerated students find helpful to their education (Moeller, Day, & Rivers, 2004; Tewksbury & Stengel, 2006) and from a “learning environment” measure used with inmate populations (Owens & Volkwein, 2002) were adapted to create a survey measure of available resources that support postsecondary program participation. The 10-item measure used the four-category agreement scale and had an alpha reliability of .83. Sample items included the following: “Books and learning materials that students need are easily available” and “I have good opportunities to study outside of class.”
Student Engagement Measures
Several posttest survey measures were used to assess student program engagement. These are described as follows.
Completion of program components
Items on the student survey were used to compute an estimate of the average percentage of five types of activities that students completed as part of their college programs. Students were asked to estimate what percentage of the following activities they completed during the academic year: viewed or attended lessons, did assigned reading, studied course materials, completed assignments, and took exams or tests.
Long-term achievement motivation
The student survey included measures about student motivation for participating in college, focusing on the importance assigned by students to long-term and internally oriented motivations (e.g., to improve academic skills or to make changes to enhance the success rate on release). The eight-item measure was adapted from the Education Participation Scale (Boshier, 1991), used a four-category importance scale (1 = not important, 2 = somewhat not important, 3 = somewhat important, 4 = important), and had an alpha reliability of .80. Sample items included the following: “to prepare for a job after release” and “to contribute better to my family or community.”
Academic engagement
Items comprising the academic engagement student survey measure were adapted from measures developed by Blumenfeld, Secada, Modell, Bartko, and Fredricks (1999) and Newmann (1981). These measures were designed to capture behavioral, cognitive, and affective components of engagement. The nine-item measure used the four-category agreement scale described earlier and had an alpha reliability of .93. Sample items included the following: “I am interested in the work in my college program” and “Time seems to pass quickly when I am doing coursework.”
Site Preparation and Program Implementation
Prior to initial fall data collection visits, the research team visited each state to conduct on-site training with state-, region-, and site-level administrators, teachers, and support staff. Training included a discussion of the rationale and goals for the study, implementation expectations, and study protocols. The team arranged schedules for recruitment and data collection, secured security protocols for entry into the institutions, and sought introductions with wardens and associate wardens during site visits to reinforce local support. Written memoranda of understanding were signed by a representative of each state who had administrative oversight of postsecondary academic programming, outlining specific expectations related to the recruitment of participants, the designation of a primary and secondary site coordinator at each prison, the provision of the intervention as prescribed in CEA/COA prisons, the facilitation of study procedures (e.g., access to prisons for data collection and to inmate records), and the administration of incentives. Regular phone conferences were held with administrators and educators in each state to reinforce expectations and address questions.
Each participating prison was asked to provide CEA/COA programming or other postsecondary education programming, as designated through the random assignment process. Prisons assigned to the intervention condition were asked to provide the CEA/COA program as the primary institution-sponsored postsecondary academic curriculum 3 for the duration of the study, and prisons assigned to the control condition were asked to provide alternative postsecondary academic programming and not to provide the CEA/COA program for the duration of the study. Designated site coordinators were typically educational staff members with primary responsibility for coordinating postsecondary education at the site, and their responsibilities included informing eligible offenders about programs, advising and registering students, serving as a liaison between students and instructors, and carrying out other functions. The CEA/COA Site Coordinator Guide (MATC, 2009) was distributed to provide detailed information about expectations for implementation of the CEA/COA program, including administrative forms and guidance to facilitate implementation. In control prisons, no specific expectations related to the implementation of college programs were provided other than that programs be offered as they normally would and that CEA/COA programming not be offered. On-site visits were made twice each year by the research team to meet with designated site coordinators to monitor implementation (and nonimplementation in control sites) and to coordinate data collection.
To encourage compliance with study protocols, states were provided with US$2,000 for each prison participating in the study to spend toward the educational expenditure of their choice during the 2008-2009 academic year. During the subsequent 2 years, states were provided with US$1,000 for each prison in the study as an incentive for their continued participation.
Implementation Fidelity at Treatment and Control Prisons
Measures of implementation fidelity were included on the posttest student survey administered to students in CEA/COA and control prisons and included ratings of the quality of resources supporting participation (e.g., books, quiet places to study) and the support provided by instructors, education staff, and peers. Table 3 shows that average ratings among students in CEA/COA prisons reflect moderate agreement about the presence of these supports. Ratings were lowest for the presence of resources supporting participation, and there was substantial variation in ratings across prisons. Average ratings of resources supporting participation were similar across CEA/COA and control prisons; however, ratings were significantly higher for control prisons for instructor support, education staff support, and peer support on a 4-point scale.
Implementation Fidelity.
Note. Items were measured on a 4-point scale: 1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree. CEA/COA = Correctional Education Association College of the Air.
Qualitative data collected in case study prisons indicated that several implementation challenges were common to both CEA/COA and control prisons, including lack of student readiness for college-level work, lack of access to needed resources like space to study and textbooks, variation in site coordinator commitment to expectations, differential peer support, and challenges communicating with instructors (Meyer, Fredericks, Borden, & Richardson, 2010). Substantial additional challenges, specific to CEA/COA program implementation, were reported by field staff members who interacted with site coordinators and students in all prisons. These challenges included insufficient communication between students and course instructors, delays in acquisition of books and other materials, and insufficient feedback to students on course assignments and assessments.
Postsecondary Education Programs at Control Prisons
Information about control site programs was collected using open-ended items from the site coordinator and student surveys. In addition, program documentation was collected during on-site visits and from program websites. All control prisons offered either one or two primary programs to students who participated in the study. 4 Programs in the 20 control prisons were provided by a total of 16 colleges and universities, each of which worked with between one and four prisons in the study. Most providers were local 2-year public community or technical colleges. In a handful of prisons, courses were provided by a local 4-year public or private university. Most control site providers offered curricula consisting of general education, liberal arts courses that were part of an associate of arts degree program. Some institutions offered related associate of arts degrees in science, applied science, general studies, and business. Control site programs included courses that are typical of a core liberal arts curriculum leading to an associate of arts degree. Nearly all control prisons provided courses in English, literature, writing, and history. Courses in mathematics and social sciences (e.g., psychology, sociology, economics, and political science) were also prevalent. Relatively few programs provided courses in the natural and physical sciences.
Course delivery included both on-site direct instruction by local college faculty and/or distance-learning programs, which consisted of instruction through interactive television (ITV), prerecorded audio and video lessons, and correspondence courses. In control prisons where direct instruction was provided, course instructors came to the prison to teach courses, with the exception of one site where students were taken to a local college campus for courses with an instructor. Students in courses that were delivered via ITV viewed live lessons on a television and used interactive media to communicate with other students and the instructor in real time. Courses with prerecorded lessons used media such as DVDs, audiocassettes, or computer files to deliver instruction. Correspondence courses presented lessons along with other course materials in printed course guides. Institutional staff assisted with courses led by on-site instructors by coordinating classroom space, providing security, and facilitating contact between students and instructors. For ITV and correspondence courses, facility staff served as liaisons with course instructors by distributing course materials, returning completed assignments, and proctoring exams.
The cost of participation in postsecondary courses at control prisons was estimated on the basis of the published cost of a three-credit course for resident students during the 2009-2010 academic year. The cost per course ranged from US$189 to US$900, with an average cost of US$395. Two institutions waived tuition for student participation. As was the case in CEA/COA prisons, in nearly every case where tuition was charged, student expenses were paid with Incarcerated Youth Offender or Incarcerated Individuals Program federal grant funds. Students in some prisons received funds for participation from other sources including foundation grants, local churches, nonprofit organizations, and tribal funds.
Analytic Approach
This section discusses methods used for impact estimation and sensitivity analysis, treatment of missing data, and multiple hypothesis testing.
Estimating Impacts for Confirmatory and Exploratory Outcomes
The primary purpose of the confirmatory and exploratory impact analyses was to provide an unbiased estimate of the impact of the CEA/COA program on each of the outcomes. Confirmatory outcomes included CAAP Critical Thinking test scores and acquisition of college credit during the 1st year of college program participation. Exploratory outcomes included achievement motivation, educational aspirations, personal development, and students’ perceptions of institutional climate.
Consistent with the random assignment of prisons to either the intervention or control group, impacts were estimated at the prison level, using a two-level, mixed-model approach to account for students nested within prisons. Impact estimation was conducted separately for each outcome. Variance components at the student and prison levels were estimated to confirm the assumption of the nested structure of the data. Student-level covariates included the respective pretest measure for each outcome and student-level characteristic variables, including cohort membership, gender, earned high school diploma, parenthood status, and ethnicity. The prison level of the model included an indicator for assignment to the treatment group, a prison-level mean pretest, a prison-level mean of students’ pretest perceptions of resources that support participation, an indicator for all-female prison, an indicator for mixed-gender prison, and dummy variables to account for the blocking variable (state) used for random assignment. Effect sizes were also calculated for each outcome. In addition, sensitivity analyses were conducted to evaluate the robustness of the findings regarding the confirmatory outcomes—scores on the CAAP Critical Thinking test and credit acquisition—to methodological decisions.
Treatment of Missing Data
Overall posttest rates of attrition ranged from 5% to 36% across the six outcome measures. Although attrition exceeded standards set by the What Works Clearinghouse (2011) for only one outcome—educational aspirations—the amount of overall attrition was sufficient to adversely affect statistical power. For this reason, missing data were imputed.
Missing pretest and posttest data were imputed using the “mi impute chained” command in Stata 12. This command imputes missing data iteratively through a sequence of linear regression equations for continuous variables and logistic regression for dichotomous variables that include all the variables specified in the command as predictor variables except for the variable whose value is being imputed (van Buuren, Boshuizen, & Knook, 1999). For example, missing values for posttest credits were predicted by regressing posttest credits on all the other variables—in this case, the pretest and posttest variables for all outcomes. The values for the missing data were calculated using all available observations, including observations with missing data. The imputed values include random error to ensure that the imputation process does not incorrectly reduce the observed variation in the data.
Missing data imputations were conducted separately for intervention and control groups as per recommendations from recent research in this area (Puma, Olsen, Bell, & Price, 2009).
A total of 40 imputed data sets were created, and impacts were estimated using the “mi estimate” command in Stata that estimated parameters for each of the 40 imputed data sets separately and then combined them.
Addressing Multiple Hypothesis Testing
Impacts were estimated for the two confirmatory outcomes, one outcome in the domain of student achievement (CAAP Critical Thinking test scores) and one outcome in the domain of educational attainment (credit acquisition). The What Works Clearinghouse (2011) does not require correction for multiple hypotheses testing across domains. Following this protocol, no correction for multiple comparisons was conducted for the findings regarding the impact estimates for the confirmatory outcomes.
Analyses of Program Implementation and Student Engagement
In addition to the impact estimation, analyses were conducted to estimate the relationship between each outcome and indicators of implementation fidelity. These models also estimated the relationship between the outcome and measures of student engagement in the program. The model included all indicators of implementation fidelity and student characteristics to estimate the unique effect of each. The implementation variables included measures of resources that support participation, instructor support, education staff support, and peer support. Measures of student engagement in the program included students’ academic engagement, motivation for educational advancement in the long term, and level of completion of program components. These measures of fidelity and engagement are considered generic measures, such that they were relevant to and obtained from both the treatment students and the control students. Implementation fidelity measures were included at the student level of the model because they are based on students’ perceptions of implementation and because the nature of the program is such that each student experienced a unique instantiation of the program due to differences in classes taken, instructors, and peer groups with whom they worked. Indicators of implementation fidelity and student engagement were obtained at posttest to reflect the students’ experience with the program during implementation.
Results
Impact of the CEA/COA Program on Student Outcomes
Table 4 presents results of the impact estimation on the intervention and control groups’ posttest CAAP Critical Thinking test scores and credit acquisition adjusted for the variables in the model, including the pretest. The impact analyses revealed that the treatment group had an adjusted mean scale score on the CAAP Critical Thinking test of 58.06, and the control group had a mean scale score of 58.77. The difference between the treatment group mean and the control group mean was −0.71, a statistically significant negative impact. The effect size for the CAAP Critical Thinking posttest difference was −.14 of a standard deviation, considered a small effect (Hill, Bloom, Black, & Lipsey, 2008). The intervention students’ adjusted mean number of credits acquired was 1.99 credits fewer than the control group students’ adjusted mean number of credits acquired, also a statistically significant negative impact, with an effect size of −.43 of a standard deviation, considered a moderate effect (Hill et al., 2008).
Impact on Confirmatory Outcomes.
Note. CAAP = Collegiate Assessment of Academic Proficiency.
Sensitivity analyses were conducted to check the robustness of confirmatory analysis findings to methodological decisions, that is, to test whether the impact findings were affected by analytic decisions made by the researchers or by the analytic methods used. Four types of sensitivity analyses were conducted. Results confirmed the findings of the benchmark impact estimates, with only two exceptions. In each case, analyses yielded a similar parameter estimate for the CAAP Critical Thinking score, but the result was not statistically significant, likely due to the reduced statistical power associated with the sensitivity analyses.
The impact of the CEA/COA program was also estimated for each of four exploratory outcomes. None of these analyses yielded statistically significant impacts (see Table 5). Effect sizes ranged from −.09 for educational aspirations to .06 for achievement motivation, all considered small effects (Hill et al., 2008).
Impact on Exploratory Outcomes.
Impact of Program Implementation and Student Engagement on Student Outcomes
Analyses were conducted to examine the influence of differences in program implementation and student engagement on each outcome measure. Program implementation measures included the presence of resources that support participation, course instructor support, educational staff support, and peer support. Student engagement measures included academic engagement, motivation for educational advancement in the long term, and completion of program components.
Program implementation
One of four implementation variables was significantly related to student outcomes. Student ratings of resources that support participation had a statistically significant relationship with CAAP Critical Thinking test scores, credit acquisition, achievement motivation, and institutional climate after controlling for the other variables in the model. On average, student scores on the CAAP Critical Thinking test decreased by .63 points for every point increase on the measure concerning perceptions of resources that support participation. Similarly, student credit acquisition decreased by .65 points for every point increase on that measure. There was a slight increase (by .12 points) on the measure of institutional climate and a slight decrease (by .08 points) on the achievement motivation measure associated with a 1-point increase on the resources measures. All of the program implementation and student engagement measures ranged from 1 to 4. This means that a student who had the maximum value on the measure concerning resources that support participation would be expected to have a CAAP Critical Thinking test score that is approximately 2.5 scale score points lower than a student who had the minimum value on that measure. Similarly, the highest rating on that measure would correspond to a decrease of approximately 2.6 credits acquired relative to someone with the lowest score.
Student engagement
One of three student engagement variables was significantly related to student outcomes. Academic engagement had a statistically significant relationship with educational aspirations, achievement motivation, personal development, and institutional climate after controlling for the other variables in the model. On average, educational aspirations increased by .44 points for each point increase in student academic engagement, which is roughly equivalent to increasing aspirations half way from obtaining some postsecondary credits to earning an associate of arts degree, or half way from earning an associate of arts degree to earning a bachelor’s degree. Stated differently, if a student’s response on the academic engagement scale increased from a 2 to a 3, then the student’s educational aspirations would be expected to increase from obtaining some postsecondary credits to earning an associate of arts degree, or from earning an associate of arts degree to earning a bachelor’s degree. Similarly, there were increases in student ratings of achievement motivation (by .34 points), personal development (by .30 points), and in institutional climate (by .12 points) for each 1-point increase in academic engagement.
Discussion
Results comparing the impact of the CEA/COA program with other postsecondary academic programs for incarcerated students show that there were significant negative findings associated with the two confirmatory outcomes. Students in the CEA/COA program had average critical thinking scores that were about three fourths of a point lower, as measured by the CAAP test (effect size = −.14) and acquired approximately two fewer credits, on average (effect size = −.43), than students from control sites after 1 year. These are considered small and moderate effects, respectively (Hill et al., 2008). There were no significant group differences associated with the four exploratory outcomes examined (educational aspirations, achievement motivation, personal development, and institutional climate). Whether the small group difference in CAAP scores represents a meaningful difference in acquisition of critical thinking skills is unclear. The difference in credit acquisition across groups was larger, however, suggesting that CEA/COA students were less successful in making progress toward degrees than their counterparts in control sites. The difference represents slightly less than a typical 1-semester course of 3 credits out of approximately 60 credits needed to earn an associate of arts degree. This negative finding may reflect implementation problems and lack of commitment by staff at some CEA/COA sites to facilitate student participation in courses that were identified through analysis of qualitative case study data (Meyer & Fredericks, 2011).
Programming in most control sites included direct instruction by instructors from local colleges and ITV. These modes of course delivery are arguably more engaging than recorded lectures provided via satellite and DVD and may explain why students from control sites rated support from course instructors, prison education staff, and peers as being significantly higher than students from CEA/COA sites. Interestingly, however, measures of program implementation were not consistently related to student outcomes, and only one of three measures of engagement was consistently related to student outcomes. Measures of support (by peers, course instructors, and education staff) were not significantly related to student outcomes, and measures of resources that support participation, such as access to needed books and learning materials, tended to be negatively associated with student outcomes. The measure of academic engagement was positively associated with student outcomes.
While analyses of the CEA/COA program impact revealed no positive findings relative to programs in control sites, they suggest that an effectively implemented distance education postsecondary academic program has the potential to achieve outcomes that are similar to those of other types of programs that include traditional on-site classroom instruction. Because they represent lower cost and more easily accessible education programs for many prisons, distance education programs such as CEA/COA may warrant consideration, particularly for prisons without access to nearby community colleges. Differences in ratings of program implementation across CEA/COA sites suggest that supports by course instructors, prison education staff, and peers could be enhanced to improve student outcomes, particularly for distance education programs. Given the potential increases in efficiency and access related to distance education, future research should more closely examine its impact as well as associated cost considerations related to technology, facility space, staff oversight, and other resources. Future research should also look more closely at the impact of other types of correctional education that focus on basic skills and secondary education, models for postsecondary correctional education that better facilitate the transition to college-level work for students with weak educational backgrounds, and postsecondary programs that integrate academic and vocational components.
Because student academic engagement emerged as a factor in student success, ways to better engage students in postsecondary correctional education programs ought to be explored as well. For example, 1st-year seminars have become common throughout colleges and universities in the United States and have been shown to increase the academic performance and persistence of participants by providing a regularly scheduled meeting time with an instructor for new college students to foster academic and social integration (Pascarella & Terenzini, 2005). Analogous activities might be explored by community colleges offering courses in prison settings. Developing and supporting learning communities in which students share knowledge, experiences, and responsibility for learning (Tinto, 2003) also offer great potential for increasing student engagement in prison settings.
This study is not without limitations. The study results pertain only to the CEA/COA program and should not be generalized to all correctional educational programs or to other distance-learning programs for incarcerated students. Furthermore, the results of this study are relevant only to the particular implementation of the CEA/COA program observed in this study. Different results may be achieved for the CEA/COA program under different implementation conditions. Caution is also warranted when generalizing the results of this study to all students and prisons. Although the sample of prisons and students was diverse, different results may be achieved when implementing the CEA/COA program in other prisons with different demographic or other characteristics.
This study was one of the first, if not the first, large-scale cluster randomized trials to be undertaken in correctional education, demonstrating that such a study is possible in prison settings. It is not uncommon for methodologically rigorous and large-scale cluster randomized trials in education research to yield few findings of statistically significant positive effects (e.g., Bernstein, Rappaport, Olsho, Hunt, & Levin, 2009; Borman et al., 2005; Gleason, Clark, Tuttle, & Dwoyer, 2010). But given the large number of incarcerated adults in the United States, and given the fact that public investments in corrections and correctional education continue to be substantial, it is important to continue to assess the impact of programs that serve this population, focusing on those interventions that show greatest promise.
For many policymakers and practitioners with an interest in correctional education, the ultimate measure of impact is the postrelease success of incarcerated students who participate in these programs. As part of this study, follow-up telephone interviews were conducted with students who were released after participating in CEA/COA or control programming in early 2011 (Meyer & Fredericks, 2011). Only a small subsample of released students could be reached for these interviews, representing only about one third of released study participants at that point in time. While the response rate raises questions about generalizability of these findings and did not allow for valid CEA/COA and control group comparisons, interviews with 143 respondents indicated that, at the time of the interview, about three fourths were employed, just over one third were enrolled in college, and over three fourths had plans to complete a college certificate or degree. Most students reported that college courses taken in prison were helpful in terms of finding employment or enrolling in college after release, indicating that the experience provided skills, afforded new opportunities, raised their self-confidence, provided a positive direction in their lives, gave them organizational skills, and helped them understand that they were capable of success in academic and professional endeavors. Future research on the impact of education programming in these settings should consider these longer term outcomes and continue to apply a high level of methodological rigor to ensure a strong basis for conclusions about the impact.
Footnotes
Acknowledgements
The authors would like to thank Cindy Borden, Penny Richardson, and Steve Steurer; the administrators, principals, instructors, staff, and students at participating sites; and state department of corrections administrators for their support and involvement in this research.
Authors’ Note
The views expressed are those of the authors and do not necessarily reflect those of the Institute of Education Sciences.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the U.S. Department of Education’s Institute of Education Sciences grant (R305B070077).