Abstract
Despite decades of research indicating classrooms shape student engagement, learning, and development, there is a dearth of empirically grounded research focusing specifically on observed classroom engagement as a predictor of student outcomes in community colleges. This article describes the development of a qualitatively grounded, quantitative classroom-level engagement measurement protocol designed for this purpose. We provide evidence for the measure’s validity and reliability via confirmatory factor analyses and descriptive analyses that offer a snapshot of the information this measure can generate. Furthermore, we examine a two-level structural equation regression model that uses student survey data from students nested in observed classrooms. We then review our results in light of the relevance this measure has for researchers and educators in community colleges.
D
Using a process lens, classroom engagement is defined as “what happens in classrooms” (Lawson & Lawson, 2013). Classroom transactions among the various individuals within the classroom—between teachers and students and students with one another—are central to classroom engagement. Research focused on elementary and secondary schools that operationalize and measure classroom engagement at the observable classroom level has established that well-performing classrooms tend to be high in emotional support, instructional quality, and organization; these characteristics are, in turn, related to learning (Pianta, La Paro, & Hamre, 2008). In higher education, evidence has established that student engagement broadly defined is critical for college completion and transfer (Harper & Quaye, 2009; Kuh, Kinzie, Cruce, Shoup, & Gonyea, 2006; Pascarella & Terenzini, 2005). Empirically, much less is known about how specifically classroom engagement may function. Previous cross-sectional data using student self-reports (Deil-Amen, 2011) and qualitative classroom studies (Cox, 2009) offer some insights into how postsecondary classrooms may shape student outcomes in community colleges. Although these studies provide evidence of meaningful links between student- and teacher-driven factors and student outcomes, they fail to systematically explore the patterns of relationships between all actors and engagement processes in community college classroom settings.
This limitation is in part due to the lack of measurement tools specifically designed to systematically assess engagement at the ecological level of the college classroom. A rigorous observational tool would help educators expand their understanding of how the classroom setting can best support student development. Researchers interested in these constructs at the elementary and high school levels have made great headway in examining classroom processes using tools designed to do so (Kindermann, McCollam, & Gibson, 1996; Pianta & Hamre, 2009), but the higher education literature has yet to make this leap. We must begin to consider how engagement functions at various ecological levels over time if we are to meet the higher education goals set forth by current policy in the United States (Lawson & Lawson, 2013). To address this need, we describe and validate a novel, classroom-level engagement measurement protocol developed for use in community college settings serving large proportions of low-income, minority, and immigrant-origin students.
The Theory of Classroom Interactions
In this study, classrooms are the social setting and unit of analysis of particular interest. Classrooms, like all social settings, involve multiple actors dynamically interacting over a sustained period of time to achieve certain goals (Blumenfeld, 1992; Pianta & Allen, 2008; Tseng & Seidman, 2007). During the course of repeated interactions, institutional norms become established, leading to varying degrees of organizational climate (Sarason & Klaber, 1985). This framework stems from systems (Super & Harkness, 1999; Tseng & Seidman, 2007) and ecological (Bronfenbrenner & Morris, 1998) theories that consider the person in context. These theories posit that variation in student outcomes is attributable to both individual- and classroom-level factors, which are theoretically and empirically distinct from one another.
Capturing complex interactions and processes of the classroom has been a focus of extensive research in primary and secondary education, driven largely by the desire to help teachers identify and address the components of classroom settings that best support their students’ academic and social-emotional development (Hamre & Pianta, 2005; Pianta & Allen, 2008). Using systematic classroom observations, for example, Pianta, Belsky, Vandergrift, Houts, and Morrison (2008) have found that relationships between teachers and students as well as cooperative peer relationships, and the positive classroom interactions that result from these relationships, have been linked to motivation and engagement for elementary and high school students (Pianta & Allen, 2008).
Extant higher education literature also provides support for teacher–student relationships (Pascarella & Terenzini, 2005) and peer relationships (Booker, 2007) as being crucial to supporting learning. Studies suggest that when students are meaningfully challenged in ways that scaffold their learning in supportive relational environments, they are more likely to be both socially and academically integrated in college (Nelson Laird, Chen, & Kuh, 2008). Within secondary and postsecondary education, however, much of the research has remained “rich and descriptive but idiosyncratic” rather than systematic (Pianta & Hamre, 2009).
Student Engagement in Higher Education
Within the higher education literature, a “tangled web of terms” (Wolf-Wendel, Ward, & Kinzie, 2009) have been used to label student engagement, including “engagement,” “integration,” “involvement,” and “social belonging,” each describing intertwined concepts of this domain. In postsecondary education, student engagement broadly has been defined as “the time and energy students invest in educationally purposeful activities and the effort institutions devote to using effective educational practices” (Kuh, Cruce, Shoup, Kinzie, & Gonyea, 2008, p. 5). This definition recognizes the interactions of students with institutional agents. Although the K–12 conceptualization of engagement focuses largely upon what occurs within the classroom, the higher education use of the concept includes processes that occur across the campus both inside and outside of the classroom (Astin, 1993; McClenney & Marti, 2006; Tinto, 1993).
Building on theories that attempt to explain the individual, societal, and campus setting factors that contribute to student persistence, Tinto (1993) developed a sociologically influenced and often cited theory of student engagement that takes into account the resources that students bring with them, including demographic variables (e.g., being first generation to college, socioeconomic status) and academic preparation (e.g., academic English skills, entrance scores, study skills). These resources interact with institutional setting–level factors (in and out of the classroom) in ways that lead to distinct academic outcomes (e.g., grades, persistence, graduation or transfer). Tinto’s theory is premised on the notion that “integrating experiences increase involvement, engagement, and affiliation” (Hurtado & Carter, 1997, p. 324) that are crucial to students’ academic progress and persistence. Tinto posits that there are social as well as academic systems to which students may be integrated. Extensive research using this academic and social engagement model in higher education has been highly predictive of persistence versus departure for mainstream students attending residential 4-year colleges (Elkins, Braxton, James, 2000; Pascarella & Terenzini, 2005; Tinto, 1993).
Notably, in this theory of campus engagement, although there is recognition of the role of the classroom, very little emphasis is placed upon it. Furthermore, though paradigm shifting, the bulk of the research using this model has been done in residential 4-year institutions (Valentine et al., 2009). Social integration may be less meaningful for nonresidential community college students who spend little time on campus (Hurtado & Carter, 1997; Tinto, 1993). In this specific context, the classroom may be particularly important as this is the space of most interpersonal contact for students attending community colleges.
Engagement Within the Classroom
Extant literature in higher education generally mirrors concepts found in the primary and secondary school literature though these domains have been more deeply researched in K–12 education. In a comprehensive review of student engagement literature in primary and secondary education, Lawson and Lawson (2013) noted that engagement at the classroom and student levels is constantly being shaped by the interactions of all members of the classroom and is in turn related to student and instructor outcomes. Said another way, engagement operationalized at the level of the classroom consists of a set of transactional factors that can each uniquely and collectively predict engagement at the level of the student. Lawson and Lawson call for researchers to broaden their agendas to include the dynamic and interdependent ways that engagement in learning functions in the context of ecological systems.
Figure 1 provides a conceptual model that integrates domains of engagement at both the classroom and student levels, and links them to related student outcomes that have emerged in the K–12 and higher education literatures. Specifically, the K–12 literature, which has benefited from the extensive use of standardized classroom observation methodologies, has provided substantial evidence of the importance of classroom-level factors as distinct and uniquely contributing to student-level outcomes. In contrast, the higher education literature has relied almost exclusively on student and teacher self-reports and qualitative data of engagement as predictors of student-level outcomes. The current study aims to validate a tool that would allow the field to explore the following engagement domains at the level of the community college classroom.
Integrated model of engagement domains and student outcomes from the K–12 and higher education literatures.
Academic Engagement
The first conceptual dimension is academic engagement—intentional activities related to involvement in courses such as attending classes, class participation, reading assignments, studying for tests, and turning in assignments are related to both grades and persistence (Kuh et al., 2008; Martin & Hand, 2009; Robbins et al., 2004; Svanum & Bigatti, 2009; Wolf-Wendel et al., 2009). Evidence is accumulating that behavioral efforts related to learning contribute to class grades and persistence outcomes in 4-year institutions as well as in less studied community college settings (Kuh et al., 2008; Robbins et al., 2004; Svanum & Bigatti, 2009). In the higher education literature, academic engagement seems most akin to what is termed behavioral engagement in K–12 education, which focuses on conduct or the degree to which students engage in behaviors that reflect they are involved in the classroom (e.g., doing assignments, attending, complying with school rules; Lawson & Lawson, 2013). The emphasis on this literature in higher education is purely on the efforts made by the student (Dowd & Korn, 2005, as cited in Bensimon, 2007; Kuh, Kinzie, Cruce, Shoup, & Gonyea, 2006), failing to consider the interactional nature of learning (Bensimon, 2007; Vygotsky, 1978).
Relational Engagement
The second conceptual dimension is relational engagement (i.e., social belonging, social integration, inclusion, involvement)—the degree to which students feel supported by and connected to their peers, their instructors, or the college’s personnel (Bensimon, 2007). Relationships play a crucial role in serving to build confidence and encourage students to redouble their efforts when motivation fails (Bensimon, 2007; Conchas, 2001; Hurtado & Carter, 1997). The concept of relatedness to “school peers, teachers, and the school overall” in the K–12 literature has been variably referred to as affective/emotional engagement (Finn & Zimmer, 2012; Fredricks, Blumenfeld, & Paris, 2004; Voelkl, 2012) as well as relational engagement (Suárez-Orozco, Pimentel, & Martin, 2009), and aligns nicely with the work done in higher education. However, within higher education, much of the work on dimensions of relational engagement tends to focus on the out-of-classroom environment (e.g., relationships with advisors, campus peers, etc.) rather than the classroom environment (Bensimon, 2007; Hurtado & Carter, 1997).
Cognitive Engagement
Finally, higher education has operationalized cognitive engagement in several ways. For example, the National Survey of Student Engagement (NSSE; 2006), a student self-report measure, taps into active and collaborative learning (i.e., student-centered instructional practices) and the overlapping constructs of deep learning (i.e., the way faculty promote higher order thinking, analysis, and integration of learning) and academic challenge (i.e., both the amount of academic work students are asked to do and the emphasis of this work on higher order thinking skills). Studies using student self-reported data have found that active and collaborative learning are associated with higher grades (Kuh, 2007), higher than expected persistence rates (Braxton, Milem, & Sullivan, 2000; Nelson Laird et al., 2008), and moral reasoning (Mayhew, Tricia, Ernest, Thomas, & Charles, 2012). In the K–12 literature, cognitive engagement has been defined as the degree to which students “think deeply about ideas and concepts” (Lawson & Lawson, 2013, p. 5) and are curious about and interested in what they are learning (Suárez-Orozco et al., 2009).
Qualitative approaches to observing processes related to cognitive engagement have been primarily concerned with the role of the instructor in promoting cognitive engagement (Bonwell & Eisen, 1991; Grubb, 1999). Other work has theorized effective teaching practices and the connection to student learning as evidenced by reading widely, integrating knowledge, discussing ideas with others, and applying knowledge to real-world situations (Pascarella & Terenzini, 2005; Ramsden, 2003; Tagg, 2003). Instructor press in the K–12 literature (Lee & Smith, 1999) posits that quality and type of instruction are key to cognitive development and learning academic content. Furthermore, other academic practices, such as communicating high expectations (e.g., demandingness) and supporting students when they fall behind (e.g., responsiveness), can also predict outcomes such as cognitive engagement and achievement as measured by increased reading scores (Farrington et al., 2012; Lee, 2012; Sebring, Allensworth, Bryk, Easton, & Luppescu, 2006). However, no studies to date in either the K–12 or higher education fields have focused on a setting-level measure of cognitive engagement in spite of recent work that suggests the dynamic, multilocational, and situated nature of engagement processes (Kahu, 2013).
College Classrooms Matter
Students’ perceptions of classroom experiences that relate to instructor-driven practices have been linked to the development of a number of student outcomes, including cognitive development, knowledge acquisition, skill development, educational goal setting, interpersonal skills, and greater involvement in educational activities (Astin, 1993; Cabrera, Cabrera et al., 2002; Chickering & Reisser, 1993; Colbeck & Terenzini, 2001; Pascarella & Terenzini, 2005). For example, a number of studies have linked students’ perceptions of instructional quality to gains in knowledge and cognitive development (Pascarella et al., 2011; Pascarella et al., 2008) as well as student persistence (Braxton, Bray, & Berger, 2000; Braxton, Milem, & Sullivan, 2000; Tinto, 1997). Similarly, a substantial amount of work has established links between interactions with faculty and a host of overlapping positive student outcomes, including gains in social self-confidence and leadership skills (Anaya & Cole, 2001; Kuh & Hu, 2001; Lundberg & Schreiner, 2004; Sax, Bryant, & Harper, 2005). Faculty practices within classrooms, such as interacting with students and challenging students academically, have been shown to increase the levels of reported student engagement (Nelson Laird et al., 2008; Umbach & Wawrzynski, 2005), highlighting the importance of instructor expectations and relationships for promoting overall engagement in higher education classrooms.
Research focused specifically on classroom climate has used whether or not students actively engage in discussion as a proxy for engagement. This research has largely been ethnographic in nature and has been limited to studies in humanities and social science classrooms where discussion is central. In this body of work, the onus for establishing classroom climate has largely been on the instructor alone (see Howard, Zoeller, & Pratt, 2006, for a review). Many of these studies operationalized levels of participation as the observed number of student interactions in classroom discussion. However, research centered on discussion alone does not account for multiple types of engagement dimensions that occur across classrooms and in other disciplines or programs.
Other qualitative research has attempted to systematically observe community college classrooms to better understand how instructional practices interact with student engagement (Cox, 2009; Grubb, 1999). These studies suggest that student engagement depends primarily on the instructor’s practices and teaching skill, rather than the mode of instruction (i.e., whether the classroom is lecture-based or collaborative; Grubb, 1999). More specifically, a combination of instructors’ interpersonal authority, ability to reduce the “fear factor,” and mastery of teaching that constitutes the elements of “successful teaching” promotes engagement (Cox, 2009). Studies on instructor effectiveness indicate that well-organized and clear instruction allows for the development of academic and cognitive outcomes, which in turn is linked to increasing the likelihood of persistence (Braxton, Bray, & Berger, 2000; Braxton, Milem, & Sullivan, 2000).
Others suggest that it is also important to consider a more dynamic view of student engagement that includes peer interaction. For example, Fassinger (1997) found that examining classrooms as social groups with their own particular characteristics was more strongly associated with student engagement than individual student or instructor characteristics alone. Peers have the potential to be both direct and indirect contributors to classroom climate. For instance, students’ with shared attributes, experiences, and ideologies may be more likely to trust and emotionally support one another. Conversely, classrooms with sets of students that do not interact in respectful ways toward one another may promote a set of classroom norms that contribute to decreased classroom relational and academic engagement. Peer interactions have been associated with improved learning outcomes in community college contexts in particular (Booker, 2007).
Much of the research in higher education focusing on classrooms has examined factors that shape involvement of students, including characteristics of the classroom such as size (Tinto, 1997) and faculty effectiveness (Nelson Laird et al., 2008) as well as student attributes such as gender, feelings of self-confidence or fear of speaking, and student learning styles (Bailey & Morest, 2006; Cabrera et al., 2002; Weaver & Qi, 2005). This body of research has tended to rely heavily on self-reported student survey data and has yet to be substantially informed by empirical studies using observational data collected at the classroom level. Furthermore, extant research on classrooms has largely been conducted at 4-year institutions.
Community Colleges and Classroom Engagement
Community colleges are a particularly important higher education context in which to consider the classroom. For many students, this is the primary setting in which they spend their time (Shamah & Ohlsen, 2013) and likely the only avenue of student engagement open to them. Furthermore, although the conceptual model of higher education student engagement was developed and largely studied with 4-year college students, today the majority of students begin their academic careers as community college students (Deil-Amen, 2015). Students are also more likely to be “nontraditional.” Of the 11.5 million students enrolled in community colleges during the 2007–2008 academic year, 40% were the first in their families to attend postsecondary school, 40% identified as ethnic/racial minorities, and 59% received some form of financial aid (American Association of Community Colleges, 2013). Community colleges have become particularly important settings for access to higher education and subsequent economic and social mobility for a large number of low-income, ethnic minority, and immigrant-origin students (Bloom, 2004).
The populations community colleges serve—their missions and structural characteristics—are all substantively different from those of 4-year institutions (Bailey & Morest, 2006). The learning contexts of community colleges serving these students, however, are largely understudied (Teranishi et al., 2011). Given classrooms are the primary and most proximal settings where learning takes place (Blumenfeld, 1992; Bronfenbrenner & Morris, 1998; Pianta & Allen, 2008), it is imperative that we gain a deeper understanding of in what ways and to what degree community college classrooms engage their students in learning.
The Current Study
The purpose of this study was to develop a standardized way to observe classroom interactions and engagement for higher education settings that would be meaningful across disciplines and contexts in linking classroom engagement to student outcomes and informing evidence-based teaching (Groccia & Buskist, 2011). To date, classroom observation protocols in higher education have typically been used for peer observation to evaluate and improve teaching (Hunzicker & Lukowiak, 2012). This focus on teaching efficacy and effectiveness has resulted dampened the study of student-to-student interactions or student-to-teacher interactions as they relate to teaching and learning. Furthermore, although student engagement has been considered as an outcome based on student reports, engagement of the whole class has not been considered using classroom-level empirical observation tools as has been done in K–12 education.
Research in 2- and 4-year colleges has demonstrated that although academic background is important in contributing to academic success, characteristics in the institutional setting also play a crucial role (Astin, 1993; Hurtado & Carter, 1997; Peña, Bensimon, & Colyar, 2006). To improve student outcomes in community college settings, it is critical to better understand how both student factors and classroom processes are related to these outcomes. Examining classroom engagement (at the ecological level of the classroom) in contrast to student engagement (at the individual level) as it relates to student outcomes is clearly a novel area of study in the field of higher education. To date, the best approximation of classroom process has come in the form of qualitative research and student perceptions of aspects of the classroom experience. Additional tools are needed that allow researchers and educators to systematically observe variation in multiple engagement domains at the classroom level. Observational tools that assess classroom processes can be used to better understand community college settings as well as identify ways to help instructors, who typically do not have a lot of training working with high-need students (Grant & Keim, 2002; Grubb, 1999; Twombly & Townsend, 2008), to improve the overall learning climate of their classroom.
To address this need for a systematic measure of classroom processes, this study describes the development of a qualitatively grounded, quantitative classroom-level (Pianta & Hamre, 2009) engagement measurement protocol. We then examine the overall construct validity of our protocol in two ways. First, we conduct a two-level confirmatory factor analysis (CFA) that utilizes available student survey data from students nested in observed classrooms. We include descriptive statistics to provide a snapshot of what type of information this protocol can offer to researchers and educators. Second, we estimate a two-level regression model to examine both the convergent and divergent validity of our tool with respect to correlations between classroom- and student-level variables. We review our results in light of the relevance such a measure has for researchers and educators in community colleges.
Method
Data for this article come from a multiphase embedded mixed-methods study (Creswell & Plano Clark, 2011). The aim of the study was to examine the relationship between classroom and campus settings and academic engagement and performance in community college settings, focusing on the experience of immigrant-origin students (Suárez-Orozco et al., 2015). This article focuses on the analyses of classroom engagement assessed using the Community College Classroom Observation (CCCO) protocol especially designed for this study (see details below) and also draws on student survey data.
Setting and Participants
Three distinct community colleges in the New York City metropolitan area were selected to participate in the study with the explicit intention of including institutions with varying campus-level characteristics and contexts. All participating community colleges offer 2-year public associate’s degree programs and serve low-income, ethnic minority, and immigrant-origin commuter populations. Located in a low-resource neighborhood, Taino (all campus names are pseudonyms) serves predominately Latino (64%) and Black (31%) students. In 2012, only 2% of the students were White and 3% were Asian/Pacific Islander. More than 90% of student body members report speaking a language other than English at home. Located in the burgeoning downtown section of a large urban center, Domino, the second school, focuses heavily on technological education and serves diverse population of students: 32.5% Black (non-Latino), 33.2% Latino, 19.2% Asian/Pacific Islander, and 11.2% White (non-Latino). Forty percent of the students were born outside of the United States, coming from 134 countries, and 62% report speaking a language other than English at home. The third school, Oakmont, although a commuter school, physically resembles more traditional 4-year university campuses. It is located in an affluent suburban county known for long-standing class-based (i.e., socioeconomic) segregation. Reflecting the shifting demographics of the county, the student population has become increasingly diverse. Forty-two percent of the student population is foreign-born, and 49% of the students identify as White, 28% as Latino, and 21% as Black.
The sample included 57 classrooms across the three participating community college campuses in the New York City metropolitan area (Taino, n = 22; Domino, n = 18; Oakmont, n = 20) and were conducted in the 2011–2012 academic year (fall 2011, n = 20; spring 2012, n = 40). Data collection began 2 weeks (i.e., after add/drop period) after classes commenced and ended 2 weeks (i.e., before final exams/reading period) prior to the end of the semester. Classrooms, on average, had 20 students (SD = 8.42) in attendance when observed. Of the total sample, 28.3% (n = 17) of classrooms were developmental/remedial, 48.3% (n = 29) were general education, and 23.3% (n = 14) were vocational/elective. Classrooms included students of diverse racial–ethnic and immigrant-origin status from primarily low-income families. The median length of classes was 75 minutes (range = 50–200 minutes). Due to the variation in length of classes and alignment with previous methods for observing classrooms, observations ranged from 2 to 8 structured 20-minute segments (Pianta & Hamre, 2009; see procedures for more detail). The breakdown of observation segments across classes was as follows: 10% (n = 6) of classrooms included two segments, 48.3% (n = 29) of classrooms included three segments, 11.7% (n = 7) of classrooms included four segments, 8.3 (n = 5) of classrooms included five segments, 15% (n = 9) of classrooms included six segments, and 6.6% (n = 4) of classrooms included seven or eight segments. Classrooms with varying segments did not differ significantly based on classroom-level demographics (e.g., instruction type, class type).
The CCCO instrument was evaluated using survey data from 313 students nested in the 57 classrooms that were observed by the research team. After classroom observations were completed, all students in the classroom were invited to participate in a student survey after class, resulting in a convenience sample of, on average, 5.49 students per classroom with a range of 1 to 15 students per class. All classes with at least one student survey are kept in the analysis as they contribute to more precise estimates of between-level parameters (Muthén & Muthén, 1998–2012). Fifty-eight percent of participants were female, and 72.8% were of first-generation (33.9%) or second-generation (38.2%) immigrant origin. Participants were predominately ethnic minorities from low-income backgrounds: 41.1% were Latino, 27.3% were Black or African American, 8.5% were Asian, 12.9% were White, and 9.4% identified as Other or Mixed race. Although we were unable to compare whether those students were representative of other students in their classrooms, the demographic distributions of ethnicity, gender, and immigrant generation status are similar to whole campus demographics for each school.
Procedures
Investigators approached college officials to first obtain permission to undertake the study at each campus. Once officials agreed to participate in the study, investigators then sought formal approval through the appropriate city and state higher education public school administration, each school’s respective Institutional Review Board (IRB), and New York University’s (NYU) IRB (i.e., the investigators’ institution). All study methods, procedures, and materials were approved by city and state administrators and all IRB committees. Study classroom data collection took place from February 2011 to August 2012, and included 60 structured classroom observations as well as 319 student surveys (matched to student records) from 1 to 15 students nested within each of the observed classrooms. Research assistants recruited faculty members in a variety of ways. First, department chairs were contacted and asked for permission to recruit faculty. After permission was given, assistants emailed faculty members and placed flyers in faculty mailboxes, or recruited members directly at faculty meetings, or through referrals. Targeted recruitment was used to ensure classrooms were balanced in representation across the three campuses, especially in English and math departments, developmental education, and vocational programs. All participating faculty members agreed to have their classrooms observed. Students nested in observed classrooms completed self-report surveys using an online software package (Qualtrics) directly after their class was observed in a designated school setting (e.g., computer lab) overseen by graduate research assistants. Students received US$25 cash or an equivalent Amazon gift certificate for completing the survey.
Classroom observations were conducted by a highly trained multiracial/ethnic research team of graduate students. Observations occurred during regularly scheduled class times. Observers conducted classroom observations to assess classroom-level engagement using a novel, theoretically grounded classroom observation instrument developed for this study, informed by the procedures and methods used by the Classroom Assessment Scoring System–Secondary (CLASS-S) observational protocol (La Paro, Pianta, & Stuhlman, 2004).
To ensure high levels of interrater reliability, all members of the research team who participated in data collection underwent rigorous CCCO protocol training. Prior to training, all observers read a detailed manual with extensive descriptions of dimensions and rating anchor points and participated in a 2-day guided practice observation training, which included extensive coding and feedback sessions of videotaped community classroom footage. Once training sessions were completed, observers were required to pass a reliability test, which included watching and coding two videotaped classrooms before being approved as a data collector. Criterion for passing the observation training test was an 80% match or higher (within 1 scale point on a 5-point scale ranging from 1 = low engagement to 5 = high engagement) on a set of master-rater coded scores. All observers met this criterion at the end of the training session or within the following 1 to 2 weeks after undergoing additional training. All observers were required to take a reliability retest in the spring semester before resuming data collection. Average interrater reliability was 83% for the first training in the fall across all raters and 91% in the spring across all raters.
Observers were scheduled by research team site leads to conduct observations in all participating community college classrooms during regularly scheduled class hours. To obtain observational data that best represented the typical engagement climate of each classroom, observers followed strict, structured coding guidelines for each of the observed segments. Two observers—one lead/master and one newly trained—arrived early to each class and sat in a place in the classroom where they would be least disruptive to the class (generally in the back or where professors asked them to be seated). Lead raters were advanced doctoral students heavily involved in the creation of the measure, who also conducted the training sessions. Observation periods started 10 minutes after the professor began the class. Observers conducted 20-minute observational segments, which included 12 minutes of observation and 8 minutes of scoring. The number of observation segments was based on the length of each class. All observers rated items across observation segments simultaneously and independently for the length of one entire class period. Lead rater scores were used to calculate mean scores for scales; however, 96% of observations were double-coded to ensure reliability of observations given the measure’s novelty. Interrater reliability was assessed as the degree to which two coders were within one point of each other’s scores (La Paro et al., 2004). Interrater reliability across all classroom scale items was .90. Observers also took structured ethnographic notes to capture information about the classroom characteristics (e.g., technology, seating arrangement, cleanliness, etc.).
CCCO Protocol
The CCCO is a qualitatively grounded, quantitative measure, which taps specific observable behavioral and interactional indicators of three dimensions of classroom engagement—academic, relational, and cognitive. These dimensions, visually represented in the middle panel of Figure 1, were introduced in the literature review. Measurement development was an iterative process informed by two classroom observational protocols in the field—the CLASS-S, an observational instrument developed for K–12 contexts to assess the quality of interactions in high school classrooms (Pianta & Hamre, 2009), as well as the Community College Survey of Student Engagement (CCSSE; http://www.ccsse.org), which assesses community college students’ perceptions of their school environment (see the Appendix, available in the online version of the journal).
The emphasis on interactions as well as the procedures of the CLASS-S provided us with guidance in the development of our instrument though we needed to extend conceptual categories to be applicable to community classroom settings. Pianta, La Paro, et al. (2008) have isolated positive climate, negative climate, teacher sensitivity, and regard for student perspective as separate and important dimensions of classroom Emotional Support. The relationships between teachers and students as well as cooperative peer relationships and the positive classroom interactions that result from these relationships have been linked to motivation and engagement (National Research Council, 2004; Pianta & Allen, 2008). Pianta and his colleagues (2008) have also examined the role of Classroom Organization (e.g., effective use of instructional learning formats, class productivity, and classroom management) as well as Instructional Support (e.g., teacher content understanding, skills in effectively presenting information, ability to stimulate metacognitive reasoning, and the quality of feedback provided to students). These dimensions—emotional support, classroom organization, and instructional support—together serve to stimulate student engagement.
The Center for Community College Student Engagement (CCCSE) has extensively contributed to the research on student engagement in higher education based on large-scale surveys of student and faculty across multiple campuses (McClenney & Marti, 2006). CCCSE developed a Classroom Observation Form based on constructs that emerged from its survey findings. Validity and reliability information were not available for this instrument at the time the study was conducted. The pilot instrument focused on instructor practices, including class organization, subject matter knowledge, supportive teaching style, diverse instructional strategies, and engaging pedagogy across classrooms.
To extend conceptual categories and methodologies from the extant literature to community college settings, the research team conducted an in-depth campus wide ethnography and nine student focus groups over the course of 6 months to additionally inform the content of CCCO. Twenty-one participants took part in three consecutive weekly focus groups that explored the experiences and perceptions of immigrant community college students at our three study sites. Participants were recruited in person by research assistants and from flyers placed in diverse campus spaces that students frequently gathered (e.g., the cafeteria, library, and other “hangout” spaces identified by ethnographic observations). All participants were between the ages of 18 and 25, attended classes full-time at the campus from which they were recruited, and were from diverse ethnic backgrounds and generational statuses. Each focus group session lasted for approximately 2 hours and took place on campus. A standardized protocol was used across each of the campuses with open-ended prompts. Across the three sessions, students discussed challenges they experienced both on and off campus. The last session focused explicitly on students’ experiences in classroom spaces, including student-to-student and student-to-teacher interactions and classroom climate. Each session was audio-recorded, transcribed, and verified by research assistants. An open-coding process using phrases as the units of analysis was employed (Strauss & Corbin, 1990). In addition, emergent descriptive themes from the transcripts were identified. These themes were compared and integrated into a single comprehensive list of coding categories (Miles & Huberman, 1994). In sum, the development of the CCCO is grounded in our review of the extant higher education student engagement literature, our ethnographic and focus group findings, CLASS-S methodology, and CCCSE student and teacher survey constructs.
The CCCO assesses three primary dimensions of classroom engagement: academic engagement, relational engagement, and cognitive engagement (see Figure 1). Each engagement dimension is comprised of items that reflect interactions theoretically and empirically related to these respective dimensions. Items were placed along a scale with behaviorally anchored descriptors of types of interactions between all members in the class (i.e., student–student, teacher–student) on a continuum from 1 = low engagement to 5 = high engagement. Classroom observers completed item ratings on all dimensions for each observational segment. A composite score was created for each item based on an average of the item’s ratings across all observation segments. Table 1 provides a detailed example of qualitative anchors and corresponding scoring for the cognitive engagement curiosity item. Further descriptions of constructs are reported along with CFA results (also see Table 2).
Example of CCCO Protocol Qualitative Anchoring, Quantitative Scoring
Note. CCCO = Community College Classroom Observation.
Two-Level Model Standardized Loadings of Observed Variables on Latent Factors (nj = 57, ni = 313)
Note. CFA conducted using CCCO master-rater classroom observations and student-reported items, χ2(156) = 275.995, p < .001; RMSEA = .049; CFI = .946; SRMR w = .031; SRMR b = .030. CCCO = Community College Classroom Observation; CFA = confirmatory factor analysis; RMSEA = root mean square error of approximation; CFI = comparative fit index; SRMR = standardized root mean square residual.
Student Self-Report Measures
Students nested in observed classrooms reported on their perceptions of their classroom environment along three dimensions that mirrored those of the classroom-level engagement constructs. The far right panel of Figure 1 highlights these student-level constructs as possible outcomes of classroom-level engagement. These dimensions were used to test the overall construct validity of the CCCO protocol. All student-level measures were included in our two-level CFA model and are fully described in the “Results” section of this article.
In line with existing evidence from empirical student-level and qualitative classroom-level studies as well as study ethnographic and focus group inquiries, we hypothesized the following:
Results
In this section, first, we present fit statistics and coefficient estimates for a two-level CFA model, followed by descriptive information on classroom- and student-level factors garnered from this analysis. Next, we present a two-level regression analysis with random intercepts using mean factor scores based on the final two-level CFA model to assess associations between CCCO classroom engagement factors and student self-report variables depicted in Figure 2. A mean factor model was estimated as we did not have the degrees of freedom needed to estimate a two-level structural equation model given the number of parameters such a model would include. These results are intended to test the construct validity of the measure.
Empirical model of classroom engagement in community colleges.
Two-Level CFA Measurement Model
Data reduction and scale development were conducted on all 18 classroom engagement scale items originally included in the piloted version of the CCCO protocol. Engagement items were calculated as the mean score of items across observation segments in each class across each respective item. First, all items were reviewed across both sets of raters to ensure normality assumptions were met. Once normality assumptions were confirmed, inter-item correlations were run to explore potential redundancies at the item level. Two sets of items correlated .85 and .96, respectively, were thought to be redundant, resulting in the random exclusion of one item from each set for further analyses. The remaining 16 items were included in all subsequent analyses. Exploratory factor analysis (EFA) was conducted on the full sample of classrooms using data collected from our secondary classroom raters. The EFA revealed that 13 out of 16 items loaded onto their pre-hypothesized factors. The remaining three items were dropped due to low loadings or cross-loadings. Remaining items fell into three categories representative of the three engagement domains the CCCO aimed to capture: academic engagement, relational engagement, and cognitive engagement. This three-factor solution was then confirmed using a two-level CFA with continuous factor indicators in which student-level factors hypothesized to correlate with classroom-level factors were included to address the multilevel nature of the data. Because the CCCO data EFA was conducted using secondary rater data, the two-level CFA model estimated used data collected by CCCO master raters in addition to student self-report data. All factor analyses were conducted in Mplus Version 7 (Muthén & Muthén, 1998–2012). Table 2 provides factor loadings and standard errors for the final multilevel measurement model in the present study. Fit statistics indicated good fit of the data to the model, χ2(156) = 275.995, p < .001; root mean square error of approximation (RMSEA) = .049; comparative fit index (CFI) = .946; standardized root mean square residual, SRMR w = .031; SRMR b = .030. Below, we provide full descriptions and reliability coefficients for the final classroom- and student-level factors we used to test validity of the CCCO protocol.
CCCO Academic Engagement
CCCO academic engagement (α = .91), measured through the observational protocol, is defined as the extent to which members of the class (i.e., students and instructor) engage in behaviors that reflect they are involved in the classroom, and was comprised of six items: Attentiveness (most of the class demonstrates attentive body language), Rule Compliance (most students follow instructor prompts), Authoritative Content (instructor delivers material authoritatively and responds knowledgably), Engaging (instructor exhibits enthusiasm and interest when engaging with students), Learning Organization (materials and discussion related to clear learning objectives); and Classroom Management (absence of disruptions in classroom) (M = 3.87, SD = 0.63).
CCCO Relational Engagement
CCCO relational engagement (α = .94), measured through the observational protocol, is defined as the extent to which class members appear relationally connected to one another in providing academic support, and was comprised of four items: Comfort (class interactions are relaxed, empathetic, and warm), Validation (class members appropriately praise and support one another’s efforts), Equity of Treatment (class members treat each other equitably/absence of microaggressions), and Fairness/Inclusion (instructor encourages participation of multiple diverse participants) (M = 3.52, SD = 0.68).
CCCO Cognitive Engagement
CCCO cognitive engagement (α = .89), measured through the observational protocol, is defined as the extent to which the class as a whole is engaged in activities and discussion that develops analytical, synthetic, and abstract cognitive abilities. This scale was comprised of three items: Curiosity (the class is involved in activities that generate how and why questions linked to critical thinking), Content Level (content in zone of proximal development; Vygotsky, 1978), and Student Balance of Involvement (the degree to which participation in the cognitive task at hand—discussion, group work, and so on—is evenly distributed among students in the class) (M = 3.36, SD = 0.76).
Peer Academic Engagement
Peer academic engagement (α = .78) was measured with a six-item student self-report scale tapping students’ perception of student academic engagement in their observed classroom (e.g., the students came prepared for class). This scale was developed by the research team based on the CCSSE and study focus group findings to address the dearth of such scales in the extant literature. Scores ranged from “strongly disagree” (1) to “strongly agree” (5), with higher scores indicating higher levels of instructor peer academic engagement.
Peer Relational Engagement
Peer relational engagement (α = .79) was measured with a five-item student self-report scale tapping students’ perception of how relationally connected members of their classroom, including the instructor, were to one another in their observed classroom (e.g., students supported each other in the class). This scale was developed by the research team based on the CCSSE and study focus group findings to address the dearth of such scales in the extant literature. Scores ranged from “strongly disagree” (1) to “strongly agree” (5), with higher scores indicating higher levels of class member relational engagement.
Instructor Press
Instructor press (α = .85), an indicator of cognitive engagement, was measured with a five-item student self-report scale adapted from the Chicago Consortium of School Research student survey (Fergus, Noguera, & Martin, 2014). This scale measures students’ perception of the instructor’s expectations of academic best efforts (e.g., the instructor for this course expects me to do my best all the time) and responsiveness to student needs. Scores ranged from “strongly disagree” (1) to “strongly agree” (5), with higher scores indicating higher levels of instructor press.
Descriptive Analyses
Basic descriptive statistics of classroom-level (CCCO) and student-level variables are presented in Table 3. Based on normed scores, classrooms, on average, had relatively high levels of academic engagement where the majority of the members of the class were on task for the majority of the class period (M = 3.87, SD = 0.63). Cognitive (M = 3.36, SD = 0.76) and relational (M = 3.52, SD = 0.68) engagement had slightly lower mean scores, suggesting inconsistencies in these domains, such that some but not all members of classes, on average, may have been cognitively and/or relationally engaged for some but not all of the class session. On average students nested in classrooms reported feeling strongly that their peers were academically (M = 3.84, SD = 0.60) and relationally (M = 4.03, SD = 0.53) engaged, and that instructor press was strong, meaning instructors were attuned to students’ learning needs in class (M = 4.11, SD = 0.68).
Descriptive Sample Statistics (nj = 57, ni = 313)
Note. CCCO = Community College Classroom Observation.
p < .001.
Two-level Regression Model
The expectation that classroom engagement variables are (or are not) correlated with students’ perceptions peer academic engagement, instructor press, and peer relational engagement in their observed classrooms was tested next. To test this convergent and divergent validity, a two-level regression analysis for continuous dependent variables with random intercepts was estimated using Mplus Version 7 (Muthén & Muthén, 1998–2012), which accounts for the multilevel nature of the data (i.e., nesting of students within classrooms). Maximum likelihood estimates with robust standard errors were used to estimate all parameters. All covariates were grand mean centered (Enders & Tofighi, 2007).
Level 1 of the model included the following student covariates: ethnicity, gender, and immigrant generation status to account for any variation in student outcomes related to these demographic variables. Student ethnicity, a five-level variable accounting for students’ self-report of their own ethnicity (i.e., White, Hispanic, Black, Asian, and Mixed race), was entered as a single covariate (rather than four dummy codes) to maximize degrees of freedom given it was not a central variable of interest. Student immigrant generation status, a three-level variable accounting for students’ self-report of their own immigrant generation status (i.e., first, second, and third or higher generation), was also entered as a single covariate (rather than two dummy codes). In addition, we controlled for Level 1 student engagement variables when regressing Level 2 classroom predictors on Level 1 student outcomes to assess the unique variance explained by classroom-level predictors on each of our respective student-level outcomes. For example, when predicting peer academic engagement as a correlate of classroom academic engagement, we controlled for student-level instructor press and peer relational engagement variables.
Level 2 of the model controlled for campus (i.e., Domino, Taino, Oakmont), class type (i.e., remedial, general education, and vocational classrooms), and semester (i.e., fall, spring) when the class was observed. Both campus and class type were treated as single covariates (rather than two dummy codes) to maximize degrees of freedom given they were not central variables of interest. All covariates included in the final model are potential confounders of the relations between classroom-level predictors and student-level outcomes in the extant literature. As such, they were included to assess whether, above and beyond these variables, classroom-level engagement variables predict student-level engagement outcomes.
Results of the two-level model, presented in Figure 3, indicated excellent fit to the data, nj = 57; ni = 313; χ2(2) = 6.865, p =.032; RMSEA = .088; CFI = .981; SRMR w = .002; SRMR b = .045. Furthermore, intraclass correlation coefficients (ICCs), which represent the ratio of the variance that lies between classrooms to total variance, were calculated for each of the outcome variables. ICCs ranged from .07 to .11 across all models, indicating that between 7% and 11% of the variance in student-level outcomes lay between classrooms. Although modest, this represents adequate variability in student-level variables between classrooms to examine classroom-level engagement predictors of the respective outcomes (Bryk & Raudenbush, 1992; Muthén & Muthén, 1998–2012). Figure 1 specifies hypothesized relations among Level 2 classroom and Level 1 student constructs of interest. Table 4 provides beta coefficients, standard errors, and p values for all variables, including covariates, in the final estimated model.
Final two-level model results, nj = 57; ni = 313; χ2(2) = 6.865, p =.032; RMSEA = .088; CFI = .981; SRMR w = .002; SRMR b = .045.
Two-Level Model Predicting Student Perceptions of Classroom Engagement Using Observed Classroom Engagement Domains
Note. χ2(2) = 6.865, p =.032; RMSEA = .088; CFI = .981; SRMR w = .002; SRMR b = .045; Level 1 covariates: ethnicity (White, Hispanic, Black, Asian, and Mixed race), gender, immigrant generation status (first, second, third+), peer academic engagement, instructor press, and peer relational engagement; Level 2 covariates: campus (Domino, Taino, Oakmont), semester (fall, spring), and class type (remedial, general education, vocational). CCCO = Community College Classroom Observation; RMSEA = root mean square error approximation; CFI = comparative fit index; SRMR = standardized root mean square residual.
First, we tested whether CCCO academic and cognitive engagement significantly predicts peer academic engagement in class. As expected, CCCO academic engagement significantly and positively predicted student perceptions of peer academic engagement in their observed classrooms (b = .154, SE = .052, p = .003); however, cognitive engagement did not predict peer academic engagement. This indicates that classrooms with higher levels of CCCO academic engagement were more likely to have students in the class report perceiving their peers where more behaviorally engaged.
Second, we tested whether CCCO cognitive engagement significantly predicts instructor press, which measures student perceptions of the instructor’s sensitivity to student academic needs in their observed classroom. Instructor press in the context of instruction in the classroom has been linked theoretically to student-level cognitive engagement (Corno & Mandinach, 1983; Pintrich & Schrauben, 1992). As expected, CCCO cognitive engagement significantly and positively predicted students’ perceptions of instructor press in their observed classrooms (b = .127, SE = .061, p = .038). This indicates that classrooms with higher levels of CCCO cognitive engagement were more likely to have students in the class report perceiving their instructors were sensitive to the academic needs of students in their class.
Third, we tested whether CCCO relational engagement significantly predicts student perceptions of how relationally connected members of their classroom are to one another. We expected that students who attended classes in which we observed high levels of relational engagement among all members of the class would be more likely to report their peers as being connected to one another; however, this was not the case. Classroom relational engagement did not predict students’ perceptions of peer relational engagement in their observed classrooms.
Discussion
In this article, we called attention to the need for tools that allow researchers, educators, administrators, and policymakers to systematically describe and assess classroom engagement in community college settings. To address this need, we developed a qualitatively grounded, quantitative classroom-level engagement measure, confirmed it could be administered reliably, and tested the measure’s validity. Our results revealed that students in community college classrooms held mixed views of how engagement played out in their class. Students felt their peers were very academically engaged and course instructors were committed to students’ learning needs, while feeling their peers had low to moderate levels of cognitive and relational engagement in class. Importantly, we found that the CCCO predicted students’ perceptions of how engaged their peers were in class in ways that would be expected based on existing literature. For example, in classrooms with high levels of academic engagement, students were more likely to perceive their peers where more behaviorally engaged, and in classrooms with high levels of cognitive engagement, students were more likely to feel their instructors were sensitive to the academic needs of students in their class.
Although extensive literature has linked student reports of classroom engagement to student motivation and learning outcomes (Reeve, 2012; Skinner & Pitzer, 2012), there has been a dearth of research in higher education that systematically examines these dynamic processes at the classroom level. Lawson and Lawson (2013) focus primarily on student-level research, however, they argue that to adequately address student engagement, the field should broaden this line of research to include the dynamic ways that student engagement functions in the context of ecological systems. While the literature on engagement has clearly recognized the features of classrooms that link to positive student outcomes, surprisingly little quantitative work has been done to systematically elucidate ways in which classroom-level processes shape classroom engagement and in turn student engagement (Lawson & Lawson, 2013). This is fueled by a particular lacuna of measurement strategies that allow for systematic observation of engagement at the classroom level in higher educational contexts (Pianta & Hamre, 2009).
The aim of this research was to fill this void by developing and validating an empirically based classroom observation protocol of classroom-level engagement for use in community colleges. Engagement researchers have been calling for expansion of research in this area that explicitly focuses on factors external to the student (Eccles & Wang, 2012; Lawson & Lawson, 2013; Skinner et al., 2008; Wentzel, 2012). Our measure of engagement considers classrooms as the unit of analysis rather than aggregating up student reports of classroom engagement—the current norm in engagement research. This approach allows for the study of classroom ecologies as indicators of engagement in addition to student and teacher variables of interest. Our approach focused on observable classroom interactions between all members of the class (i.e., students and students, and students and instructors; Tseng & Seidman, 2007) across a wide range of types of community college classrooms. We sought to extend important work conducted at the primary and secondary levels (Pianta et al., 2008), attending to the ecological structure of community college settings serving diverse students.
The CCCO protocol demonstrated promise as a reliable and valid metric to assess classroom-level engagement in community college classroom settings. As the seminal work of Lawson and Lawson (2013) posits, our analyses yielded three distinct but interrelated classroom-level engagement domains—academic engagement, cognitive engagement, and relational engagement (what Lawson and Lawson refer to as affective engagement)—which, taken together, are thought to holistically represent classroom engagement. Given the relatively small sample size of classrooms in which we were able to pilot the CCCO protocol, results provide promising preliminary evidence of the measure’s ability to separately identify all three engagement dimensions while accounting for their correlated nature. This suggests that the CCCO protocol would allow for both the individual and synergistic study of these engagement domains in relation to student-level outcomes.
Our results largely support the construct validity of our model. More specifically, we were able to examine both convergent and divergent validity to assess the extent to which the CCCO was or was not associated with student-level outcomes that have been theoretically related to classroom engagement. First, as expected, we found that observed classroom academic engagement predicted students’ perceptions of how engaged their peers were in class. Thus, there was consistency between our standardized observation protocol and what students reported about their own classmates’ observable behaviors of being prepared for class and participating in class activities. This supports convergent validity.
Also as expected, the CCCO was able to predict students’ perceptions of how attuned and responsive the instructor was to the academic needs of students (i.e., instructor press) in the class based on observed classroom cognitive engagement. This finding supports the convergent validity of our tool and is aligned with qualitative work done in college classrooms that promotes engagement as a dynamic process (Cox, 2006) influenced by more than mode of instruction (Grubb, 1999). Furthermore, numerous empirical studies focusing on student reports of engagement have linked students’ perceptions of instructional quality to cognitive development and knowledge accrual (Pascarella, Salisbury, & Blaich, 2011; Pascarella, Seifert, & Whitt, 2008).
Although the findings related to classroom academic and cognitive engagement support the convergent validity of our tool, classroom relational engagement did not predict students’ perceptions of peer relational engagement supporting divergent validity of our tool. Research in higher education suggests classroom climate—shaped by peer interactions, and characterized as trusting and emotionally supportive—promotes norms that contribute to relational engagement for students (Booker, 2007). However, this research has relied on students’ reports of their own relational engagement rather than on reports of how relationally engaged they perceive their peers to be. It may be that students are well attuned to their own relational engagement with peers, but do not typically pay strong attention to relational engagement dynamics among others in the classroom. In contrast, the trained observers had been taught to consider many ways of observing indicators of relational engagement between class members. Future work should consider student self-reported relational engagement and observed classroom-level relational engagement as potentially distinct correlates of student-level outcomes. Interestingly, students’ perceptions of peer academic engagement (entered as a control) significantly predicted their perceptions of peer relational engagement, suggesting that classrooms where students think their peers are academically engaged may be more likely to believe their peers are also relationally engaged.
Because we measured classroom engagement in community colleges serving large numbers of low-income, immigrant origin, and ethnic minority students, this study was designed to be particularly applicable to schools serving these populations. This is of particular importance given the focus on enhancing engagement outcomes for the most vulnerable youth at all levels of education study, including postsecondary schooling (Lawson & Lawson, 2013). Although the protocol was normed to the community college context in diverse metropolitan settings, the items may have potential applicability to less diverse community colleges as well as 4-year colleges and university classrooms. The observable indicators used to measure each of the classroom engagement domains are universal indicators of engagement, meaning they can be observed in any higher education setting. However, the degree to, and conditions under, which the CCCO tool can predict student outcomes in other higher education settings is still unknown. To use the CCCO in other higher education settings, the tool would need to be normed to the populations and settings being observed, and reliability of raters would need to be cultivated, assessed, and maintained. If these conditions are met, it is likely that the CCCO has the potential to be a valid and reliable measure for the higher education field overall. Future studies should test the CCCO’s applicability in these settings.
Within the complex and multifaceted crisis of low graduation and high transfer rates in community colleges, the CCCO has the potential to be an improvement science tool to be used by both researchers and practitioners to coinvestigate classroom engagement. Bryk, Gomez, Grunow, and LeMahieu (2015) have argued that as a first step to solving problems, whether on campus or within a classroom, we must carefully and narrowly specify “the problem to be solved.” Within improvement science, the CCCO can be used as a tool to aid in developing a deeper understanding of what engagement domains look and feel like in classrooms. The CCCO could also be used as a data collection tool that can inform rapid feedback to determine whether targeted efforts are effective.
Implications
Classrooms are the settings around which the whole educational enterprise is constructed. They are intentionally designed to provide students with tools and supports central to learning. They are bounded spaces where the same individuals meet on a regular basis, over a set period of time, to engage in critical thinking and information exchange. It is imperative that we better understand what community college classroom–level engagement dimensions look like and how they may influence student-level academic outcomes. Furthermore, classrooms are optimal sites for intervention, as all students must engage in classroom learning while attending school. A better understanding of how they engage students can aid in improving these settings and in turn student outcomes.
The CCCO protocol has promise as a tool for both researchers and practitioners seeking to address the needs of increasingly diverse student populations in community colleges and higher education classrooms. The CCCO allows for the observation of both factors and processes that may shape classroom engagement, and thus broadens the current focus of engagement research in higher education. Use of the CCCO would allow researchers to move beyond the current practice of focusing largely on instructor quality to a more dynamic, setting-level approach that considers transactions among all members of the classroom. For intervention scholars, the CCCO could be used to assess unique variation in student outcomes that may be associated with classrooms as a distinct ecology in order to elucidate areas for intervention. More specifically, researchers can use the CCCO to examine correlations between classroom-level engagement domains and other student outcomes of interest long studied in the higher education literature, such as student engagement, grade point average (GPA), credit accrual, attendance, degree completion, persistence, and a number of psychosocial outcomes.
The CCCO can also serve as a tool for evidence-based teaching (Groccia & Buskist, 2011) in higher education classrooms providing educators an assessment strategy to improve classroom engagement. Educators and administrators can use the CCCO as a guide to assess classroom engagement and develop instructor professional development and coaching interventions to address areas of classroom-level engagement that are low or unsatisfactory. K–12 classroom observational tools are already widely used to assess classroom processes and provide professional development to teachers based on observational scores (Pianta & Allen, 2008; Pianta, Belsky, et al., 2008; Pianta & Hamre, 2009). Such a model would allow community college educators to focus on both social interactions that can bolster relational engagement and academic and instructional strategies that can increase academic and cognitive engagement for the whole class.
Limitations
Several limitations are noteworthy. Although quota sampling was used to ensure that classroom and student diversity in the study sample was proportionate to the diversity of respective campuses, both the classroom and student samples were convenience samples. We cannot say with certainty that classrooms and students were representative of all campuses. For example, it is possible that instructors who allowed us to observe their classrooms felt they had more engaging or highly functioning classrooms. Similarly, students who participated in our survey may have been more likely to be engaged in their class or in school as a whole, which could influence how they perceived their class. Future studies can best address this limitation via random selection at the classroom and student levels.
This study was conducted on the east coast in a metropolitan area. To increase generalizability, future work should include regions across the country, including suburban and rural settings. Only 60 classrooms across three campuses were observed during the course of the study, and only 57 classrooms had student-level data available. Thus, descriptive, measurement model, and regression analyses were conducted with a small sample of classrooms. Despite this, measurement and regression model fit statistics were adequate, and analyses revealed significant correlations between variables of interest after controlling for a number of student- and classroom-level covariates. Nonetheless, some of these results may be spurious or underestimated given sample size.
Finally, each participating classroom was only observed at one time, either during the fall or spring of the 2011–2012 academic year, providing us with only cross-sectional data on which to examine the CCCO’s measurement validity. For example, we were unable to look at the CCCO’s predictive validity as it pertains to student learning outcomes (e.g., Does classroom-level behavioral engagement correlate with student-level GPA or credit accrual?). Future work with the CCCO should aim to observe a larger number of classrooms over time using random sampling to address these limitations.
Concluding Thoughts
The CCCO protocol represents an important first step toward systematically understanding community college classroom contexts serving diverse, ethnic minority, and immigrant-origin emerging adults. Given past research has linked differences in classroom engagement to myriad student academic outcomes for diverse adolescent populations (Fredricks et al., 2004; Marks, 2000; Suárez-Orozco et al., 2009; Teranishi et al., 2011), this work is an important first step toward systematically understanding community college contexts serving diverse emerging adult populations. The results of our analyses indicate there may indeed be important differences in classroom engagement dimensions across community college campuses. They also suggest the CCCO can predict selected student outcomes as we would expect based on previous theory and empirical literature, making it a potentially valid and reliable measure of community college classroom engagement. Future studies should aim to further pilot and empirically assess the validity and reliability of the CCCO measure in community colleges and other higher education settings on a larger scale.
Footnotes
Authors’ Note
The opinions expressed are those of the authors and do not represent the views of the W. T. Grant Foundation and the Ford Foundation.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The W. T. Grant Foundation and the Ford Foundation supported the research reported here.
Authors
STACEY ALICEA is senior director of Ramapo Training at Ramapo for Children and a postdoctoral fellow at the McSilver Institute for Poverty Policy and Research at New York University (NYU). Her work focuses on strength-based approaches to adolescent development in educational settings with the goal of applying this work to intervention and policy targeting social capital accrual and life skills development for youth.
CAROLA SUÁREZ-OROZCO is a professor of human development and psychology at the University of California, Los Angeles (UCLA). Her domain of research focuses on elucidating the child, adolescent, and young adult experience of immigration across a variety of contexts.
SUKHMANI SINGH is director of the Resilience Opportunity Safety Education Strength (ROSES) Program at the Metropolitan Center for Research on Equity and the Transformation of Schools and an adjunct professor at NYU Steinhardt. Broadly, she is interested in how marginalized youth negotiate systems of oppression, and her areas of interest include student (dis)engagement, immigrant youth, critical consciousness, and the school-to-prison pipeline.
TASHA DARBES is an assistant professor of TESOL (Teaching English to Speakers of Other Languages) and bilingual education at Pace University. Her scholarship examines areas where immigration, education, and language acquisition intersect, such as the assessment and identification of multilinguals, preparing teachers to work with diverse student populations, and school–community engagement with immigrant families.
ELVIRA JULIA ABRICA is an assistant professor of educational administration at the University of Nebraska-Lincoln. Her research focuses on students’ subjective meaning-making around race and immigrant generation in community college and STEM (science, technology, engineering, and math) contexts.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
