Measuring Afterschool Program Quality Using Setting-Level Observational Approaches

Supportive Relationships With Staff

A broad literature documents that supportive relationships with extra-familial adults help reduce problem behaviors and improve school-related outcomes (DuBois, Holloway, Valentine, & Cooper, 2002). Positive relationships with teachers have been conceptualized as a primary mechanism through which classroom experiences lead to positive youth outcomes (Pianta & Allen, 2009). Similarly, in afterschool research, supportive relationships with staff differentiate high-functioning afterschool programs from less effective ones (Eccles & Gootman, 2002; Grossman, Goldmith, Sheldon, & Arbreton, 2009). Specifically, supportive relationships with afterschool staff are related to increased learning and engagement in afterschool activities (Grossman et al., 2009), heightened sense of self-efficacy (Walker & Arbreton, 2004), fewer internalizing and externalizing problems (Pierce et al., 1999), and higher reading and math grades (Pierce et al., 2010).

Supportive Relationships With Peers

The accumulated body of research indicates that academic, psychological, social, and behavioral adjustment of children and youth is influenced by peer relations, whether measured as characteristics of peers or peer groups (Véronneau & Dishion, 2011), peer attachment, and peer support (Gonzales, Cauce, Friedman, & Mason, 1996) or friendship quality (Franco & Levitt, 1998). Furthermore, E. P. Smith, Faulk, and Sizer (2013) have shown that though supports from adult family members and the community are associated with reduced delinquency, positive peer interactions are more strongly related to positive youth development. Research evidence indicates that positive peer relations in afterschool programs are one of the primary mechanisms through which afterschool program participation affects a wide range of youth outcomes, including delinquent behavior (Gottfredson, Gerstenblith, Soulé, Womer, & Lu, 2004), internalizing and externalizing problems and social skills (Pierce et al., 1999), and sense of self-efficacy and response to social challenges (Walker & Arbreton, 2004).

Developmentally Appropriate Structure and Supervision

Provision of structured and supervised time after school is the core raison d’être of afterschool programs (Eccles & Gootman, 2002), and unstructured socializing with peers in the absence of adult supervision increases delinquency and problem behaviors among youth and adolescents (Osgood & Anderson, 2004). Mahoney and colleagues have found that participation in unstructured and less well-supervised programs increases rather than decreases youth involvement in antisocial behavior (Mahoney et al., 2004) while Gerstenblith and colleagues (2005) have shown that youth benefit from attending structured afterschool programs with organized transitions, planned activities, structured skill-oriented programming, and quality behavior management.

Youth Engagement

Opportunities for youth to make a difference or “matter” in their environments are associated with positive outcomes (Eccles & Gootman, 2002). McLaughlin, Irby, and Langman (1994) found that community-based programs with high levels of youth participation provided increased opportunities to build skills, competencies, and self-worth. Highly engaging activities offer opportunities that promote achievement, skill-building, and intrinsic motivation (Larson, 2000). Furthermore, youth involved in interesting programs are less likely to discontinue participation in the program (Perkins & Borden, 2003). Higher levels of engagement in afterschool programs are linked with positive youth outcomes, including increased social competence (Shernoff, 2010), improved intrinsic motivation (Mahoney et al., 2007), and decreased school suspension rates (Blazevski & Smith, 2007).

Participants

This study is part of LEGACY Together, ² which is a randomized controlled trial study of a behavioral intervention in afterschool settings. The project involved 1,200 children and staff and 75 afterschool programs in Pennsylvania. Data were collected from 3 cohorts over a 3-year period (2009-2012) using data from various sources (children, staff, director, school teachers) and methods (direct observation, self-report surveys, documents).

The present study used data drawn from the observation of 44 afterschool programs that participated in the first 2 years of the larger program evaluation project. The programs were clustered in a limited set of geographic locations in central or eastern Pennsylvania, and inclusion required agreement to participate in a random trial of a program to enhance effectiveness. These afterschool programs were diverse in mission, locale, and demographic composition, which is a valuable strength for a measurement study. All the programs were run by community-based organizations and housed in school buildings. The programs included two 21st-Century Community Learning Centers, 23 programs operated by Young Men’s Christian Associations (YMCAs) in various communities, and 19 programs operated by private providers, with many of the latter participating in the Keystone STARS program—a quality rating system for improving child care settings in Pennsylvania. The primary activities of all the programs were snacks, homework, and recreation. Nine of the programs put greater emphasis on structured learning activities than did the others, but none was predominantly academic. The programs typically served youth from the end of the school day until 5:30 or 6:00 p.m. All operated 5 days per school week, with most (but far from all) students attending almost every day.

The demographic characteristics of the program participants appear in Table 1. The locales of the afterschool sites were 11% rural, 57% suburban, and 32% urban. In terms of racial/ethnic composition, 36% were predominantly White (minority enrollment ≤ 25%) and 27% of the programs had a high concentration of minority children (minority enrollment > 75%). The programs were also diverse in socioeconomic composition: 25% of the programs had disproportionately high percentages of low-income children with over 75% eligible for free or reduced-price meals while another 25% were economically advantaged with less than 25% eligible. These afterschool programs were small to moderate in size: The number of enrolled children ranged from 11 to 50 ( X ¯ = 24.24, SD = 10.67) and the number of staff ranged from 1 to 8 ( X ¯ = 3.75, SD = 1.75). All the programs enrolled children between the kindergarten and seventh grade ( X ¯ = 2.5, SD = 1.70).

OPEN IN VIEWER

Table 1.

Demographic Characteristics of the Programs (N = 44).

Variable	Percent or X ¯ (SD)
Locale
Rural	11.4%
Suburban	56.8%
Urban	31.8%
Minority enrollment
Low (≤ 25%)	36.4%
Low-medium (26%-50%)	11.4%
Medium-high (51%-75%)	18.2%
High (>75%)	27.3%
Percent free or reduced lunch eligible students
Low (≤25%)	25.0%
Low-medium (26%-50%)	29.5%
Medium-high (51%-75%)	13.6%
High (>75%)	25.0%
Lowest grade in attendance
Kindergarten	64.3%
First grade	21.4%
Second grade	9.5%
Highest grade in attendance
Fourth grade	2.4%
Fifth grade	76.2%
Sixth grade	11.9%
Seventh grade	4.8%
Grade of enrolled students	2.50 (1.70)
Total number of children enrolled	24.24 (10.67)
Total number of staff	3.75 (1.75)

Direct Observation Tools

Caregiver Interaction Scale

Developed by Arnett (1989), the original Caregiver Interaction Scale (CIS) consisted of 26 items examining the interactions of caregiving staff with preschool children in the following 4 areas: (a) harshness and more authoritarian manner, expecting children to rigidly conform to adult rules and behavior; (b) sensitivity, which is defined as not only communication with young people but the provision of appropriate levels of guidance, redirection, praise, and contingent rewards; (c) detachment, uninterested in children and involved in more adult-oriented activities that exclude interaction with children; and (d) permissiveness, the degree to which staff fail to appropriately provide guidance and redirection when necessary. Based on our pilot work, we adapted some of the items to make them more observable and as unambiguous as possible, resulting in 23 items (see Table 2 for the full list of items). Trained field observers rated up to 3 permanent, non-volunteer staff ³ in each afterschool program on a 4-point response scale indicating the extent to which they engage in a particular behavior or practice: 1 = never (0%); 2 = few instances (1%~30%); 3 = many instances (31%~60%); and 4 = consistently (>61%).

OPEN IN VIEWER

Table 2.

Sample Descriptive Statistics and CFA Result for CIS (N = 722).

			CFA result
Item	X ¯	SD	Factor loading	Item R2
Sensitivity (α = .90)
1. Speaks warmly to children	3.03	0.86	0.86	.74
2. Listens attentively to children	3.02	0.85	0.84	.71
3. Excited about teaching	2.76	0.90	0.85	.72
4. Encourages children to try different experiences	1.86	0.91	0.55	.30
5. Enthusiastic about children’s activities/efforts	2.73	0.92	0.87	.76
6. Encourages children to exhibit prosocial behavior	2.32	0.97	0.66	.44
7. Positive physical contact	1.83	0.87	0.48	.23
8. Sincere in tone of voice and manner	3.24	0.80	0.76	.58
Harshness (α = .79)
9. Seems critical of children	1.16	0.46	0.79	.62
10. Places high value on obedience-directive	1.68	0.87	0.51	.26
11. Speaks with irritation or hostility to children	1.26	0.54	0.77	.59
12. Negative physical contact	1.01	0.11	0.17	.03
13. Punishes children w/o explanation or other redirection	1.10	0.36	0.58	.34
14. Finds fault easily with children	1.12	0.42	0.86	.74
15. Seems to prohibit many things children want to do	1.30	0.54	0.61	.37
Detachment (α = .79)
16. Seems emotionally distant or detached from children	1.74	0.87	0.72	.52
17. Spends considerable time not involving children	1.83	0.86	0.58	.34
18. Seems interested in children’s activities a	2.06	0.89	0.86	.74
19. Supervises children closely a	2.09	0.90	0.62	.38
Permissiveness (α = .80)
20. Exercises little control over children	1.55	0.72	0.29	.08
21. Reprimands children when they misbehave a	2.35	1.05	0.85	.72
22. Exercises firmness when necessary a	2.31	1.01	0.93	.86
23. Expects children to exercise self-control a	2.09	0.91	0.74	.55

Note. CIS = Caregiver Interaction Scales; CFA = confirmatory factor analyses.

a

Reverse-coded items.

Promising Practices Rating Scale

The Promising Practices Rating Scale (PPRS) is an observational tool developed by Vandell and colleagues (2004) to study the relationship between afterschool program quality and youth outcomes. The PPRS collects information on types of activities offered in afterschool programs, resources available for those activities, and various dimensions of program quality and practices during those activities. These dimensions included supportive relations with adults, supportive relations with peers, level of engagement, opportunities for cognitive growth, appropriate program structure, over-control, chaos (a problematic manifestation of inadequate structure), and mastery orientation.

We modified the original version of the PPRS in several ways. In the original PPRS, each dimension of program quality was assessed by a single global item with a 4-point scale indicating the extent to which a given construct is characteristic of the program (1 = highly uncharacteristic, 2 = somewhat uncharacteristic, 3 = somewhat characteristic, and 4 = highly characteristic). Single-item measures, however, may have limited reliability (Nunnally & Bernstein, 1994). In order to address this concern, we developed multi-item subscales representing each dimension of program quality by treating specific exemplars for each dimension provided in the PPRS as separate items that could serve as subscale components. If the PPRS manual included more than four exemplars, then four were chosen to represent as much as possible the full range of content in the entire set. ⁴ The ratings on the set of items for each dimension were then averaged to obtain a subscale score.

Next, we modified the standard PPRS observational protocol, which calls for the observation of six successive activities for 15-minute intervals (Vandell et al., 2004). Our data collectors instead observed three successive activities during each day of observation for an afterschool program. Doing so reduced the required time, which was necessary for gathering other data. Compensating for this reduced sample of activities at each visit, we observed each program twice over a period of 4 to 6 weeks. This provided an equal amount of observing time while also enabling us to determine the contribution of day-to-day variation to measurement error.

Another modification was to omit three of the PPRS subscales, Over-Control, Mastery Orientation, and Opportunities for Cognitive Growth, because the program being evaluated did not seek to affect these dimensions. Our implementation of the PPRS scale therefore consisted of 17 items, with the following five subscales: Supportive Relations With Adults, Supportive Relations With Peers, Level of Engagement, Appropriate Structure, and Chaos (see Table 3 for the list of items).

OPEN IN VIEWER

Table 3.

Sample Descriptive Statistics and CFA Result for PPRS (N = 1,092).

			CFA result
Item	X ¯	SD	Factor loading	Item R2
SRA (α = .86)
1. Staff communicates clear expectations/guidelines	2.92	0.77	0.61	.37
2. Positive behavior management techniques	2.94	0.82	0.70	.49
3. Staff positive affect with students	3.02	0.77	0.82	.68
4. Positive personal interactions with students	2.97	0.8	0.77	.59
5. Positive engagement with students/activities	3.01	0.81	0.83	.69
SRP (α = .89)
6. Positive peer interactions	3.18	0.69	0.86	.73
7. Students work well together	3.21	0.70	0.90	.82
8. Inclusive student atmosphere	3.18	0.73	0.81	.65
AS (α = .76)
9. Well-managed activities	3.24	0.76	0.87	.76
10. Clear and appropriate activity instructions	3.08	0.76	0.69	.47
11. Staff cooperation and consistency	3.21	0.76	0.54	.29
12. Safe student work environment	3.63	0.63	0.52	.27
LE (α = .85)
13. Students positively engaged	3.24	0.71	0.87	.76
14. Students on-task during activity	3.17	0.72	0.89	.78
15. Students actively participating in on-task discussion	2.97	0.83	0.70	.48
Chaos (α = .83)
16. Inappropriate/disruptive student behavior	1.53	0.70	0.82	.67
17. Ineffective student management skills/abilities by staff	1.48	0.71	0.87	.75

Note. PPRS = Promising Practices Rating Scales; CFA = confirmatory factor analyses; SRS = Supportive Relations With Adults; SRP = Supportive Relations With Peers; AS = Appropriate Structure; LE = Level of Engagement.

Data Collection Procedure

Direct observation is a complex and costly approach to setting-level measurement, but likely provides the least biased assessment of social ecological settings. As discussed above, quality observation-based setting measures require quality observation systems. We sought to enhance reliability through a carefully designed observation system with clear operational definitions, trained observers, and well-established systems of data recording.

To assess the quality of afterschool settings, observational data were collected 5 times across the academic year—twice in the fall with 4 to 6 weeks between observations, once in January to February, and twice in the spring in 4- to 6-week intervals. Approximately half of the observations in fall, winter, and spring were conducted with two observers per site. Given the variability of activities and staff in afterschool we thought that multiple measurements on varying days with varying observers would be necessary to characterize the nature of these afterschool programs accurately (Raudenbush, Martinez, Bloom, Zhu, & Lin, 2011). Observers were instructed that on each visit they should observe the full range of activities offered that day and as broad a sample of the staff and students as possible. The mean number of observers per site visit was 1.7. On average, 1.97 staff members and 2.95 activities were rated for each combination of site, occasion, and observer. In total, 722 ratings were obtained for each CIS item and 1,092 ratings for each PPRS item, which served as the analytic sample for this study. A total of 14 observers collected the data reported here. Observers for each site varied across rounds.

Observers participated in an initial training consisting of two 8-hour sessions that included an overview of the constructs and measures, showing videos illustrating the concepts, and scoring practices and discussion. Training was conducted by project research staff and the university survey research center that managed the data collection effort. The training also included live-practice in afterschool programs. Two additional booster trainings occurred before the winter and spring data collection periods. In the first year, the data manager conducted a portion of the observations with the data collectors so that observer scores could be verified with a “Standard Coder.” Beginning with the second year, the data collectors were required to attend a refresher training for each wave of data collection and pass a reliability test on which they had to attain at least 80% agreement (within one scale point) with scores agreed on by research team members, known as our Gold Standard Video (GSV) process (Stuhlman, Hamre, Downer, & Pianta, 2010). The GSV scores were established by consensus of the research team after viewing and discussing the videos and establishing agreement of at least 80% and often 90% within the research team of 3 to 4 members. Data collectors scoring beneath 80% on any scale received a call from the GSV coordinator, who would inquire about why certain scores were given and discuss the GSV score and the rationale. This process was completed before the deployment of observers to the field. This served as a way to prevent drift and promote the reliability and accuracy of observer ratings (Stuhlman et al., 2010). The data collectors used mini-laptop computers to record the observations in real time and then transmitted the data to the project offices.

Descriptive Data and Internal Consistency Reliability

The scales and items were evaluated in terms of descriptive statistics and internal consistency (Cronbach’s α). Here, alpha reflects the coherence of an observer’s ratings, for a single segment (i.e., staff or activity) in a single program at a single time, across the items assessing that concept (N = 722 for CIS; N = 1,092 for PPRS). We used the conventional criteria of .70 for an acceptable level of coefficient alphas (Nunnally & Bernstein, 1994).

Inter-Rater Reliability

The level of agreement between observers is a crucial aspect of the quality of our measures. Inter-rater reliability was assessed with intra-class correlation coefficients (ICCs) reflecting the percent of variance shared across observers rating the same program at the same time, rather than unique to one observer. The reliability associated with the ICC values depends on the number of observers used in any study, with more observers yielding more reliability assessments. We therefore illustrate the implications of ICCs by reporting reliabilities computed both for a single observer and for the average number of observers in our data (1.7; McGraw & Wong, 1996).

The results of the analyses of inter-rater reliability for the CIS, the PPRS, and their subscales appear in Table 4. The reliability for a measure that is the average of two raters was .77 for overall measures of both CIS and PPRS. The analyses of subscales yielded average inter-rater reliabilities (for an average of two raters) ranging from .59 to .75 for the CIS subscales and from .56 to .69 for the PPRS subscales. According to criteria proposed by Fleiss (1981; i.e., <.40, poor; .40-.59, fair; .60-.74, good; >.74, excellent), inter-rater reliability values for a majority of our scales and subscales were good or above. Note that the ICC values imply that measures based on observations by a single rater would have less satisfactory reliability.

OPEN IN VIEWER

Table 4.

Inter-Rater Reliability.

		ICC reliability
	Number of items	Single rater		Average of two raters
Scale		(95% CI)		(95% CI)
CIS	23	0.63	[0.61, 0.64]	0.77	[0.76, 0.78]
Sensitivity	8	0.59	[0.57, 0.62]	0.75	[0.72, 0.76]
Harshness	7	0.42	[0.38, 0.46]	0.59	[0.54, 0.63]
Detachment	4	0.55	[0.50, 0.59]	0.71	[0.67, 0.74]
Permissiveness	4	0.45	[0.41, 0.50]	0.63	[0.58, 0.67]
PPRS	17	0.62	[0.61, 0.64]	0.77	[0.76, 0.78]
SRA	5	0.45	[0.42, 0.49]	0.62	[0.59, 0.65]
SRP	3	0.39	[0.34, 0.44]	0.56	[0.51, 0.61]
AS	4	0.52	[0.48, 0.56]	0.69	[0.65, 0.72]
LE	3	0.42	[0.38, 0.47]	0.59	[0.55, 0.64]
Chaos	2	0.51	[0.45, 0.56]	0.67	[0.62, 0.72]

Note. ICC = intra-class correlation coefficients; CIS = Caregiver Interaction Scales; PPRS = Promising Practices Rating Scales; SRA = Supportive Relationship With Adults; SRP = Supportive Relationship With Peers; AS = Appropriate Structure; LE = Level of Engagement.

Construct Validity: Confirmatory and Exploratory Factor Analysis

We tested the consistency of our data with the hypothesized factor structures of the CIS and PPRS through CFA using AMOS 20 software (Arbuckle, 2011). We used a bootstrapping procedure because several items in both instruments were distributed non-normally (Brown, 2006). The results of the CFAs indicated little discrepancy between the bootstrap estimates and their original Maximum Likelihood (ML) counterparts, so we report only the results of the standard ML estimation based on the normality assumption. There was a fairly high rate of missing cases for the PPRS Item 11, “staff cooperation and consistency” (25.7%), because the item was “not applicable” to afterschool sites with only one staff member present at the time of observation, which we addressed through the multiple imputation procedures of AMOS, using five imputed data sets.

We evaluated the adequacy of fit for each model using indices commonly recommended in the literature including Tucker-Lewis Index (TLI), comparative fit index (CFI), standardized root mean square residual (SRMR), and root mean square error of approximation (RMSEA; Brown, 2006). In the present study, we adopted cut-off values of >.90 for TLI and CFI and <.08 for SRMR and RMSEA (Bentler, 1990; Browne & Cudeck, 1992). In case the result of the CFA did not support the a priori factor models, we conducted an exploratory factor analysis (EFA). The EFA analyses used a random split half of the full sample to identify a factor model, the fit of which we then tested through a CFA using the other half of the sample.

In addition, we were interested in determining whether the measurement models held across different types of afterschool programs. We split the sample programs into two groups based on the socio-demographic compositions of student body, using the criterion of economic disadvantage. About 40% of the afterschool programs in the sample (17 of 44) were classified as economically disadvantaged due to more than 50% of the student population being eligible for free or reduced-price meals. These programs also served predominantly ethnic minority students (16 of 17 for the disadvantaged group versus 4 of 27 for the advantaged group) and urban populations (13 of 17 for the disadvantaged group versus 1 of 27 for the advantaged group). Thus, the contrast between these sets of programs provides an appropriate test of whether previous examinations of the reliability and validity of the PPRS generalize beyond previous assessments for programs serving disadvantaged, urban, minority populations.

We performed a series of nested multiple group CFA to test the equivalence of factor pattern, factor loadings, factor intercorrelations, and residual variance across afterschool programs serving more disadvantaged students and relatively advantaged students. We began with a baseline model with no constraints across the two sets of programs except that the same factor structure was applicable to both groups (Model 1), and we subsequently tested three increasingly restrictive models. Model 2 constrained factor loadings to be equal, Model 3 added the constraint of invariant factor variance/covariance, and Model 4 included the constraints from Model 3 plus the restriction of equal error variances across the two groups (Vandenberg & Lance, 2000). To compare the relative fit of the nested models, we relied on changes in CFI index, following Cheung and Rensvold’s (2002) guideline of 0.01 or smaller as support for invariance.

Caregiver Interaction Scale

Table 2 shows the standardized factor loadings obtained from the CFA of the a priori four-factor model from Arnett (1989); that is, sensitivity, harshness, detachment, and permissiveness. Items 12 and 20 had low factor loadings of 0.17 and 0.29, respectively, which indicates that these items do not significantly contribute to their hypothesized factor. The poor performance of item 12 is understandable, in that observers almost never reported negative physical contact ( X ¯ = 1.01 on a 1-4 metric). In addition, the correlation between the Sensitivity and Detachment latent variables was −.97, indicating a lack of discriminant validity of these two dimensions. The model also yielded less than adequate fit indices; specifically χ²(224, 722) = 1562.13 (p < .001), TLI = 0.843, CFI = 0.861, RMSEA = 0.091(90% CI = [0.087, 0.095]), and SRMR = 0.082.

As the hypothesized four-factor model was not supported by our data, we performed an EFA on the first half (n = 364) of the randomly split data set (see Online Appendix A, Table A1, for descriptive statistics) using principal axis factoring with promax oblique rotation. The eigenvalues for the first four factors obtained were 5.92, 2.08, 0.90, and 0.59. We retained the third factor because it proved to exceed Kaiser’s criterion (i.e., eigenvalues > 1) after rotation. The total variance accounted for by the 3 factors was 49.85%. Both pattern and structure matrices (Online Appendix A, Table A2) showed a relatively clear three-factor solution: Factor 1 consists of a combination of the original model’s Sensitivity and Detachment subscales (CIS-SD), Factor 2 represents the Harshness subscale (CIS-H), and Factor 3 corresponds to the Permissiveness subscale (CIS-P). Most items loaded strongly on their expected factors, but there were some exceptions with weak or mixed loadings. Items 12 and 20 did not strongly load on any factors, Item 6 had a substantial and inappropriate secondary loading, and Item 19 loaded highest on a factor other than the intended one.

We re-ran the EFA after removing those four items with problematic loadings. For this EFA, three factors accounted for 53.77% of the total variance (the first factor 33.41%, the second 12.11%, and the third 8.25%). All 19 items showed unique factor loadings of more than 0.4 on their intended factors (Online Appendix A, Table A3). We then replicated this EFA solution via CFA with the second half of the sample (n = 358). The fit results of the CFA for this three-factor model with 19 items indicated an adequate fit to the data—specifically, χ²(149, 722) = 469.31 (p < .001), TLI = 0.914, CFI = 0.925, RMSEA = 0.078 (90% CI = [0.07, 0.086]), and SRMR = 0.061. The EFA results indicate that the remaining modest lack of fit stems from secondary cross-loadings of items on other factors, which do not appear problematic in relation to much stronger primary loadings. As displayed in Table 5, all the standardized factor loading for the 19 items were above the conventional cut-off value of 0.5, and 13 items had values of 0.7 or higher. The refined subscales yielded very favorable Cronbach’s alphas of .92, .83, and .87 for CIS-SD, CIS-H, and CIS-P, respectively. The correlation statistics indicated that the CIS-SD subscale had moderate correlations with the CIS-H (−.36) and the CIS-P (−.53), but the CIS-H and the CIS-P showed minimal intercorrelation (−.05).

OPEN IN VIEWER

Table 5.

CIS Item Fit to the Revised 3-Factor Model (N = 358).

Dimension/item	Path coefficient	Item R2	α
Sensitivity-detachment			.92
Speaks warmly to children	0.88	.78
Listens attentively to children	0.86	.74
Excited about teaching	0.86	.74
Seems emotionally distant or detached from children	−0.69	.47
Encourages children to try different experiences	0.58	.34
Enthusiastic about children’s activities/efforts	0.86	.74
Considerable time in activity not involving children	−0.52	.28
Positive physical contact	0.50	.25
Seems interested in children’s activities a	−0.85	.73
Sincere in tone of voice and manner	0.78	.60
Harshness			.83
Seems critical of children	0.75	.56
Places high value on obedience-directive	0.59	.35
Speaks with irritation or hostility to children	0.73	.54
Punishes children w/o explanation or other redirection	0.61	.38
Finds fault easily with children	0.88	.78
Seems to prohibit many things children want to do	0.70	.49
Permissiveness			.87
Appropriately reprimands children when they misbehave a	0.86	.74
Exercises firmness when necessary a	0.94	.89
Expects children to exercise self-control a	0.70	.49

Note. Reliability analysis was performed using the full sample (N = 722).

a

Reverse-coded items.

The results of multiple group CFAs indicate that the factor configuration is invariant across the two types of afterschool programs. The baseline model showed an adequate fit with CFI = 0.923, RMSEA = 0.054 (90% CI = [0.05, 0.058]), and SRMR = 0.059. The incremental changes in CFI between the two nested models were smaller than 0.01 for all levels of invariance, providing support for invariance in factor loadings, factor variance/covariance, and error variance across the two groups (see Online Appendix A, Table A4, for full results).

Generalizability Theory–Based Reliability

Given that our afterschool program quality measures derive from a complex measurement strategy involving multiple observers and multiple observational visits, we applied generalizability (G) theory in addition to those more traditional analyses based on classical test theory. While classical test theory treats error variance as undifferentiated random variation, G theory partitions error variance into multiple sources (Raudenbush et al., 2011; Shavelson & Webb, 1991). Doing so enables us to assess the relative contributions to the reliability of measurement of each of the different sources of error and their combinations, versus the genuine variation between programs. This information, in turn, informs decisions about how the quality and efficiency of behavioral observations can be maximized through measurement design.

In the present study, our concern is estimating qualities of afterschool programs assessed through multiple observations, each of which is a joint product of the judgment of the observer and the specific events that occurred at that time. Each observer would make somewhat different judgments, and the events to be observed will differ from day to day. Thus, in research on the quality of afterschool programs, we are interested in determining the impact of observers, occasions (which we refer to as rounds of assessment), and their combinations on the reliability of our observational measures. ⁵ More specifically, our measurement design allowed us to estimate five separate sources of variance: (a) between-site variance, which corresponds to true score variance in classical test theory given that site is the target of measurement; (b) observer variance, reflecting observer-to-observer differences in their average ratings of afterschool program quality; (c) variance across rounds, nested within site, which represents the changes in the ratings of the quality of any program from one round to another; (d) site-by-observer interaction, indicating the extent to which between-site differences in the ratings varied across observers; and (e) error variance confounded with the interaction between observer and round nested within site, which reflects the extent to which between-observer differences varied across rounds within sites.

We used repeated-measures analysis of variance framework to conduct a G-study of these sources of variation. The resulting variance component estimates were then used to calculate reliability coefficients under the given measurement condition and also with hypothetical variation in numbers of observers and rounds. The latter is referred to as a decision study or D-study. We calculated two types of reliability coefficients: generalizability coefficient for relative decisions and dependability index for absolute decisions (Shavelson & Webb, 1991). Online Appendix B provides greater detail on generalizability theory-based methods.

As mentioned earlier, each of the afterschool programs in our study was visited 5 times for site observations. Among the five rounds of data collection, the first two rounds were intended to establish pre-intervention characteristics of the afterschool program while the last two rounds were used to obtain stable estimates of post-intervention program characteristics. To conduct a generalizability study independent from the influences of the intervention, we stacked pre- and post-intervention data after assigning new site IDs to post-intervention data, thereby treating the pre- and post-data as though they represented different sites. The data from the interim round (third round) were not included in the current analysis. As a result, the analysis includes a total of 276 data points (44 × 2 sites, 2 rounds, 1-2 observers).

Results of the G-study

Table 6 presents the variance component estimates for each of three CSI and five PPRS scales as well as the reliability coefficients obtained for our research design. The amount of variance attributable to each component varied across scales. The “true score” variance of interest, site ( σ s 2 ), accounted for 11% to 43% of total variance across the CIS and PPRS scales. Variance across observers’ mean ratings ( σ o 2 ) was smaller than the other components, accounting for 1% to 18% of variance. In contrast, a relatively large percentage of variance was attributable to day-to-day variation in program functioning, as reflected in round nested within site ( σ r : s 2 ), ranging from 14% to 39%. The site-by-observer interaction term ( σ s o 2 ) contributed almost negligible variance to the model for some scales (0%-3.7%), but explained a modest proportion of variance for others (12%-18%). The last component, observer-by-round interaction plus error ( σ o ( r : s ) , e 2 ), explained 23% to 38% of total variance.

OPEN IN VIEWER

Table 6.

Variance Components Analysis Results (N = 276).

			Variance as a percent of total variance					Reliability coefficients
Subscale	X ¯	Variance	Site ( σ s 2 )	Observer ( σ o 2 )	Round: Site ( σ r : s 2 )	Site × Observer ( σ s o 2 )	Observer × Round: Site + Residuals ( σ o ( r : s ) , e 2 )	D- index	G-coefficient
CIS-SD	2.75	0.34	35.8	9.6	22.9	0.0	28.7	.63	.68
CIS-H	1.30	0.14	42.8	6.0	20.4	11.9	22.8	.67	.69
CIS-P	2.16	0.51	13.1	17.6	13.6	17.7	35.8	.33	.39
PPRS-SRA	2.95	0.29	34.5	4.4	26.7	0.0	31.5	.62	.64
PPRS-SRP	3.17	0.27	17.0	11.7	31.5	3.7	30.0	.38	.42
PPRS-AS	3.27	0.23	23.9	1.1	21.1	14.6	37.8	.50	.51
PPRS-LE	3.11	0.28	10.6	4.5	39.0	13.1	27.8	.25	.26
PPRS-C	1.53	0.28	32.4	4.5	25.2	0.8	35.3	.60	.62

Note. CIS-SD = Caregiver Interaction Scales–Sensitivity and Detachment; H = Harshness; P = Permissiveness; PPRS = Promising Practices Rating Scales; SRA = Supportive Relationship With Adults; SRP = Supportive Relationship With Peers; AS = Appropriate Structure; LE = Level of Engagement; C = Chaos.

Table 6 also reports the two types of reliability coefficients calculated, given the number of rounds and observers for the current research design. Reliability coefficients varied across the scales, with generalizability coefficients ranging from .26 to .69 and dependability indices ranging from .25 to .67. There was little difference between generalizability coefficients and dependability indices because the variance attributable to observer was small relative to the other components. Detailed variance component results specific to each scale are presented below.

For CIS-SD, CIS-H, and PPRS-SRA, the largest proportion of observed score variance was accounted for by site (35%-43%), indicating that these afterschool programs differed systematically in terms of staff’s sensitivity-detachment and harshness and overall relationship between staff and youth. The second largest proportion of variance was attributed to the observer-by-round interaction plus unexplained variance (23%-32%). The next largest proportion of variance was explained by round nested within site (20%-27%), indicating that there were fairly substantial changes in the ratings of program quality from round to round within sites. The variances due to observer and site-by-observer interaction were relatively small, revealing that the relative standing of afterschool programs was constant across observers. The small proportion of variance attributable to observer (4%-10%) also indicates that there were minimal differences in the mean ratings of these three scales across observers. Reliability coefficients were relatively high for these scales. Under the given measurement condition, generalizability coefficients were .64, .68, and .69, and dependability indices were .62, .63, and .67, for PPRS-SRA, CIS-SD, and CIS-H, respectively.

For PPRS-C and PPRS-AS, a substantial amount of variance was explained by site (32% and 24%), but the largest amount of variance was attributable to the observer-by-round interaction confounded with residual error (35% and 38%). Another large source of error was the variance due to round-to-round differences within a site (21% and 25%). The site-by-observer variance was negligible for PPRS-C (0.8%), but fairly substantial for PPRS-AS (15%). The proportion of variance attributable to observer was small for both scales, 1% for PPRS-AS and 5% for PPRS-C. Given the conditions of our measurement procedure, both generalizability coefficients and dependability indices were about .6 for PPRS-C and around .5 for PPRS-AS.

The last three scales, CIS-P, PPRS-SRP, and PPRS-LE, consisted of a relatively small percentage of variance attributable to sites (13%, 17%, and 11%, respectively) and thus a relatively larger percentage of variance due to the error components. This led to low reliability coefficients, with generalizability coefficients of .26 (PPRS-LE), .39 (CIS-P), and .42 (PPRS-SRP), and dependability indices of .25 (PPRS-LE), .33 (CIS-P), and .38 (PPRS-SRP). In particular, for PPRS-SRP and PPRS-LE, the between-round variance and observer-by-round interaction plus error together explained roughly two thirds of the observed score variance. For CIS-P, the between-observer variance and site-by-observer variance together accounted for 35% of the total variance, much more than the other scales.

Results of the D-study

Using the variance component estimates obtained from the generalizability study, we calculated dependability indices for different numbers of observers (1-10) and different numbers of rounds within a site (1-4). Figure 1 shows how dependability indices will change across different combinations of numbers of observers and rounds for each of eight observation scales. It is evident that a single round of observation would not produce an adequate level of reliability. Figure 1 also shows that increasing the number of observers and/or the number of rounds raises the dependability index, but that returns diminish as the number of either increases beyond a few. For example, the estimated dependability index of the CIS-H scores is about 0.80 for a 4-observer and 4-round design and around 0.85 for a 10-observer and 4-round design.

OPEN IN VIEWER

Figure 1.

Reliability coefficients across observers and rounds for the CIS and PPRS scales.

However, the extent of gains achieved by adding the number of rounds or observers varies somewhat across scales, reflecting which error variance components are largest for each of them. Particularly notable is that the CIS-P Scale benefits more by increasing the number of observers than the number of rounds while the opposite is generally the case for other scales. Figure 2 illustrates that the addition of observers and rounds has substantially different impacts on CIS-P and PPRS-LE, for example.

OPEN IN VIEWER

Figure 2.

Reliability coefficients for different combinations of observers and rounds for CIS-P and PPRS-LE.