Additional Validity Evidence and Across-Group Equivalency of the HOPE Teacher Rating Scale

Underrepresented and Underperforming Students

Students underrepresented in programs for the gifted and talented typically include those from African American, Native American, Hispanic, and low-income families (Wyner, Bridgeland, & Diiulio, 2009; Yoon & Gentry, 2009). Additionally, Plucker, Burroughs, and Song (2010) reported on a widening excellence gap that exists between represented and underrepresented students at the highest levels of achievement. Although underrepresentation has been a persistent problem in the field of gifted education for decades, and some Native American groups have made modest progress toward more-proportional representation, low-income, African American, and Hispanic students have seen few gains (Donovan & Cross, 2002; Yoon & Gentry, 2009). Caucasian and Asian American students continue to be overrepresented, and low-income students have only recently received attention concerning their underrepresentation. Despite comprising 42% of the student population in the United States (as measured by free and reduced lunch eligibility), students from low-income families make up only 28% of top-achieving students in first grade, and they tend to fall from the highest level of achievement as they progress through school (National Center for Educational Statistics, n.d.-a; Wyner et al., 2009). At the same time, the number of K-12 students from traditionally underrepresented populations has continued to increase in the general K-12 population during the past decade, with the most recent data indicating that 40% of K-12 students come from minority populations (National Center for Educational Statistics, n.d.-b). With such large proportions of students from traditionally underrepresented populations, methods and procedures to more accurately identify gifted students from these groups are needed.

A variety of factors have been proposed as roots of the current underrepresentation problem. In addition to potential cultural bias in some assessment instruments, personal bias or inexperience on the part of educators (Briggs, Reis, & Sullivan, 2008) are factors that can prevent high-ability students who are also from low-income or culturally diverse families from being identified for gifted and talented programs. This bias is unlikely due to purposeful discrimination on the part of the teacher, but is more likely due to teachers who lack in-depth understanding of diverse cultures or who have different societal expectations or values than do their students (Peterson, 1999; Peterson & Margolin, 1997). Despite this particular source of bias, teachers often serve as gatekeepers to gifted and talented programs exacerbating the problem of underrepresentation (Peterson, 1999; Siegle & Powell, 2004). Finally, gifted and talented students are identified based on qualities valued in mainstream, Caucasian culture rather than by using attributes and strengths valued by diverse cultural groups (Oakland & Rossen, 2005; Peterson & Margolin, 1997). Importantly, much of this bias and inaccuracy cited in the literature relates to students who were simply nominated by a teacher as gifted and not by teachers using a structured rating system such as a teacher rating scale (e.g., Pegnato & Birch, 1959). In such systems the rater is provided with specific characteristics with which to evaluate his or her students. This can limit the degree to which raters rely on their own preconceptions or values (Peterson, 1999) but instead focuses ratings on typical gifted and talented behaviors.

It is important to note that the problem of underrepresentation is multifaceted and is likely caused by a number of societal and value factors far beyond the scope of measurement issues or assessment. However, despite a variety of causes, the fact remains that many students who come from low-income or culturally diverse families are overlooked when placement decisions are made even when they are already high achieving (Plucker et al., 2010). Such occurrences are not only contrary to the current federal definition of giftedness and talent (U.S. Department of Education, Office of Educational Research and Improvement, 1993) but also stand in opposition to the idea of equal opportunities for all students.

In one of only a few studies to research specific procedures for addressing underrepresentation of certain cultural, racial, and income groups, Briggs et al. (2008) qualitatively investigated seven gifted and talented programs to determine what practices consistently contributed to the success of culturally, linguistically, and ethnically diverse students. Questionnaires, document review, in-depth interviews, and observations were used to create detailed case studies of each program. One of the five themes identified by Briggs et al. was modified identification procedures, which included early identification as well as the use of multiple pathways for students to demonstrate talent in a variety of content areas. Worrell (2009) emphasized that using multiple pathways as a general best practice in the field.

Additional studies have shown that early and ongoing identification as well as the use of teacher-rating scales may be helpful in locating gifted and talented students from low-income families (Stambaugh, 2007). VanTassel-Baska (2008) also recommended that teacher-rating scales be used in an initial screening process to locate as many underrepresented students as possible who might not be identified using traditional means. This argument by VanTassel-Baska suggested that information provided by a student’s teacher regarding gifted and talented behaviors is in some way different from that information gained by more traditional forms of identification such as standardized tests. Thus, teacher ratings can serve as an additional pathway to identify students who would undoubtedly be overlooked in a system that based identification solely on high test scores. This exact practice was evaluated by Peters and Gentry (2012) and was found to differentiate among identified students in order to better target programming and to increase the overall number of low-income students identified. As noted by Wyner et al. (2009), children from low-income families frequently score below the traditional “cutoff” scores often used to identify children for gifted services. In fact, they suggested that a 75th percentile score by a child from a low-income family is equivalent to a 95th percentile score of a student not from a low-income family. Teacher ratings can provide an important pathway to allow access to gifted programs for underrepresented students who would be simply overlooked if test scores were the only indicator used for identification.

Teacher Rating Scales

Teacher rating instruments are not a new source of information in gifted and talented identification. One of the earliest examples of such forms was the Scales for Rating the Behavioral Characteristics of Superior Students (SRBCSS; Renzulli, Smith, White, Callahan, & Hartman, 1976). Although some research did support the general five-factor structure of the instrument (Burke, Haworth, & Ware, 2001), independent reviews of the original SRBCSS noted a general lack of validity information and a need for more empirical evidence (Rust, 1985). In 1997, the SRBCSS was revised during which process the authors made corrections and modifications (Renzulli et al., 2002). Unfortunately, the current version is still very long, with 96 items on 10 subscales. The authors have recently proposed four additional subscales, making the SRBCSS even longer (Renzulli, Siegle, Reis, Gavin, & Reed, 2009), although not all subscales need always be used. Unfortunately, important statistical information is still lacking on the SRBCSS, including subscale reliability information. Finally, for such a long-standing instrument, complete internal validity studies including the evaluation of multigroup validity evidence do not exist. What research has been conducted has been done on small, nonrepresentative samples of students.

Two additional teacher-rating instruments exist that have at least some level of empirical support. These include the Gifted Rating Scales (GRS; Pfeiffer & Jarosewich, 2003) and the Scales for Identifying Gifted Students (SIGS; Ryser & McConnell, 2004). The GRS contains six subscales measured using 72 items. As with the SRBCSS, limitations to the GRS relate to its development and its length. Although it is possible that such length is necessary to adequately assess the constructs of interest, this does not matter if logistics prevent raters from completing the forms. This situation was observed in New York City Public Schools where teachers failed to complete GRS forms on approximately 3,000 students thereby preventing those students from being identified (Otterman, 2010). Peters (2012) noted a similar situation with regard to the SRBCSS.

The SIGS (Ryser & McConnell, 2004) is the newest of the three major rating scales. In a slight deviation for the others, the SIGS focuses on academic subscales including language arts, math, science, and social studies, as well as more traditional general intellectual, creativity, and leadership. The SIGS is also different in that it is the only form to have been evaluated for differential item functioning (DIF). However, the limited diversity of the standardization sample calls into question the utility of the positive findings (Matthews, 2007). However, like the GRS and SCRBCSS, the SIGS is rather long at 84 items calling into question the utility of this instrument in rating large numbers of students.

The SIGS, GRS, and HOPE Scale were recently evaluated with regard to how well collected data matched with the theoretical structure of each instrument (Peters, 2012). As stated above, additional items and an attempt to more accurately represent the theory of the underlying construct does not increase the validity of an instrument unless the theoretical structure of the instrument is upheld using real data. In this study, Peters (2012) gathered data from a single, diverse school district in a metropolitan area in the Midwest. Using matched pairs of diverse and nondiverse schools, the author found that the theoretical models of the SIGS and the GRS did not match the data collected. The SIGS resulted in comparative fit index (CFI), Tucker–Lewis index (TLI), and root mean square error of approximation (RMSEA) values of .82, .82, and .08, respectively, and the GRS resulted in similar values of .88, .87, and .09. This was in contrast to HOPE Scale values of .93, .91, and .12. None of these fit statistics are ideal and point to more complex models as potentially problematic when teachers will be using such models on a large number of students. If general models of internal validity structure do not withstand scrutiny then there is little reason to progress to subgroup invariance testing.

HOPE Scale Development

The HOPE Scale is an instrument used by teachers to rate the prevalence and frequency of certain behaviors in their students that are commonly associated with giftedness and talent. These behaviors are broadly categorized into subscales: Academic and Social. To define each subscale, items were written after considering extant definitions of giftedness and after reviewing the literature for common behaviors of gifted children that would also be readily observable by a student’s classroom teacher with little to no specific training in gifted education. Specifically, we chose to follow Renzulli’s (1978) notion that giftedness is a behavior that can be observed, and the federal definition that suggests that students with gifts and talents perform or have the potential to perform at remarkably high levels of accomplishment when compared with similar others (U.S. Department of Education, Office of Educational Research and Improvement, 1993). We focused on writing items that together defined academic and social behaviors that might indicate giftedness. Sample items from the academic scale include the following: exhibits intellectual intensity and has intense interests. Items that define the social scale include is sensitive to larger or deeper issues of human concern and shows compassion for others. Teachers use a 6-point response scale to rate how frequently they observe behaviors described by each of the 11 items on the instrument (6 = always; 5 = almost always; 4 = often; 3 = sometimes; 2 = seldom; 1 = never). Items were not meant to be content-specific, but the HOPE Scale does include a nonscored item, which asked for specific content areas in which the student demonstrated special skills. The overarching idea was that together these items would help focus teacher recommendations and ratings and remove some degree of personal bias or variability associated with simple teacher recommendations. When teachers are simply asked to recommend student for gifted programs, they only have their prior life experience and their personal beliefs from which to draw. However, we believe that by providing a structured rating form, teachers can rely more heavily on the items themselves as opposed to their own personal opinions or perceptions. This was supported by previous research that found intraclass correlations (ICCs) for the HOPE Scale Academic subscale of .13 and .15 for the Social subscale (Peters, 2009).

The initial development of the HOPE Scale, accompanied by exploratory factor analysis, confirmatory factor analysis, and invariance testing with regard to income groups, has been previously reported at professional conferences (Peters, 2010; Peters, Gates, Gentry, Peterson, & Mann, 2009) and in the literature (Peters & Gentry, 2010). These analyses found that a two-factor model best fit the HOPE Scale data, which were collected on 5,995 K-5 students rated by their 349 respective teachers in five metropolitan and rural school districts in Indiana. In addition, HOPE Scale items were found to be invariant with regard to family income in these data. This finding indicated that the HOPE Scale data were not affected by differential item functioning with regard to income groups. However, despite this finding, students from low-income families received lower mean subscale scores than did students from non-low-income families. This finding could be explained by actual lower levels of the underlying factors—academic and social components of giftedness and talent—or by consistently lower ratings by teachers regardless of actual student behaviors. Because the HOPE Scale is a teacher- rating instrument, the results reflect the individual teacher’s perspective.

Previous research did not investigate racial/ethnic or gender group differences. In addition, at that time the HOPE Scale included a total of 13 items, only three of which loaded on the Social factor. One of these three was also found to be noninvariant with regard to income and was removed (Exhibits a strong sense of moral justice and fairness). Because three items are insufficient to measure a latent construct (Brown, 2006), six additional items that helped define the construct were added to the Social factor on the HOPE Scale, an action that necessitated further testing of the instrument. To create these items, we returned to the same expert panel as was used in the original HOPE Scale creation (Peters & Gentry, 2010). This panel included researchers and scholars in the field of gifted and talented education (n = 12) as well as those specifically interested in social/emotional topics (n = 4). Because the focus of the current study was on racial/ethnic group differences in addition to income and gender group differences, a more diverse sample was needed than was used in the original study.

Content experts approved the six new items added to HOPE Scale, which were written to better define the continuum of the social subscale. The original items included the following: Is empathetic, Shows compassion for others, and Exhibits a strong sense of social justice and fairness (although the third item was eventually removed as discussed above). These original items were created after consulting the literature and based on feedback from gifted and talented content experts (see Peters, Gates, Gentry, Peterson, & Mann, 2009). At the direction of the content experts, the following items were added to the social subscale:

When combined with the two items that were retained from the earlier draft of the HOPE Scale (Is Empathetic and Shows Compassion for Others) the content experts and authors agreed that the items on the Social subscale adequately represented many of the observable behaviors often associated with gifted and talented social/emotional characteristics (Davis, Rimm, & Siegle, 2011). Figure 1 presents the evolution of the HOPE Scale items starting with Peters and Gentry’s (2010) research and ending with the results of the present study.

OPEN IN VIEWER

Figure 1.

Evolution of HOPE Scale Items.

The middle column presents the HOPE Scale as used for the present study, and the third column displays the resulting items from the analyses of this study. Details regarding the evolution are discussed in the following pages.

Criterion-Related Validity

Another problem related to the initial development of the HOPE Scale was the lack of external/criterion-related validity evidence. Sources of validity information often include concurrent and predictive sources from other instruments that are designed to measure similar constructs, as well as long-term student outcomes. In other existing teacher rating scales authors have compared teacher rating scores with those on ability tests, achievement, tests, tests of creativity, and even other competing rating scales (Pfeiffer & Jarosewich, 2003; Ryser & McConnell, 2004). Because this information determines the degree to which an instrument is actually measuring what it is intended to measure, the lack of criterion-referenced validity was a serious weakness of the HOPE Scale and was addressed as part of the current study. HOPE Scale scores were compared with one state’s achievement tests in the same year (concurrent) and with scores from 1 year later (predictive).

Differential Item Functioning

Although external validity evidence was not addressed in the initial research on the HOPE Scale, evaluating group differences was discussed extensively (Peters & Gentry, 2010). However, additional information is presented here for clarity. Traditional methods of evaluating across- or between-group differences have often focused on mean-difference testing using t tests or similar procedures. However, in their research, Lohman (2006), Lohman, Korb, and Lakin (2008), and Brown (2006) suggested that these practices are insufficient in determining whether an instrument can yield valid information across multiple groups. Instead, an investigation of specific item-level and sub-item-level parameters needs to be conducted in order to better understand any statistical bias, or what is referred to as differential item functioning (DIF) or general measurement noninvariance (Brown, 2006; Osterlind & Everson, 2009). As discussed in previous research (Peters & Gentry, 2010) and as suggested by Brown (2006), French and Finch (2006), and others (Finch, 2005; Flowers, Raju, & Oshima, 2002; Vandenberg & Lance, 2000; Wang & Shih, 2010), structural equation modeling methods such as multiple indicator multiple causes and multigroup confirmatory factor analysis (MCFA) can be used to evaluate individual item parameters across multiple groups. These methods yield similar results to traditional item response theory (IRT) methods for evaluating DIF, and Flowers et al. (2002) found them more appropriate than IRT methods when dealing with nondichotomously scored items while at the same time yielding lower Type I error rates (French & Finch, 2006). Still, it is important to note that this is only one way in which DIF can be evaluated, and it proceeds along the following steps:

At each step, increasingly stringent restraints are placed on the model, and each is evaluated using chi-square difference tests, comparative and absolute fit indices, and predictive fit indices (Akaike information criterion [AIC] and Bayesian information criterion [BIC]). French and Finch (2006) suggested that multiple measures of model fit are least likely to result in inflated Type I error rates; therefore, this is the process we followed. If at any point a model demonstrates a significant decrease in fit as indicated by a statistically significant increase in chi-square value, change in AIC or BIC value, or a marked decrease in fit statistics below traditional cutoff values (Hu & Bentler, 1999), then some DIF is said to be present in the factors that were constrained for that particular model. If this is found to be the case, the item does not necessarily need to be removed, but special considerations need to be made when this item is included when evaluating groups for which the item has shown noninvariance. Kline (2005) suggested the AIC and BIC predictive fit indices are useful when comparing models that otherwise have very close chi-square values or fit indices. The only inherent issue is that there are no set criteria for whether or not a change in AIC or BIC value between two different models is considered significant. Instead, models with lower values are preferred (assuming similar fit indicated by other measures) as they are more likely to be replicated in the future.

Bias and Invariance Versus Fairness

Perhaps no issue permeates discussions of gifted and talented identification or student testing and assessment in general than does fairness and bias. Often used interchangeably, these two issues are not only complex but they are also inherently different in what they represent. This study deals almost universally with statistical bias as defined by Camilli (2006), “Broadly speaking, statistical bias can be thought of as a systematic difference between two parameters that should be equal . . . bias is a kind of systematic error” (p. 225). This is similar to the definition of measurement invariance presented by French and Finch (2006), “The extent to which items or subtests have equal meaning across groups of examinees” (p. 379). These two terms for the same general concept are importantly different from the larger and inherently more subjective issue of test fairness. “Only if an item is relatively more difficult for one group (statistically biased) and the source of this difficulty is irrelevant to the test construct is an item [or test] said to be unfair” (Camilli, 2006, p. 234). In this way, statistical bias is necessary but not sufficient for establishing unfairness. Fairness, which is beyond the scope of this study, requires a more subjective sensitivity review (a kind of external validity evidence) by experts.

Research Questions

Participants

Seventy-one teachers completed the HOPE Scale on their respective 1,700 K-5 students from two Midwestern states. The students all came from one of two metropolitan or one rural school district. The two metropolitan districts were chosen because of their large populations of students from low-income or culturally diverse families, with one district composed of 33% Hispanic students and the other 33% African American students, and each with more than 40% of their students eligible for free or reduced lunch. The rural district served primarily Caucasian (96%) and non–free or reduced lunch (86%) students and was selected to include students from demographic groups frequently represented in gifted and talented programs. As a viable instrument the HOPE Scale should perform equally well in both settings. All data were for students in their respective grade for the 2007-2008 schools year. The demographics of the sample are presented in Table 1.

OPEN IN VIEWER

Table 1.

Revised HOPE Scale Sample Demographics.

	West	Southern	Tiller
Designation	Metro	Rural	Metro
K-5 sample	486	675	539
Free/reduced lunch students	59%	14%	43%
Caucasian	22%	96%	33%
African American	33%	1%	17%
Hispanic	16%	1%	36%
Asian	20%	<1%	12%
Multiracial	5%	1%	0%
Native American	1%	<1%	2%
Unknown ethnicity	3%	0%	0%
Gender	51% Male	51% Male	48% Male

One of the metropolitan school districts displayed in Table 1 was used as a subsample to evaluate the relationship between HOPE Scale scores and one state’s achievement test. This district was chosen over the other two because it was located in a state that administered higher quality achievement test than the others. In this case the achievement test was conormed with and included specific items from the Stanford Achievement Test–Tenth Edition. We believed that due to the high quality of the achievement data focusing on this particular site would yield more accurate criterion-related validity information that would the state achievement test used by the remaining participants in the full sample.

Validity Evidence for Construct Interpretation

To investigate construct validity and to evaluate the overall model of the HOPE Scale, a CFA was run using data from the above-described sample. Nine items defined the Academic factor (1, 3, 8, 9, 10, 13, 15, 16, 17), and the Social factor contained eight items (2, 4, 5, 6, 7, 11, 12, 14). Although a two-factor model (Academic and Social) was hypothesized to best fit the data, a single-factor model was also evaluated for comparison purposes. These base models were evaluated using chi-square values, fit indices, modification indices, and R² values. Revisions were made based on these statistics, and revised models were evaluated with additional CFA testing. Once revisions were complete, the HOPE Scale was evaluated for measurement invariance/DIF using MCFA on the sample described in Table 1.

All the validation data analyses were conducted using MPlus (Muthén & Muthén, 1998-2007). Maximum likelihood (ML) estimation was used in all CFA and MCFA analyses. All regression analyses were conducted in SAS (SAS, 2009).

Finally, because it is appropriate practice to include either a covariance matrix or a correlation matrix with accompanying means and standard deviations (Brown, 2006; Thompson & Daniels, 1996), the general 17-item correlation and covariance matrices are presented in Figure 2. The covariance matrix includes the bold numbers on the diagonal and below while the correlation matrix is presented in the numbers above and to the right. Item and subscale means and standard deviations can be found in Table 4. Group-specific covariance matrices and correlation matrices can be requested from the first author.

OPEN IN VIEWER

Figure 2.

Revised HOPE Scale item correlations (upper) and covariance (lower) matrices.

Across-Group Equivalence for Gender, Ethnicity, and Income: Differential Item Functioning

Following CFA analyses, we analyzed the data for DIF and measurement invariance as described above and as outlined by Brown (2006) and French and Finch (2006). Three different across-group equivalence tests were conducted. The first evaluated across-gender equivalency by dividing the sample into two groups for whom gender was known. This resulted in a group of 758 male students and a group of 753 female students. The similarity of group sizes allowed for direct chi-square comparison. The second test involved across-race/ethnicity evaluation. The group sizes for this test were unbalanced with 876 Caucasian, 157 Asian, 202 African American, and 223 Hispanic students; thus, the analyses were conducted with the understanding that some small effects may have gone unnoticed due to uneven group and small sample sizes. Finally, across-income group equivalency was tested. For this step, subsamples were randomly selected from the 1,700 student sample with 500 students drawn from each income groups (i.e., free/reduced lunch eligible or non–free/reduced lunch eligible). As with the gender comparisons, this equal sample size allowed for direct chi-square comparisons. Because the subsamples were randomly selected from the larger sample, they can be considered representative of the full sample. All three of the analyses followed the above-described steps and involved computing chi-square difference tests and reporting a range of fit statistics.

Criterion-Related Validity Evidence

The Midwest State Achievement Test (MSAT: pseudonym) was used to evaluate concurrent and predictive validity information of the HOPE Scale. The MSAT measures how well students have met the state learning standards (Illinois State Board of Education, 2009). The test is also aligned with the Stanford Achievement Test (SAT10) and includes some items from the SAT10 to allow for national norm comparisons in addition to measuring the state learning standards. The MSAT is used yearly to assess reading and math achievement, beginning in Grade 3, and it is used to assess science achievement in Grades 4 and 7. Reported alpha reliability values for each grade and content area range from .89 to .94 (Illinois State Board of Education, 2009). In addition, the test manual presents detailed information with regard to item development including subscale information functions and classical test and item response theory methods of development. Similarly detailed information is available on MSAT validity including item- and subscale-total correlations, concurrent validity coefficients with the SAT10 (.91 and greater), and details on the three-parameter item response theory model used for most item revision and evaluation.

To evaluate concurrent and predictive validity, simple and Tobit regression were conducted. Pearson correlations were computed among the Academic and Social subscales of the HOPE Scale as well as the reading and math subscales of the MSAT from the 2007-2008 and 2008-2009 school years. In addition to using correlations to show the degree of relationship among the various subscales, a Tobit regression was used to determine what amount of HOPE Scale score variation could be explained by traditional achievement test scores. This was important as the relationship between an instrument being developed and the criterion-related comparison should be strong and significant, while the instrument under development also contributes some unique information (Gable & Wolf, 1993). A Tobit regression was used to address the censored nature of the HOPE Scale data. In this case, censored refers to items “piling up” at the highest score levels (McBee, 2010). This results in loss of information about these individuals as the restriction of range presents them as if they were all identical in level of the underlying construct.

Construct Validity: Confirmatory Factor Analysis

The first CFA evaluated whether the addition of the new items resulted in an improvement to the model. When this test was conducted, the results of the general CFA indicated a worse fitting model than was found using the previous version of the HOPE Scale (Peters & Gentry, 2010). All fit indices (CFI, TLI, RMSEA, and standardized root mean square residual) indicated a worse fitting model. The CFA fit indices and chi-square values for the HOPE Scale are presented in Table 2.

OPEN IN VIEWER

Table 2.

HOPE Scale Base Model Fit Statistics.

	Two-factor model		Single-factor model
Index	Value	Notes	Value	Notes
Chi-square	2862.764, df = 117	p < .001	4336.941, df = 118	p < .001
RMSEA	.119	90% CI = .115, .123	.147	90% CI = .143, .151
CFI	.917		.872
TLI	.903		.853
SRMR	.052		.055

Note. CI = confidence interval; RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker–Lewis index; SRMR = standardized root mean square residual.

As hypothesized and presented in Table 2, the two-factor solution fit the model better than the single-factor model (χ² values of 2862.764, df = 117 vs. χ² value of 4336.941, df = 118); however, the fit for the two-factor model was not as strong as recommended in the literature and not as strong as found in previous research (CFI = .917 vs. .967; RMSEA = .119 vs. .113). As a result of these initial findings, revisions to the model were made and tested.

First, we examined R² values and modification indices in making revision decisions, always within the context of the overarching theory. Items 4 and 14 had the lowest R² values at .504 and .214, respectively. Because these two items accounted for such a small proportion of the variation in the model, both were removed. All other items had R² values of between .665 and .859. Modification indices were then considered along with the content of poorly functioning items. The largest modification indices had to do with correlated error (theta-delta) terms. These modifications, if made, would allow for the error parameters of different items to correlate. In general, correlated errors should be avoided because these correlations signify an overlap in content coverage and the existence of nonunidimensional items (Asparouhov & Muthén, 2009).

The largest modification index suggested that the errors of Items 6 (Shows compassion for others) and 12 (Is empathetic) be allowed to correlate. This single modification decreased the model chi-square value by more than 624. In this case, Item 12 was removed from the HOPE Scale instead of allowing its error to correlate with Item 6. Item 12 was chosen because its removal contributed to better overall model fit than did the removal of Item 6, and Item 6 described a more directly observable behavior. This choice made sense from the standpoint of content because the items addressed very similar behaviors but also allowed for better fit by requiring one less parameter to be estimated.

The same issue was present for Items 8 (Has desire to work with advanced concepts and materials) and 9 (Is eager to explore new concepts). The errors of these two items were originally allowed to correlate as part of the HOPE Scale model. However, at this point in the Scale’s development, such a pairing was not ideal. In this case, removing Item 8 contributed to better model fit than did removing Item 9. The final suggested error correlation was between Items 15 (Is insightful and intuitive) and 16 (Thinks “outside the box”). As with the other two suggested pairings, one item, in this case Item 15, was removed as Item 16 was judged as something teachers were more likely to notice in the course of a regular classroom experience.

Item 3 (Is curious, questioning) was the only item that loaded on both factors. The modification index suggested a chi-square decrease of approximately 200 if this item was cross-loaded onto the Academic factor instead of contributing only to the Social factor as had been hypothesized. Because cross-loading items are undesirable, Item 3 was removed. Two other modification indices were above 100 but were not made since they did not make sense from the theoretical perspective of the two-factor model. Once these changes were made to the HOPE Scale and the CFA model, the revised model was reevaluated. Fit statistics for the revised model are presented in Table 3.

OPEN IN VIEWER

Table 3.

Revised Model Fit Statistics.

Index	Value	Notes
Chi-square	1051.705, df = 53	p < .001
RMSEA	.107	90% CI = .101, .113
CFI	.953
TLI	.941
SRMR	.033

Note. CI = confidence interval; RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker–Lewis index; SRMR = standardized root mean square residual.

The fit statistics and chi-square values in Table 3 show clear improvement in model fit as compared to the results of the initial CFA shown in Table 2. The CFI value (.953 vs. .917 in the original model) exceeded the traditional cutoff criteria and the TLI was much closer to this criterion (.941 vs. .903). The chi-square value also decreased by approximately 1,700 (1,051 vs. 2,862). However, the RMSEA value of .107 (vs. .119) still exceeded the recommended maximum values of .05 to .10. Once these revisions were made to the HOPE Scale, alpha reliability estimates of the data were examined (see Table 4) as were item skewness and kurtosis. Skewness and kurtosis values were computed at the item and subscale levels to assure that ML estimation was appropriate based on criteria established by Finney and DiStefano (2006), who suggested that ML estimation could be used when item skewness was less than two and kurtosis less than seven. Item skewness and kurtosis are presented in Table 5.

OPEN IN VIEWER

Table 4.

HOPE Scale Descriptive Statistics.

		Response percentage
Factor	Item	1	2	3	4	5	6	Mean	SD	r With total a	Alpha if removed b	Alpha
Academic								19.34	7.84			.93
	1	11	25	24	16	15	9	3.27	1.47	.80	.92
	6	4	13	29	24	18	12	3.77	1.33	.70	.93
	7	13	24	26	16	12	9	3.18	1.49	.81	.91
	9	7	20	27	17	16	13	3.53	1.48	.87	.91
	10	10	26	27	14	13	10	3.22	1.47	.86	.91
	11	5	13	28	20	19	15	3.81	1.41	.77	.92
Social								17.73	5.99			.92
	2	7	18	31	21	15	8	3.42	1.34	.81	.90
	3	5	16	29	23	17	10	3.60	1.34	.83	.90
	4	6	22	31	20	13	8	3.36	1.33	.73	.92
	5	4	16	32	23	16	9	3.56	1.30	.82	.90
	8	10	23	25	15	16	11	3.37	1.52	.81	.90

a

Standardized correlations. ^bStandardized coefficients.

OPEN IN VIEWER

Table 5.

Item Stem, Skewness, and Kurtosis.

Factor	Item	Item stem	Skewness	Kurtosis
Academic			.409	−.646
	1	Performs or shows potential for performing at remarkably high levels	.276	−.915
	6	Is eager to explore new concepts	.013	−.721
	7	Exhibits intellectual intensity	.354	−.812
	9	Uses alternative processes	.180	−.971
	10	Thinks “outside the box”	.403	−.790
	11	Has intense interests	−.037	−.880
Social			.170	−.649
	2	Is sensitive to larger or deeper issues of human concern	.210	−.664
	3	Is self-aware	.097	−.711
	4	Shows compassion for others	.300	−.628
	5	Is a leader within his/her group of peers	.187	−.623
	8	Effectively interacts with adults or older students	.235	−.990

Alpha reliability estimates for the two subscales were .93 (Academic) and .92 (Social), indicating a high level of internal consistence of items within each factor. The means and standard deviations of the academic and social subscales were 19.34 (SD 7.84) and 17.73 (SD 5.99), respectively. Interfactor correlations between the two subscales were also computed and found to be .88.

Item score distributions, skewness, and kurtosis values indicate a slightly lower peak (frequency) of mean scores as well as slightly thicker tails in the distributions of both the individual items and the subscales. This indicates that the teachers rated students more frequently with high and low scores than would be expected in a standard normal distribution. The skewness and kurtosis values as well as the mean and standard deviation values indicate that the Academic subscale has a slightly lower model value (12.00) than mean (19.34). This further reinforces the image of a longer positive tail with larger number of average to below average ratings. The same cannot be said for the Social subscale where the mode and mean values were closer (17.00 and 17.73, respectively). Despite the flatter peaks and longer tails that were observed in both subscales, none of this lack of normality is severe enough to violate the criteria established by Finney and DiStefano (2006). This model was then evaluated with MCFA procedures to evaluate measurement invariance. The final item stems are presented in Table 5.

Differential Item Functioning and Invariance

Although invariance was evaluated with regard to income in previous research, it was reevaluated here with additional items and a new sample. In addition, invariance due to racial/ethnic group membership and gender were evaluated. All invariance tests were conducted on the model after the modifications were made to the general CFA model as described above. Although gender equity was not the primary focus of the HOPE Scale, such invariance is important in any standardized measure. The first step in the invariance testing involved the model being evaluated on each group separately, followed by testing seven increasingly restrictive models. Tables 6, 7, and 8 present the results of the gender, race/ethnicity, and income group invariance testing.

OPEN IN VIEWER

Table 6.

Invariance Tests for Gender.

	χ2	df	χ diff 2	Δdf	RMSEA (90% CI)	SRMR	CFI	TLI	AIC	BIC
Single group solutions
Male (n = 758)	394.934*	43	—	—	.104 (.095, .113)	.03	.96	.95	20,892	21,050
Female (n = 753)	464.294*	43	—	—	.114 (.105, .124)	.03	.95	.94	20,938	21,095
Measurement invariance
Equal form	859.228*	86	—	—	.109 (.103, .116)	.03	.96	.95	41,831	42,193
Equal factor loading	876.031*	95	16.803	9	.104 (.098, .111)	.04	.96	.95	41,830	42,144
Equal indicator intercepts	918.633*	104	42.602*	9	.102 (.096, .108)	.04	.95	.95	41,854	42,120
Equal indicator error variances	946.210*	115	27.577*	11	.098 (.092, .104)	.04	.95	.95	41,860	42,067
Population heterogeneity
Equal factor variance	949.146*	117	2.936	2	.097 (.091, .103)	.04	.95	.96	41,859	42,056
Equal latent mean	1027.202*	119	78.056*	2	.101 (.095, .106)	.06	.95	.95	41,933	42,119

Note. RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean square residual; CFI = comparative fit index; TLI = Tucker–Lewis index; AIC = Akaike information criterion; BIC = Bayesian information criterion.

*

p < .001.

OPEN IN VIEWER

Table 7.

Invariance Tests for Ethnic/Racial Groups.

	χ2	df	χ diff 2	Δdf	RMSEA (90% CI)	SRMR	CFI	TLI	AIC	BIC
Single group solutions
Caucasian (n = 876)	528.487*	43	—	—	.114 (.105, .122)	.03	.96	.94	23,879	24,042
Asian (n = 157)	193.284*	43	—	—	.149 (.128, .171)	.04	.92	.90	4,467	4,571
African American (n = 202)	140.207*	43	—	—	.106 (.087, .126)	.04	.96	.94	5,715	5,828
Hispanic (n = 223)	222.616*	43	—	—	.137 (.119, .155)	.04	.92	.90	6,082	6,198
Measurement invariance
Equal form	1084.594*	172	—	—	.121 (.114, .128)	.04	.95	.93	40,145	40,864
Equal factor loading	1117.481*	199	32.887	27	.113 (.106, .119)	.05	.95	.94	40,124	40,700
Equal indicator intercepts	1148.74*	226	31.259	27	.106 (.100, .112)	.05	.95	.95	40,101	40,535
Equal indicator error variances	1327.819*	259	179.079*	33	.106 (.101, .112)	.05	.94	.95	40,214	40,473
Population heterogeneity
Equal factor variance	1346.148*	265	18.329*	6	.106 (.100, .111)	.113	.94	.95	40,221	40,448
Equal latent mean	1351.703*	271	5.555	6	.105 (.099, .110)	.113	.94	.95	40,214	40,410

Note. RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean square residual; CFI = comparative fit index; TLI = Tucker–Lewis index; AIC = Akaike information criterion; BIC = Bayesian information criterion.

*

p < .001.

OPEN IN VIEWER

Table 8.

Invariance Tests for Low-Income Versus Non-Low-Income Students.

	χ2	df	χ diff 2	Δdf	RMSEA (90% CI)	SRMR	CFI	TLI	AIC	BIC
Single group solutions
Paid (n = 500)	294.961*	43	—	—	.108 (.097, .120)	.03	.96	.95	13,741	13,885
Free/reduced (n = 500)	423.454*	43	—	—	.133 (.122, .145)	.04	.93	.91	13,652	13,796
Measurement invariance
Equal form	718.415*	86	—	—	.121 (.113, .130)	.03	.94	.93	27,394	27,728
Equal factor loading	737.171*	95	18.756	9	.116 (.109, .124)	.04	.94	.93	27,395	27,684
Equal indicator intercepts	756.596*	104	19.425	9	.112 (.105, .120)	.04	.94	.94	27,396	27,642
Equal indicator error variances	790.630*	115	34.034*	11	.108 (.101, .116)	.04	.94	.94	27,408	27,600
Population heterogeneity
Equal factor variance	801.35*	117	10.72*	2	.108 (.101, .115)	.08	.94	.94	27,415	27,597
Equal latent mean	832.37*	119	31.02*	2	.109 (.103, .117)	.10	.94	.94	27,442	27,614

Note. RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean square residual; CFI = comparative fit index; TLI = Tucker–Lewis index; AIC = Akaike information criterion; BIC = Bayesian information criterion.

*

p < .001.

Results depicted in Tables 7 and 8 indicate that no differential item functioning was present at the item level when race/ethnic and income groups were compared. These results were the same as was found in previous research with the HOPE Scale in that no item-level differences were found (Peters & Gentry, 2010). The same was true for the race/ethnicity results. This means that family background, with regard to income and race/ethnicity, did not result in a worse fitting model for the HOPE Scale. In fact, AIC and BIC predictive index values were smallest for tests of equal indicator intercepts and equal latent means—meaning that these models held even under additional constraints and are most likely to reproducible. Despite this positive finding, the single group results were mixed. Although the fit statistics met or exceeded traditional criteria for CFI and TLI indices for Caucasian and African American groups, the same cannot be said for Asian and Hispanic groups. Both indicated some degree of model misfit (particularly in RMSEA values).

Gender results were also problematic (Table 6). When invariance was tested across gender groups there was a significant increase in chi-square value, indicating noninvariant intercepts of the individual items. Because of this finding, further interpretation of the gender results was not possible as the tests of equal latent means and variances would be at least partially due to item-level noninvariance. It is also interesting to note that despite some degree of general model misfit (as indicated by high RMSEA values in the general CFA tests), the tests for race/ethnicity and income groups still resulted in finding of invariance among these grouping variables.

Because of the noninvariant findings across the board for gender, we followed the initial evaluation with an analysis of partial- or item-level invariance. This was done to see if one or two particular items were problematic or if the problem was more systemic. Unfortunately, the latter proved to be the case. Each individual item constraint resulted in a significant increase in the overall test of model fit (chi-square range of 40.74 to 44.107). This suggests that gender is a more systemic factor in HOPE Scale ratings and that no single item is the cause of the problem.

The latent variance and latent mean results for race/ ethnicity and income group were mixed. Although no differential functioning was found at the subscale level due to race/ethnicity, the same was not true for income group. Again, this result is similar to that which was found with previous research of the HOPE Scale. This finding indicated that although no differential item functioning was found to be due to income group membership, low-income students still received significantly lower scores than did students from non-low-income families (as indicated by a significant increase in chi-square value for the equal latent means test). Despite this fact, the change in AIC and BIC indices from the test of equal factor variances to equal latent means was slight, suggesting that the additional constraints were not as problematic as the significant chi-square increase might indicate. For income group comparisons, the results were again mixed. Some findings were positive (e.g., nonsignificant increase in chi-square when indicator intercepts were held constant and the consistent fit index values for most MCFA steps), but room still exists for improvement. This does not mean the instrument cannot be used for students of different income groups, but rather indicates that comparisons should not be made across such groups, as doing so would introduce an inappropriate influence of measurement error. Instead, within-group comparisons should be made.

Criterion-Related Validity Evidence

Table 9 presents the correlations among the two HOPE Scale subscales (Academic and Social) and the MSAT reading, math, and science scores completed in the same school year (2007-2008).

OPEN IN VIEWER

Table 9.

Concurrent and Predictive Correlations and Sample Sizes Among HOPE Scale and MSAT Scores.

Subscale	HOPE Academic	HOPE Social	MSAT a Reading	MSAT a Math
HOPE Academic	—
HOPE Social	.87 (519)	—
MSAT 08 Reading	.54 (176)	.51 (177)	—
MSAT 09 Reading	.55 (265)	.49 (268)	—
MSAT 08 Math	.56 (179)	.54 (180)	.82 (276)	—
MSAT 09 Math	.55 (266)	.51 (269)	.83 (456)	—
MSAT 08 Science	.57 (82)	.50 (82)	.84 (84)	.85 (85)
MSAT 09 Science	.54 (90)	.46 (91)	.86 (183)	.84 (184)

Note. MSAT = Midwest State Achievement Test.

a

Year of MSAT scores is the same as that in the horizontal row of the cell.

The correlation values presented in Table 9 indicate a consistent relationship among the Academic and Social subscales and all three content areas of the MSAT in the mid to lower .50s. This is consistent with other teacher rating instruments when compared with external aptitude tests (e.g., Ryser & McConnell, 2004). Table 9 also presents the simple correlations among the two HOPE Scale subscales completed in the 2007-2008 school year and the MSAT reading, math, and science scores received in the 2008-2009 school year.

The predictive correlation values presented in Table 9 are very similar to those concurrent correlations. This means that the concurrent validity of the information gained from the HOPE Scale is very similar to the predicative validity of the information gained from the HOPE Scale. The similar values indicate the relationship between the HOPE Scale and the MSAT is almost identical when the two are administered in the same year or one year apart.

To further investigate the predictive validity of the HOPE Scale on the various MSAT scores received 1 year later, a Tobit regression was performed. This specific type of regression was used to take into account the censored nature of the HOPE Scale scores, while analyzing the math and reading MSAT subscale scores together. Science was not used in this analysis because data were only available for one class of students (the science test is not given every year meaning only half the number of students’ data were available). Tobit regression was important because 21 students received the maximum score on the Social subscale, and 20 received the maximum score on the Academic score. Without the Tobit model, traditional regression would assume these students were exactly the same on the construct being measured. Because this is unlikely the case, the data in the Tobit model are treated as if information is missing from these cases (McBee, 2010). This analysis determined the amount of the variation in the Academic and Social subscale scores accounted for by the 2009 reading and math MSAT scores. Although it is likely that some of the MSAT scores are censored as well (low test ceiling), multiple regression does not make the same distributional assumptions on covariates/ redictor variables as it does on outcome variables, meaning the any censoring taking place in the MSAT scores does not bias the model.

Whereas the predictive validity of a traditional regression model is evaluated using an R² value, a squared multiple correlation (SMC) between the actual HOPE Scale subscale scores and the scores predicted using the Tobit model was used here. The significance tests and parameter estimates of the two model predictors (2009 reading and math MSAT scores) are presented in Table 10.

OPEN IN VIEWER

Table 10.

Significance Tests for Academic and Social Subscale Model.

Subscale	Parameter	Estimate	Standard error	t Value	p Value
Academic	Intercept	−14.433161	3.161307	−4.57	<.0001
	Read09	0.075415	0.021905	3.44	.0006
	Math09	0.074422	0.018546	4.01	<.0001
	Sigma a	5.999219	0.268348	22.36	<.0001
Social	Intercept	−4.101832	2.417824	−1.70	.0898
	Read09	0.049316	0.016811	2.93	.0034
	Math09	0.049403	0.014311	3.45	.0006
	Sigma a	4.573642	0.205982	22.20	<.0001

a

Sigma signifies the adjusted standard error compared with the standard regression.

The results presented in Table 10 confirm a statistically significant predictive relationship among the Academic and Social subscales of the HOPE Scale and the reading and math MSAT scores. SMCs were computed for both the models to explain the Academic and Social subscale scores. The SMC is interpreted in the same fashion as a traditional R² effect size. The resulting SMC values indicated the models explained 57.9% of the variation in the HOPE Scale Academic subscale score and 51.9% of the variation in the HOPE Scale Social subscale score. Both results indicate the HOPE Scale measures information similar to traditional achievement tests while also measuring additional constructs of potential interest.

Implications and Recommendations

The results discussed above are encouraging given previous, and often cited, negative research with regard to teacher ratings (e.g., Pegnato & Birch, 1959). The present research indicates that teachers can effectively rate their students from various economic and racial/ethnic backgrounds, without especially high levels of error due to group membership. The effect of race/ethnicity and income, two common variables when investigating student achievement, in invariance evaluation was relatively small given how often these two variables are cited as potential causes for underrepresentation (e.g., Briggs et al., 2008; Lohman, 2006; Peters, 2012; Plucker et al., 2010). However, the results of this study were far from perfect. Students from low-income families still received lower mean scores than students from high-income families. This could be due to a variety of outside factors and additional research is certainly warranted. Still, the race and ethnicity invariance evaluation revealed no differential item functioning or latent mean differences, indicating there were no significant differences in mean scores due to race/ethnicity groupings.

Another implication is that no instrument, regardless of the design or validation procedures, can be assumed to be free of differential item functioning or bias for or against any individual group. However, when such issues are evaluated and understood, as with the HOPE Scale, accurate comparisons can be made, within specific normative groups, without the fear that scores are inappropriately influenced by extraneous variables. This practice is suggested in the Code of Fair Testing Practices in Education (Joint Committee on Testing Practices, 2005) and should be expanded to all instruments used for identification in gifted and talented education.

A teacher-rating instrument such as the HOPE Scale, which has evidence of yielding valid and reliable information, when used to identify aspects of giftedness, can provide corroborating evidence in the identification process as well as alternate pathways for recognizing talented students who come from traditionally underserved populations (VanTassel-Baska, 2008). Finding and nurturing talents among underserved populations can help ensure equity in programs developed for gifted children as well as slow or reverse the achievement gap that exists as the highest levels of academic performance (Plucker et al., 2010). The future of America’s productivity and status as a world leader rests with the next generation of learners. They must be educated and encouraged to reach their fullest potentials, and talent from children from all economic and cultural backgrounds must be identified and cultivated.

Limitations

This study is limited by a sample that is not representative of the U.S. population and by fewer subjects in the racial/ethnic subgroup analyses than were ideal (Muthén & Muthén, 2002). Therefore, additional research is needed before generalizations can be made. In addition, teacher rating scales are always limited by the knowledge, background, and personal biases of the person completing the rating as was noted earlier. The current study did not investigate the magnitude of teacher-level effect on HOPE Scale ratings. As such, future research might investigate whether professional development in gifted education and about underrepresented populations helps yield even more accurate ratings by teachers of their students. More data are needed regarding the effect of the individual raters on the quality of teacher recommendations and ratings. Although past results noted this effect is smaller than is typical for educational assessments (ICCs of .13 for the Academic and .15 for the Social subscale), more research is still needed. In addition, further research is necessary on the revised 11-item version of the HOPE Scale regarding criterion-related validity. Although the criterion analyses conducted as part of the present study used only the retained 11 items, the form as completed by the participants included all 17 items.