Abstract
The study explored the invariance of Filipino and English versions of the Inventory of School Motivation (ISM) for Filipino-English bilingual students. There was invariance in the factor structure and factor loadings across the two language versions. Between-network construct validation showed consistent associations between ISM-mastery goals and sense-of-self dimensions.
This study tests the invariance of the original English version and the Filipino translation of the Inventory of School Motivation (ISM; McInerney & Sinclair, 1992), an instrument that assesses student motivation in schools. In particular, the study (a) assessed the invariance of the ISM across two Filipino-English bilingual samples that answered one of the two language versions, and (b) assessed the between-network construct validity of the two language versions, by examining how selected motivation scales of the two versions relate to students’ self-perceptions.
Inventory of School Motivation
The ISM measures four motivational constructs that are proposed to influence learning: mastery, performance, social, and extrinsic goals. The ISM has been validated in various countries, and consistent relationships between the dimensions of school motivation and student learning variables have been found (McInerney & Ali, 2006; McInerney, Hinkley, Dowson, & Van Etten, 1998; McInerney, Yeung, & McInerney, 2001). Consistent patterns of associations have also been found between ISM scales and outcomes such as intentions for further education, positive affect, and value for school (McInerney, 2008).
Watkins, McInerney, and Boholst (2003) found good internal consistency reliability and construct validity of the scores of the English version of the ISM with Filipino students using an exploratory factor analytic approach. However, previous research is equivocal about whether the language used in psychological tests makes a difference when administered to Filipino students who speak English as a second language. Some studies (e.g., Bernardo, 2008; Watkins & Gerong, 1999) found different patterns responses and factors with different language versions of scales administered to Filipino bilinguals, but other studies have (e.g., Ganotice & Bernardo, 2010) found no differences when using different language versions. In addition, no previous study has used a CFA (confirmatory factor analysis) approach to assess the construct validity of the ISM scores in a Philippine setting.
In this study, we developed a Filipino translation of the ISM for use with Filipino-English high school students using a committee approach. A team of fluent bilinguals who were given readings and orientations on the theory and properties of the ISM scales translated, reviewed, back-translated, reviewed, and revised the items (Maneesriwongul & Dixon, 2004). The Filipino translation used conversational Filipino, which typically involves code mixing and borrowings from English. Thus, some items in the Filipino version of the ISM actually include words in English that are commonly used by Filipino high school students. In such cases, the borrowed English terms are marked by single quotation marks, as shown in the following example: “Lalo akong nagsisikap sa mga gawaing ‘interesting’” (I try harder with interesting work).
Different degrees of invariance were assessed to investigate the equivalence of the two versions of the ISM: configural invariance, measurement invariance, structural invariance, and error invariance (Byrne, 2010). A stepwise procedure was followed for testing, which we describe in the Results section. Bagozzi and Heatherton (1994) observed that it is more likely that having more than five indicators per factor in a large sample would lead to an unsatisfactory fit in the measurement model, and this was a likely scenario for the factors of the ISM. To address this concern, we aggregated the items to form item “parcels” as indicators in the analysis (Little, Cunningham, Shahar, & Widaman, 2002). Using parcels reduces the likelihood that parameters will be affected by item-specific variance (Lance, Woehr, & Fisicaro, 1991). Parcels were created by pairing items based on divergent skewness; items in a factor that were most skewed to the left were paired with those most skewed to the right. The distribution of such parcels are more likely to approximate a normal distribution better than that of individual items, and the results based on such parcels are more likely to be generalizable across samples (West, Finch, & Curran, 1995).
To explore the between-network construct validity (Martin, 2007) of the two language versions of the ISM, we looked at the correlations between the Mastery Goal scale and the dimensions of the sense of self: sense of self-reliance, sense of purpose, and positive and negative self-esteem. Previous research (Maehr & McInerney, 2004) has indicated that Mastery Goals and the Positive Sense-of-Self scales are consistently associated with positive learning processes and outcomes. Thus, we predicted that mastery goals scores should be positively associated with the scores of the Self-Reliance, Sense of Purpose, and Positive Self-Esteem scales, but negatively associated with the scores of the Negative Self-Esteem scale in both the English and Filipino versions of the ISM.
Method
Measures
Inventory of School Motivation
The original English version (McInerney et al., 1998) and the Filipino translation of the ISM comprised 43 items on different achievement-related goals that are grouped into four dimensions: mastery, performance, extrinsic, and social goals. We refer readers to McInerney et al. for details about the factors and the items therein. Participants are asked to indicate their agreement for each item using a scale from 1 (strongly agree) to 5 (strongly disagree.)
Sense-of-Self Scales
The English (McInerney et al., 2001) and the Filipino version of the Sense-of-Self Scale (Ganotice & Bernardo, 2010) was used for between-network construct validation. We refer readers to McInerney et al. for more details on the scale items. Participants responded to the items by using a scale ranging from 1 (strongly agree) to 5 (strongly disagree).
Participants
The participants were 765 students from 5 public and private high schools in 3 regions of the Philippines. Participants were randomly assigned to answer either the English or Filipino version; 378 participants (age: M =14.62, SD = 1.39; 160 males, 218 females) completed the English version, and 387 (age: M =13.98, SD = 1.41; 141 males, 246 females) completed the Filipino version. Only participants with complete data on the ISM items were included in the analyses.
Results
Preliminary Analyses
Reliability estimates (coefficient alphas) for each of the scale scores for the data on the two versions of ISM were acceptable for research purposes (Henson, 2001; see Table 1), with values comparable to those found by Watkins et al. (2003). Descriptive statistics are also presented in Table 1.
Alphas, Means, and Standard Deviations of the Inventory of School Motivation
Tests of Invariance
Four types of invariance were assessed: configural, measurement, structural, and error invariance. For the first multigroup CFA, the same pattern of fixed (zero) and free factor loadings were specified for each of the two language versions (Dimitrov, 2010). This tested for configural invariance (i.e., whether the number of factors and pattern of indicator-factor loadings was identical across the two versions). The model had adequate fit with the data supporting the configural validity of the ISM across the two language versions.
Although the traditional method of showing invariance is through a chi-square difference test, Cheung and Rensvold (2002) argued that this is too stringent. They suggested looking at the comparative fit indices (CFI), with a decrease in CFI greater than or equal to 0.01 as indicator of noninvariance. We followed their recommendation in this study. For the second model, we held the factor loadings invariant across Filipino and English versions (i.e., measurement invariance). The third model holds the factor variances and covariances in the structural model invariant (i.e., structural invariance). The final model holds the error variances and covariances invariant (i.e., full invariance). Results in Table 2 indicate that the factor loadings for the two measures were invariant. But the difference in CFI for the last two models suggests noninvariance, although the fit indicators suggested adequate fit.
Invariance t Tests Across the English and Filipino Versions of the Inventory of School Motivation
Note: χ2 = Chi-square; df = degrees of freedom; RMSEA = root mean square error of approximation; TLI = Tucker–Lewis Index; CFI = comparative fit index.
Between-Network Construct Validity
To test the predicted relationships between Mastery Goals and Sense-of-Self scales, two sets of zero-order correlations were computed, one for each of the two language versions. The predicted relationship between the scores of the Mastery Goals and the scores of the Sense-of-Self scales were verified. In the English version, the mastery goals scores were positively correlated with the scores of the Sense of Purpose (r = .49, p < .001), Self-Reliance (r = .39, p < .001), and Positive Self-Esteem scales (r = .19, p < . 001) and negatively correlated with the scores of the Negative Self-Esteem scale (r = –.19, p < . 001). In the Filipino version, the mastery goals scores were positively correlated with the scores of the Sense of Purpose (r = .54, p < .001), Self-Reliance (r = .41, p < .001), and Positive Self-Esteem scales (r = .31, p < . 001) and negatively correlated with the Negative Self-Esteem scale scores (r = –.12, p < . 01).
Discussion
This study showed that there was configural and measurement invariance across the English and Filipino versions of the ISM. The results also provide preliminary evidence for the between-network construct validity of the two language versions by showing how the Mastery scale scores related to the scores of the students’ Sense-of-Self scale in school settings.
We should note that there are some limitations of the study. Ideally the cross-linguistic validity of the translations would be assessed using a bilingual sample that answers both language versions; in our study, a different set of bilinguals answered each version; however, there is no reason to suspect that there were marked differences between the two groups as both were randomly drawn from one population. Another limitation is that we only tested the relationship of the Mastery scale scores to Sense-of-Self scores for the between-network validity. Future studies may test for a wider range of theoretically related constructs.
These limitations notwithstanding, the results of the study provide further evidence supporting the validity of the ISM across cultures, and in this case, across language versions in a bilingual culture. Note that although this is not the first study that validated the ISM in the Philippines, this is the first study that tested the invariance of the ISM in the two languages used by many Filipino-English bilinguals. It is also likely that this is the first effort to assess the invariance across two language versions of the ISM with a bilingual sample. Thus, this study contributes to further validation of the ISM as a psychological tool that can be used in a variety of educational contexts and cultures. Our results indicate that there is strong support for assumption that the ISM measures the same four motivational constructs in both language versions and that the relationship between these constructs and other external constructs are comparable in both versions (Dimitrov, 2010). However, the evidence for noninvariance at the structural level indicates that we cannot assume that the mean scores in the two language versions are comparable. Thus, scale scores may be compared and interpreted only in relation to other scale scores in the same language version, but we cannot compare scale scores across the Filipino and English versions.
In a recent discussion of achievement motivation of Asian students, Hau and Ho (2008) noted the need to extend research on school motivation to a more diverse range of cultural groups, encompassing a more varied range of motivation constructs and outcome measures. We think that our small effort to validate two language versions of the ISM for use with Filipino-English bilingual students in the Philippines is a useful step in this direction.
Footnotes
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Support for this research was provided by a grant from the Commission on Higher Education Faculty Development Program to the first author.