Testing the Factorial Invariance of the English and Filipino Versions of the Inventory of School Motivation With Bilingual Students in the Philippines

Abstract

The study explored the invariance of Filipino and English versions of the Inventory of School Motivation (ISM) for Filipino-English bilingual students. There was invariance in the factor structure and factor loadings across the two language versions. Between-network construct validation showed consistent associations between ISM-mastery goals and sense-of-self dimensions.

Keywords

academic motivation Inventory of School Motivation invariance bilinguals

This study tests the invariance of the original English version and the Filipino translation of the Inventory of School Motivation (ISM; McInerney & Sinclair, 1992), an instrument that assesses student motivation in schools. In particular, the study (a) assessed the invariance of the ISM across two Filipino-English bilingual samples that answered one of the two language versions, and (b) assessed the between-network construct validity of the two language versions, by examining how selected motivation scales of the two versions relate to students’ self-perceptions.

Inventory of School Motivation

The ISM measures four motivational constructs that are proposed to influence learning: mastery, performance, social, and extrinsic goals. The ISM has been validated in various countries, and consistent relationships between the dimensions of school motivation and student learning variables have been found (McInerney & Ali, 2006; McInerney, Hinkley, Dowson, & Van Etten, 1998; McInerney, Yeung, & McInerney, 2001). Consistent patterns of associations have also been found between ISM scales and outcomes such as intentions for further education, positive affect, and value for school (McInerney, 2008).

Watkins, McInerney, and Boholst (2003) found good internal consistency reliability and construct validity of the scores of the English version of the ISM with Filipino students using an exploratory factor analytic approach. However, previous research is equivocal about whether the language used in psychological tests makes a difference when administered to Filipino students who speak English as a second language. Some studies (e.g., Bernardo, 2008; Watkins & Gerong, 1999) found different patterns responses and factors with different language versions of scales administered to Filipino bilinguals, but other studies have (e.g., Ganotice & Bernardo, 2010) found no differences when using different language versions. In addition, no previous study has used a CFA (confirmatory factor analysis) approach to assess the construct validity of the ISM scores in a Philippine setting.

In this study, we developed a Filipino translation of the ISM for use with Filipino-English high school students using a committee approach. A team of fluent bilinguals who were given readings and orientations on the theory and properties of the ISM scales translated, reviewed, back-translated, reviewed, and revised the items (Maneesriwongul & Dixon, 2004). The Filipino translation used conversational Filipino, which typically involves code mixing and borrowings from English. Thus, some items in the Filipino version of the ISM actually include words in English that are commonly used by Filipino high school students. In such cases, the borrowed English terms are marked by single quotation marks, as shown in the following example: “Lalo akong nagsisikap sa mga gawaing ‘interesting’” (I try harder with interesting work).

Different degrees of invariance were assessed to investigate the equivalence of the two versions of the ISM: configural invariance, measurement invariance, structural invariance, and error invariance (Byrne, 2010). A stepwise procedure was followed for testing, which we describe in the Results section. Bagozzi and Heatherton (1994) observed that it is more likely that having more than five indicators per factor in a large sample would lead to an unsatisfactory fit in the measurement model, and this was a likely scenario for the factors of the ISM. To address this concern, we aggregated the items to form item “parcels” as indicators in the analysis (Little, Cunningham, Shahar, & Widaman, 2002). Using parcels reduces the likelihood that parameters will be affected by item-specific variance (Lance, Woehr, & Fisicaro, 1991). Parcels were created by pairing items based on divergent skewness; items in a factor that were most skewed to the left were paired with those most skewed to the right. The distribution of such parcels are more likely to approximate a normal distribution better than that of individual items, and the results based on such parcels are more likely to be generalizable across samples (West, Finch, & Curran, 1995).

To explore the between-network construct validity (Martin, 2007) of the two language versions of the ISM, we looked at the correlations between the Mastery Goal scale and the dimensions of the sense of self: sense of self-reliance, sense of purpose, and positive and negative self-esteem. Previous research (Maehr & McInerney, 2004) has indicated that Mastery Goals and the Positive Sense-of-Self scales are consistently associated with positive learning processes and outcomes. Thus, we predicted that mastery goals scores should be positively associated with the scores of the Self-Reliance, Sense of Purpose, and Positive Self-Esteem scales, but negatively associated with the scores of the Negative Self-Esteem scale in both the English and Filipino versions of the ISM.

Method

Measures

Inventory of School Motivation

The original English version (McInerney et al., 1998) and the Filipino translation of the ISM comprised 43 items on different achievement-related goals that are grouped into four dimensions: mastery, performance, extrinsic, and social goals. We refer readers to McInerney et al. for details about the factors and the items therein. Participants are asked to indicate their agreement for each item using a scale from 1 (strongly agree) to 5 (strongly disagree.)

Sense-of-Self Scales

The English (McInerney et al., 2001) and the Filipino version of the Sense-of-Self Scale (Ganotice & Bernardo, 2010) was used for between-network construct validation. We refer readers to McInerney et al. for more details on the scale items. Participants responded to the items by using a scale ranging from 1 (strongly agree) to 5 (strongly disagree).

Participants

The participants were 765 students from 5 public and private high schools in 3 regions of the Philippines. Participants were randomly assigned to answer either the English or Filipino version; 378 participants (age: M =14.62, SD = 1.39; 160 males, 218 females) completed the English version, and 387 (age: M =13.98, SD = 1.41; 141 males, 246 females) completed the Filipino version. Only participants with complete data on the ISM items were included in the analyses.

Results

Preliminary Analyses

Reliability estimates (coefficient alphas) for each of the scale scores for the data on the two versions of ISM were acceptable for research purposes (Henson, 2001; see Table 1), with values comparable to those found by Watkins et al. (2003). Descriptive statistics are also presented in Table 1.

Table 1.

Alphas, Means, and Standard Deviations of the Inventory of School Motivation

	Cronbach’s alpha		M		SD
Factors	English	Filipino	English	Filipino	English	Filipino
Mastery goal	.82	.76	20.98	19.33	5.35	4.37
Performance goal	.86	.81	36.30	31.80	8.81	7.07
Social goal	.73	.72	16.90	16.24	4.49	4.18
Extrinsic goal	.88	.85	32.46	28.88	9.27	7.71

Tests of Invariance

Four types of invariance were assessed: configural, measurement, structural, and error invariance. For the first multigroup CFA, the same pattern of fixed (zero) and free factor loadings were specified for each of the two language versions (Dimitrov, 2010). This tested for configural invariance (i.e., whether the number of factors and pattern of indicator-factor loadings was identical across the two versions). The model had adequate fit with the data supporting the configural validity of the ISM across the two language versions.

Although the traditional method of showing invariance is through a chi-square difference test, Cheung and Rensvold (2002) argued that this is too stringent. They suggested looking at the comparative fit indices (CFI), with a decrease in CFI greater than or equal to 0.01 as indicator of noninvariance. We followed their recommendation in this study. For the second model, we held the factor loadings invariant across Filipino and English versions (i.e., measurement invariance). The third model holds the factor variances and covariances in the structural model invariant (i.e., structural invariance). The final model holds the error variances and covariances invariant (i.e., full invariance). Results in Table 2 indicate that the factor loadings for the two measures were invariant. But the difference in CFI for the last two models suggests noninvariance, although the fit indicators suggested adequate fit.

Table 2.

Invariance t Tests Across the English and Filipino Versions of the Inventory of School Motivation

Model	χ²	df	χ²/df	p	RMSEA	TLI	CFI	Change in CFI
1. Configural model	948.457	406	2.336	.001	0.042	0.908	0.919	—
2. Invariant factor loadings	996.544	424	2.350	.001	0.042	0.907	0.915	0.004
3. Invariant factor correlations	1072.663	434	2.472	.001	0.044	0.899	0.905	0.01
4. Invariant measurement errors	1160.636	456	2.545	.001	0.045	0.894	0.895	0.01

Note: χ² = Chi-square; df = degrees of freedom; RMSEA = root mean square error of approximation; TLI = Tucker–Lewis Index; CFI = comparative fit index.

Between-Network Construct Validity

To test the predicted relationships between Mastery Goals and Sense-of-Self scales, two sets of zero-order correlations were computed, one for each of the two language versions. The predicted relationship between the scores of the Mastery Goals and the scores of the Sense-of-Self scales were verified. In the English version, the mastery goals scores were positively correlated with the scores of the Sense of Purpose (r = .49, p < .001), Self-Reliance (r = .39, p < .001), and Positive Self-Esteem scales (r = .19, p < . 001) and negatively correlated with the scores of the Negative Self-Esteem scale (r = –.19, p < . 001). In the Filipino version, the mastery goals scores were positively correlated with the scores of the Sense of Purpose (r = .54, p < .001), Self-Reliance (r = .41, p < .001), and Positive Self-Esteem scales (r = .31, p < . 001) and negatively correlated with the Negative Self-Esteem scale scores (r = –.12, p < . 01).

Discussion

This study showed that there was configural and measurement invariance across the English and Filipino versions of the ISM. The results also provide preliminary evidence for the between-network construct validity of the two language versions by showing how the Mastery scale scores related to the scores of the students’ Sense-of-Self scale in school settings.

We should note that there are some limitations of the study. Ideally the cross-linguistic validity of the translations would be assessed using a bilingual sample that answers both language versions; in our study, a different set of bilinguals answered each version; however, there is no reason to suspect that there were marked differences between the two groups as both were randomly drawn from one population. Another limitation is that we only tested the relationship of the Mastery scale scores to Sense-of-Self scores for the between-network validity. Future studies may test for a wider range of theoretically related constructs.

These limitations notwithstanding, the results of the study provide further evidence supporting the validity of the ISM across cultures, and in this case, across language versions in a bilingual culture. Note that although this is not the first study that validated the ISM in the Philippines, this is the first study that tested the invariance of the ISM in the two languages used by many Filipino-English bilinguals. It is also likely that this is the first effort to assess the invariance across two language versions of the ISM with a bilingual sample. Thus, this study contributes to further validation of the ISM as a psychological tool that can be used in a variety of educational contexts and cultures. Our results indicate that there is strong support for assumption that the ISM measures the same four motivational constructs in both language versions and that the relationship between these constructs and other external constructs are comparable in both versions (Dimitrov, 2010). However, the evidence for noninvariance at the structural level indicates that we cannot assume that the mean scores in the two language versions are comparable. Thus, scale scores may be compared and interpreted only in relation to other scale scores in the same language version, but we cannot compare scale scores across the Filipino and English versions.

In a recent discussion of achievement motivation of Asian students, Hau and Ho (2008) noted the need to extend research on school motivation to a more diverse range of cultural groups, encompassing a more varied range of motivation constructs and outcome measures. We think that our small effort to validate two language versions of the ISM for use with Filipino-English bilingual students in the Philippines is a useful step in this direction.

Footnotes

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Support for this research was provided by a grant from the Commission on Higher Education Faculty Development Program to the first author.

Bios

Fraide A. Ganotice, Jr. is the Chairperson of the Graduate Education Department, College of Teacher Education of Palawan State University, Philippines. He obtained his PhD in Educational Psychology from De La Salle University. His research interests include scale development, translation and validation of psychological tests, motivation and learning.

Allan B. I. Bernardo is Professor and University Fellow of De La Salle University, Philippines. His research interests are in socio-cultural dimensions of learning, motivation, and achievement; social psychology of education; language and cognition, and intergroup cognitions. He obtained his PhD (Psychology) from Yale University, USA.

Ronnel B. King is a PhD candidate in Educational Psychology at The University of Hong Kong. His research interests are in the cross-cultural investigations of student motivation, engagement, and well-being.

References

Bagozzi

R. P.

Heatherton

T. F.

(1994). A general approach to representing multifaceted personality constructs: Application to state self-esteem. Structural Equation Modeling, 1, 25-67.

Bernardo

A. B. I.

(2008). Exploring epistemological beliefs of bilingual Filipino preservice teachers in the Filipino and English languages. The Journal of Psychology, 142, 193-208.

Byrne

B. M.

(2010). Structural equation modelling with AMOS: Basic concepts, applications, and programming (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9, 233-255.

Dimitrov

(2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43, 121-149.

Ganotice

F. A.

Bernardo

A. B. I.

(2010). Validating the factors of the English and Filipino versions of the Sense of Self Scale. Philippine Journal of Psychology, 43, 81-99.

Hau

K. T.

I. T.

(2008). Editorial: Insights from research on Asian students’ achievement motivation. International Journal of Psychology, 43, 865-869.

Henson

R. K.

(2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177-189.

Lance

C. E.

Woehr

D. J.

Fisicaro

S. A.

(1991). Cognitive categorization processes in performance evaluation: Confirmatory tests of two models. Journal of Organizational Behavior, 12, 1-20.

10.

Little

T. D.

Cunningham

W. A.

Shahar

Widaman

K. F.

(2002). To parcel or not to parcel: Exploring the question: Weighing the merits. Structural Equation Modeling, 9, 151-173.

11.

Maehr

M. L.

McInerney

D. M.

(2004). Motivation as personal investment. In McInerney

D. M.

Van Etten

(Eds.), Research on sociocultural influences on motivation and achievement, Vol. 4: Big theories revisited (pp. 61 -90). Greenwich, CT: Information Age Publishing.

12.

Maneesriwongul

Dixon

J. K.

(2004). Instrument translation process: A methods review. Journal of Advanced Nursing, 48, 175-186.

13.

Martin

A. J.

(2007). Examining a multidimensional model of student motivation and engagement using a construct validation approach. British Journal of Educational Psychology, 77, 413-440.

14.

McInerney

D. M.

(2008). Personal investment, culture and learning: Insights into school achievement across Anglo, Aboriginal, Asian, and Lebanese students in Australia. International Journal of Psychology, 43, 870-879.

15.

McInerney

D. M.

Ali

(2006). Multidimensional and hierarchical assessment of school motivation: Cross-cultural validation. Educational Psychology, 26, 595-612.

16.

McInerney

D. M.

Hinkley

Dowson

Van Etten

(1998). Aboriginal, Anglo, and immigrant Australian students’ motivational beliefs about personal academic success: Are there cultural differences? Journal of Educational Psychology, 90, 621-629.

17.

McInerney

D. M.

Sinclair

K. E.

(1992). Dimensions of school motivation. A cross-cultural validation study. Journal of Cross-Cultural Psychology, 23, 389-406.

18.

McInerney

D. M.

Yeung

S. Y.

McInerney

(2001). Cross-cultural validation of the Inventory of School Motivation (ISM). Journal of Applied Psychological Measurement, 2, 134-152.

19.

Watkins

D. A.

Gerong

(1999). Language of response and the spontaneous self-concept. A test of the cultural accommodation hypothesis. Journal of Cross-Cultural Psychology, 30, 115-121.

20.

Watkins

D. A.

McInerney

D. M.

Boholst

F. A.

(2003). The reliability and validity of the Inventory of School Motivation: A Philippine investigation. The Asia-Pacific Education Researcher, 12, 87-100.

21.

West

S. G.

Finch

J. F.

Curran

P. J.

(1995). Structural equation models with nonnormal variables: Problems and remedies. In Hoyle

R. H.

(Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 56-75). Thousand Oaks, CA: SAGE.