Assessing Motivation of Secondary School Students: An Analysis of Promotion and Prevention Orientations as Measured by the Regulatory Focus Questionnaire

Abstract

Measuring human motivation requires understanding the outcomes individuals value and the strategies they prefer to employ to attain them. Knowledge of promotion and prevention, two pivotal motivation orientations, provide key information regarding these aspects. The Regulatory Focus Questionnaire, which measures these two independent constructs, was validated using data provided by university students and alumni of an elite U.S. university. Thus, little is known whether this instrument provides reliable and valid measures of promotion and prevention in a population of younger respondents from a different culture. To bridge this gap, the study employed data collected from three independent large samples of New Zealand secondary school students and used the jigsaw piecewise technique in combination with confirmatory factor analyses. Findings show that, in this population, items in the Regulatory Focus Questionnaire measure promotion and prevention as theoretically distinct constructs.

Keywords

motivation promotion prevention Regulatory Focus Questionnaire (RFQ)

Motivation is an important factor that influences how students participate in, and engage with, school-related activities (Eccles & Wigfield, 2002). The research literature has provided evidence that knowledge of two motivation orientations, namely, promotion and prevention, enables access to information regarding key antecedents of performance and well-being in school (e.g., expectancies of success, strategic selection/pursuit of valued goals; Higgins, 2012b; Hodis & Hodis, 2015). Regulatory focus theory proposed that individuals who hold a promotion orientation have strong concerns with growth and advancement, construe goals as ideals, and seek opportunities for growth and advancement (Higgins, 1997, 2012b; Wigfield, Muenks, & Rosenzweig, 2015). In contrast, individuals holding a prevention orientation are motivated by concerns with safety and security. As a consequence, they represent their key goals as duties and responsibilities that must be upheld and guard vigilantly against anything that may compromise their ability to fulfill them (Higgins, 1997, 2012b; Wigfield et al., 2015).

Regulatory focus theory (Higgins, 1997) provided the conceptual foundation for several instruments measuring promotion and prevention. A recent meta-analysis (Gorman et al., 2012) found that the Regulatory Focus Questionnaire (RFQ; Higgins et al., 2001) has been the second most widely used instrument in research investigating interrelationships among promotion, prevention, and key work-related outcomes (e.g., task performance, commitment). As the validation of the RFQ used data collected from students and alumni of an elite U.S. university, it is unclear whether this instrument could be employed to gauge promotion and prevention in developmentally younger populations. In light of recent findings, this is an important limitation of extant knowledge. Specifically, Hodis, Hattie, and Hodis (2016) collected data from two independent samples of high school students and examined the measurement properties of another widely used instrument developed on the conceptual framework of regulatory focus theory. The results of their investigation showed that, in this population, promotion and prevention were not empirically distinguishable when measured by means of Lockwood, Jordan, and Kunda’s (2002) General Regulatory Focus Measure (GRFM). These findings prompted Hodis et al. (2016) to question whether it is possible to measure promotion and prevention as independent constructs in adolescent populations.

Findings reported by Rosenzweig and Miele (2016) suggest that regulatory focus influences academic test performance. In this light, it is important to ascertain whether students’ promotion and prevention could be measured in a theory-consistent way across multiple developmental ages/stages. This endeavor is especially critical for adolescent populations where research has identified significant interrelationships among regulatory focus and key antecedents of academic achievement (e.g., expectancies of success and utility value; Hodis & Hodis, 2015). Moreover, as classroom interventions altering regulatory focus could provide “effective, viable ways to enhance achievement” (Wigfield et al., 2015, p. 15), it is essential to use reliable and valid measures to quantify regulatory focus and changes therein. Yet, as findings from Hodis et al. (2016) cautioned, it cannot be assumed that all measures of promotion and prevention that are grounded in the regulatory focus theorizing could be used successfully with adolescent populations.

Given these aspects and consistent with Gorman and colleagues’ (2012) recommendation that the exploration of the psychometric properties of the major regulatory focus instruments “should continue” (p. 170), this study has two key aims. The first is to evaluate the extent to which the RFQ items are strong indicators of promotion and prevention, respectively, in a population (i.e., secondary school students) that differs significantly from that in which the instrument was validated in terms of developmental age and heterogeneity in reading abilities. Consistent with theoretical tenets of regulatory focus theory (Higgins, 1997), the second goal of this research is to assess whether the RFQ gauges promotion and prevention as independent constructs in this population.

Conceptual Framework Anchoring the RFQ

Regulatory focus theory (Higgins, 1997) draws from self-discrepancy theory (Higgins, 1987) to posit that early interactions among children and their parents/caregivers shape the type of self-guide they develop and adopt. These self-guides, or “self-directive standards” (Higgins, 1987, p. 321), are key goals that guide an individual’s self-regulation (Higgins, 2015). Accessibility theory proposed that mental representations encoded in memory are accessible and, thus, can be activated for use when they are relevant to a given context of self-regulation (Higgins, 2012a). In line with this theory, it follows that self-guides are easily accessible in mind and, therefore, influence children’s behavioral actions, affective reactions, and social judgments (Higgins, 2012b). In particular, when early interactions emphasize the importance of children’s ideals and hopes, children adopt an ideal self-guide (i.e., a representation of who they aspire and hope to be; Higgins, 2015; Higgins et al., 2001) and develop a strong promotion orientation. In contrast, when early socializations highlight primarily the need for safety and the importance of meeting one’s duties and responsibilities, children are likely to adopt an ought self-guide. An ought self-guide encompasses children’s beliefs about who is their obligation or duty to be (Higgins, 2015; Higgins et al., 2001) and facilitates the development of a strong prevention orientation (Higgins, 1997). Importantly, regulatory focus theory argued that promotion and prevention are independent motivation orientations (Higgins, 1997, 2012b; Molden, 2012).

People driven by a promotion focus strive to attain their ideals (Higgins, 2012b). As long as progress appears likely, they prefer to use strategies that maximize the chances to gain and advance even at the cost of making errors and sustaining losses (i.e., eager strategies; Higgins, 2012b; Molden, 2012). Consistent with their preoccupation with achievement and gains, the self-regulation of promotion-oriented individuals revolves around positive reference points (e.g., they strive to attain positive outcomes). As a result, these individuals do not perceive neutral states as satisfactory but strive to gain and advance (Higgins, 1997, 2012b). Adoption of ideal self-guides and a focus on gains and advancement render promotion-oriented individuals vulnerable to depression when they consistently fail to attain their promotion goals (Higgins, 2012b).

People having a strong prevention orientation adopt ought self-guides; the essential drivers of these ought self-guides are either “conveyed from parents or other authority figures, or are intrinsic to the adherence to social roles” (Summerville & Roese, 2008, p. 248). In addition, prevention-focused individuals are preoccupied with safety and responsibility. Consistent with this focus, the self-regulation of prevention-oriented individuals centers on negative reference points. More specifically, they try to maintain safety by striving to secure the absence of negative outcomes (e.g., by avoiding losses; Higgins, 1997, 2012b). As long as they perceive a current state as being neutral or satisfactory, they prefer to use strategies that vigilantly guard the status quo (Higgins, 2012b). This strategic outlook makes it likely that they prefer to forsake opportunities to gain and advance rather than risk a loss (Higgins, 1997, 2012b; Molden, 2012). Notably, even when they are successful in avoiding losses, individuals having a strong prevention orientation tend to be somewhat worried and apprehensive. As a consequence, systemic failure regarding prevention is associated with increased anxiety (Higgins, 2012b).

Research reported by Miele, Molden, and Gardner (2009) highlighted other consequential effects of regulatory focus. Specifically, results of two experimental studies indicated that promotion and prevention orientations are associated with using different types of strategies to regulate comprehension. In particular, consistent with their focus on achieving gains (maintaining safety/security), promotion-oriented (prevention-oriented) individuals are likely to aim to advance their understanding (minimize their lack of comprehension) when reading difficult or mildly confusing texts. As a consequence of setting these distinct aims, promotion- and prevention-oriented individuals chose different strategies to regulate their comprehension: the former were “more likely to read on and wait for additional information” (Miele et al., 2009, p. 784), whereas the latter were more likely to reread and reprocess previously encountered information. Moreover, results reported by Miele et al. (2009) highlighted that, in some circumstances, the adoption of distinct strategies may lead to differential levels of comprehension. Thus, these findings suggest that regulatory focus plays an important role in shaping cognitive processes that are at the core of learning.

Method

Samples

The undertaking of this study was approved by the University’s Ethics Committee. Individual consent for participation was obtained from each respondent and from the principals of the schools in which students were enrolled. Every school that took part in this research received the student surveys in paper form and organized the data collection for its pupils. In each school, all students enrolled in years 10 to 13 were invited to take part in the research; in the New Zealand system of education, Year 13 is typically the last year of secondary school. Respondents received no incentives for participating in this study.

In this research, we used data collected from three independent samples. The first sample consisted of 972 secondary school students, who attended one of four secondary schools; these schools were randomly selected from all secondary schools in New Zealand. Given the socioeconomic status (SES) of the community in which the schools are located, three of the schools can be classified as high decile (i.e., high SES) schools; the fourth school was a medium SES school. The second sample comprised 605 students who attended a low SES secondary school. The third sample included 332 students who attended another low SES secondary school. The demographic characteristics of all samples are summarized in Table 1.

Table 1.

Demographic Characteristics (in %) of Sample 1 (N = 972), Sample 2 (N = 605), and Sample 3 (N = 332).

	F	Year in school				Ethnicity
	F	10	11	12	13	Māori	Pacific	European	Asian	Other
Sample 1	80.97	30.35	22.22	22.84	20.27	2.99	0.93	68.93	11.52	11.93
Sample 2	45.95	31.57	29.26	19.00	12.56	22.48	20.99	19.50	21.16	10.75
Sample 3	43.37	26.51	25.60	18.37	15.96	12.05	14.16	31.63	9.04	24.10

Note. Percentages do not add up to 100 because some respondents did not provide answers on one or more demographic questions. F = Female.

Instrument

The RFQ scale (Higgins et al., 2001) is an instrument that has been used often in research gauging promotion and prevention (Gorman et al., 2012; Summerville & Roese, 2008). Several studies undertook extensive investigations of the RFQ’s reliability as well as of its convergent, divergent, and predictive validity; for details and discussions, interested readers could see Higgins et al. (2001); Haws, Dholakia, and Bearden (2010); and Summerville and Roese (2008). The RFQ includes six promotion and five prevention items. The items center primarily on a “self-guide” (Summerville & Roese, 2008, p. 248) conceptualization of regulatory focus, which has at its core the idea that people employ ideal and ought self-guides to regulate their goal pursuits. Consistent with this conceptualization, most of the RFQ items tap into respondents’ past self-guide experiences (e.g., “Not being careful enough has gotten me into troubles at times”) or into their interactions with parents (e.g., “Did you get on your parents’ nerves often when you were growing up?”) (Summerville & Roese, 2008). Although no item in the RFQ references school or academic domains, the instrument has been often administered to university students (e.g., Haws et al., 2010; Summerville & Roese, 2008). All items used in this investigation were rated on a 7-point Likert-type scale anchored at 1 (strongly agree) and 7 (strongly disagree).

In this research, we needed to alter slightly some of the original items to make them easy to understand across a broad range of reading abilities; significant variations in reading ability are likely to be more common among secondary school students than among students and alumni of an elite university, who provided data to validate the RFQ. Specifically, we (a) replaced all instances of negative wording with their positive equivalent (e.g., we replaced “typically unable” with “typically able”; “I don’t perform” with “I perform”); (b) rephrased items written in a question format, whose structure may be confusing for less accomplished readers, with the corresponding item written in a sentence format (e.g., we replaced “Did you get on your parents’ nerves often when you were growing up?” with “When I grew up, I often got on my parents’ nerves”); (c) rephrased expressions that were ambiguous or unfamiliar to respondents from this population (e.g., we replaced “How often have you accomplished things that got you ‘psyched’ to work even harder?” with “Accomplishing things motivates me to work even harder”; we replaced “I have found very few hobbies or activities in my life that capture my interest or motivate me to put effort into them” with “There are activities in my life that capture my interest and motivate me to put effort into them”).

Analysis

Consistent with Bollen’s (2000) recommendations, we employed the “jigsaw piecewise technique” (p. 78, emphasis in original) to assess whether the RFQ scores provide reliable and valid measures of promotion and prevention in our target population. An important feature of this technique is that all the components of the overall measurement model are first examined separately before fitting the overall model (Bollen, 2000). This strategy has three key advantages that are pivotal to our research. First, investigating individual model components separately (i.e., fitting separate models for promotion and prevention, respectively) enables the identification of construct-specific sources of misfit. Second, examining the overall model incorporating both promotion and prevention allows assessing whether analyzing the measurement models of individual constructs misses/obscures spurious or suppressor relations. Third, contrasting corresponding parameter estimates between individual and overall models enables identifying potential model misspecifications (Bollen, 2000).

In the first phase of the research, which used data collected from Sample 1, we conducted separate one-factor confirmatory factor analyses (CFAs) for promotion and prevention. Following, we used the magnitudes of the standardized factor loadings in the two unidimensional CFAs to flag strong versus weak indicators of the constructs. In the second phase, which employed data collected from Sample 2, we once more conducted separate unidimensional CFAs for promotion and for prevention. We used the results of the analyses in Phases 1 and 2 to identify items that measure well promotion and prevention. Specifically, we considered that items having in both samples standardized factor loadings of 0.50 or higher were strong indicators; we retained these items for the analysis in Phase 3. We selected 0.50 as the cutoff value for standardized loadings because Widaman, Little, Preacher, and Sawalani (2011) showed that a standardized loading of 0.50 corresponds to a mean interitem correlation (MIC) of about 0.25; in turn, a MIC of 0.25 lies in the (0.15, 0.50) interval in which MIC values for scales should fall (Clark & Watson, 1995).

The third phase of the investigation, which used data collected from Sample 2, included only the items identified as being strong indicators of their respective constructs. In this phase, we evaluated the two-factor CFA model incorporating both promotion and prevention to check whether fitting separate one-factor models obscures spurious/suppressor relations (Bollen, 2000) and to examine the magnitude of the interrelationship between the promotion and prevention factors. To avoid capitalizing on chance, in the fourth phase, we repeated the analysis conducted in Phase 3 on data collected from a fresh sample of participants, namely Sample 3.

All the analyses were conducted in Mplus, version 6.11 (Muthen & Muthen, 2010). To use all available data, we employed full information maximum likelihood (FIML; Arbuckle, 1996). This procedure is robust to small and medium violations of multivariate normality (MVN; Fan & Wang, 1998). Following, we provide evidence suggesting that no problematic violations of the MVN were detected in this study for any of the items in any of the samples. Specifically, for Sample 1, the skew of the promotion items ranged between −1.10 and −0.29 (median = −0.57); their kurtosis ranged between 0.08 and 1.16 (median = 0.55). In this sample, the skew of the prevention items ranged between −0.74 and 0.17 (median = −0.19); their kurtosis ranged between −1.00 and −0.14 (median = −0.82).

For Sample 2, the skew of the promotion items ranged between −0.83 and −0.27 (median = −0.48); their kurtosis ranged between 0.03 and 0.41 (median = 0.24). In Sample 2, the skew of the prevention items ranged between −0.72 and 0.23 (median = 0.13); their kurtosis ranged between −1.08 and −0.14 (median = −0.87). For Sample 3, the skew of the promotion items ranged between −0.79 and −0.15 (median = −0.30); their kurtosis ranged between −0.55 and 0.11 (median = −0.19). In Sample 3, the skew of the prevention items ranged between −0.76 and 0.25 (median = 0.09); their kurtosis ranged between −0.93 and −0.22 (median = −0.65). Importantly, violations of the MVN are suspected only when absolute values of skew (kurtosis) exceed 2.00 (respectively, 7.00; Curran, West, & Finch, 1996). The results reported afore indicate that this was not the case in the current research.

Consistent with Hu and Bentler’s (1999) guidelines, we used the comparative fit index (CFI; Bentler, 1990), the Tucker–Lewis index (TLI; Tucker & Lewis, 1973), and the root mean square error of approximation (RMSEA; Steiger, 1990) to assess model fit. For these indices, values of .90 and higher (for CFI and TLI) and 0.05 and lower (for RMSEA) indicate good fit.

Results

None of the analyses we conducted had estimation problems or inadmissible solutions. For both promotion and prevention, we identify individual items with labels recording their order in the original instrument; for instance, I1 denotes the first item of the RFQ scale.

Phase 1.1: The One-Factor CFA Model for Promotion in Sample 1

This model had a good fit: χ²(9, N = 972) = 58.416, p < .001; CFI = .965; TLI = .941; RMSEA = .075, 90% confidence interval (CI) for RMSEA being [.057, .094]. Results reported in Table 2 show that all six promotion items had standardized loadings equal to or larger than 0.50 (within rounding error). Based on these findings, for the next phases of this research, we did not make any change to the promotion items used with Sample 1.

Table 2.

Maximum-Likelihood Estimates of Descriptive Statistics and One-Factor CFA Standardized Loadings and Standard Errors of Promotion Items in Sample 1.

Item	Standardized factor loading	SE	Mean item	Variance item
I1	0.619	0.025	4.846	1.449
I3	0.529	0.028	5.496	1.502
I7	0.644	0.023	4.846	1.469
I9	0.495	0.028	5.092	1.546
I10	0.796	0.019	4.953	1.453
I11	0.606	0.025	5.900	1.361

Note. Item numbers, in the first column, correspond to those in the RFQ instrument (Higgins et al., 2001). CFA = confirmatory factor analysis; RFQ = Regulatory Focus Questionnaire.

Phase 1.2: The One-Factor CFA Model for Prevention in Sample 1

This model had an excellent fit: χ²(5, N = 971) = 6.750, p = .240; CFI = .999; TLI = .997; RMSEA = .019, with the 90% CI for RMSEA being [.000, .051]. Results reported in Table 3 reveal that four of the five prevention items had standardized factor loadings larger than 0.50. The item having a loading below 0.50 (i.e., I5, “As a child, I often obeyed rules and regulations that were established by my parents”; loading = 0.435) was clear, unambiguous, and asked for the kind of information that all respondents in this population are reasonably likely to possess and recall. As we had no hypothesis regarding the types of changes that could strengthen this indicator, we did not modify this item for the data collection involving the second and third samples.

Table 3.

Maximum-Likelihood Estimates of Descriptive Statistics and One-Factor CFA Standardized Loadings and Standard Errors of Prevention Items in Sample 1.

Item	Standardized factor loading	SE	Mean item	Variance item
I2	0.823	0.018	4.539	3.033
I4	0.694	0.021	3.888	2.954
I5	0.435	0.030	5.082	2.400
I6	0.742	0.020	4.428	2.490
I8	0.507	0.028	3.723	2.616

Note. Item numbers, in the first column, correspond to those in the RFQ instrument (Higgins et al., 2001). Items in italics (i.e., I2, I4, I6, and I8) were reverse scored before the analyses. CFA = confirmatory factor analysis RFQ = Regulatory Focus Questionnaire.

Phase 2.1: The One-Factor CFA Model for Promotion in Sample 2

This model had an acceptable fit: χ²(9, N = 605) = 44.838, p < .001; CFI = .953; TLI = .922; RMSEA = .081, with the 90% CI for RMSEA being [.058, .106]. An examination of the standardized factor loadings, reported in Table 4, revealed that only one item (i.e., I1) had a standardized factor loading below 0.50. However, considering that the magnitude of this loading was only slightly smaller than that of the criterion (i.e., 0.486 in Sample 2) and that I1 was a strong indicator of promotion in Sample 1 (i.e., had a standardized loadings of 0.619), we concluded that this item is a useful measure of promotion and we retained it for the analysis in Phase 3. Finally, I9, which had a standardized loading of 0.495 in Sample 1, had a higher loading in Sample 2 (i.e., 0.543). Taking all of these results into consideration, we retained all six promotion indicators for Phase 3.

Table 4.

Maximum-Likelihood Estimates of Descriptive Statistics and One-Factor CFA Standardized Loadings and Standard Errors of Promotion Items in Sample 2.

Item	Standardized factor loading	SE	Mean item	Variance item
I1	0.486	0.037	4.539	1.609
I3	0.577	0.034	5.555	1.669
I7	0.699	0.029	4.984	1.453
I9	0.543	0.035	5.140	1.738
I10	0.751	0.027	5.095	1.528
I11	0.519	0.036	5.588	1.752

Note. Item numbers, in the first column, correspond to those in the RFQ instrument (Higgins et al., 2001). All of these items were retained for the analysis in Phase 3. CFA = confirmatory factor analysis; RFQ = Regulatory Focus Questionnaire.

Phase 2.2: The One-Factor CFA Model for Prevention in Sample 2

This model had an excellent fit to the data: χ²(5, N = 599) = 7.854, p = .164; CFI = .995; TLI = .990; RMSEA = .031, with the 90% CI for RMSEA being [.000, .070]. Results reported in Table 5 show that, as it was the case in Sample 1, I5 was the only item having a standardized loading below 0.50 (i.e., 0.276). As these findings suggest that I5 is a suboptimal measure of prevention in this population, we removed this item for the analyses in Phases 3 and 4. In all, considering the results of the analyses in Phases 1 and 2, we used all six promotion items (i.e., I1, I3, I7, I9, I10, and I11) and four of the five prevention items (i.e., I2, I4, I6, and I8) in Phases 3 and 4 of the research.

Table 5.

Maximum-Likelihood Estimates of Descriptive Statistics and One-Factor CFA Standardized Loadings and Standard Errors of Prevention Items in Sample 2.

Item	Standardized factor loading	SE	Mean item	Variance item
I2	0.819	0.028	4.020	3.459
I4	0.636	0.032	3.407	2.960
I5	0.276	0.043	5.170	2.572
I6	0.627	0.033	3.879	2.622
I8	0.540	0.036	3.335	2.703

Note. Item numbers, in the first column, correspond to those in the RFQ instrument (Higgins et al., 2001). Items in italics (i.e., I2, I4, I6, and I8) were reverse scored before the analyses. Bolded items were retained for the analysis in Phase 3. CFA = confirmatory factor analysis; RFQ = Regulatory Focus Questionnaire.

Phase 3: The Two-Factor CFA Model for Promotion and Prevention in Sample 2

The two-factor CFA model of promotion and prevention imposed a highly restrictive structure on the measurement model, as it did not include any cross-loadings or correlated residuals. This model had a good fit to the data: χ²(34, N = 605) = 88.899, p < .001; CFI = .958; TLI = .945; RMSEA = .052, with the 90% CI for RMSEA being [.039, .065]. The standardized parameter estimates associated with this model are summarized in Table 6. All factor loadings were statistically significant and of sizable magnitude. Both promotion and prevention had good construct reliability, as measured by the H coefficient (Mueller & Hancock, 2008): H = .79 for promotion; H = .78 for prevention. An examination of the association between the promotion and prevention factors revealed that they were uncorrelated (i.e., r = −.019; p = .717).

Table 6.

Maximum-Likelihood Estimates of Two-Factor CFA Standardized Loadings and Standard Errors of Promotion and Prevention Items in Sample 2/Sample 3.

Item	Scale	Standardized factor loading	SE
I1	PRO	0.485/0.586	0.037/0.044
I3	PRO	0.576/0.538	0.034/0.047
I7	PRO	0.699/0.622	0.029/0.042
I9	PRO	0.543/0.589	0.035/0.045
I10	PRO	0.751/0.822	0.027/0.031
I11	PRO	0.519/0.560	0.036/0.047
I2	PRE	0.801/0.867	0.029/0.032
I4	PRE	0.647/0.749	0.032/0.036
I6	PRE	0.637/0.600	0.033/0.046
I8	PRE	0.543/0.580	0.036/0.044

Note. Item numbers, in the first column, correspond to those in the RFQ instrument (Higgins et al., 2001). All prevention items were reverse scored before the analysis. The values in italics correspond to Sample 3. CFA = confirmatory factor analysis; PRO = promotion; PRE = prevention; RFQ = Regulatory Focus Questionnaire.

To examine the impact of purging I5 from different vantage points, we compared the interfactor correlations between promotion and prevention in the two-factor CFA models with and without I5. The results showed that both correlations were not significantly different from 0 and differed in magnitude by only .03. In addition, we analyzed the stem of I5 in the context of all the other prevention items. This analysis revealed that the content of I5 has a significant overlap with that of I2, which is the prevention item with the highest standardized factor loadings in both samples. Taking all of these aspects into consideration, we concluded that purging I5 is unlikely to affect the interrelation between the promotion and prevention factors or to narrow the conceptual domain of the prevention construct.

In line with Bollen’s (2000) recommendations, to examine whether potential misspecifications were apparent, we compared and contrasted the corresponding standardized factor loadings derived from the one- and two-factor CFAs in Sample 2 (see Tables 4, 5, and 6). For the promotion items, these comparisons revealed that all corresponding loadings were identical up to at least the second decimal point. Differences for prevention items were also very small (i.e., ranging between 0.003 and 0.018). Taken together, these results suggest that model misspecifications are unlikely for both promotion and prevention factors.

Phase 4: The Two-Factor CFA Model for Promotion and Prevention in Sample 3

The two-factor CFA model of promotion and prevention that was fit to the data collected from Sample 3 was identical to the corresponding model used in Sample 2. This model had an excellent fit: χ²(34, N = 330) = 61.877, p = .002; CFI = .966; TLI = .955; RMSEA = .050, with the 90% CI for RMSEA being [.029, .069]. The standardized parameter estimates associated with this model are summarized, in italics, in Table 6. As it was also the case for the corresponding model in Sample 2, all factor loadings were statistically significant and of sizable magnitude. Importantly, the items that had standardized factor loadings slightly below the criterion in Sample 1 (i.e., I9) or in Sample 2 (i.e., I1) had strong standardized loadings in Sample 3 (i.e., 0.589 and 0.586, respectively). Thus, the analysis of the data provided by Sample 3 offers additional support to argue that these items are strong indicators of the promotion construct in this population (i.e., that this construct and its indicators share strong interrelationships).

In Sample 3, both promotion and prevention had excellent construct reliability, as measured by the H coefficient (Mueller & Hancock, 2008): H = .82 for promotion; H = .84 for prevention. The correlation between the promotion and prevention factors was not statistically significant (r = .093; p = .176). Overall, the results of fitting the two-factor CFA model were consistent across Samples 2 and 3. This finding enhances confidence regarding the validity of our conclusions.

Discussion

Findings in this research show that, with one exception, the RFQ items (Higgins et al., 2001) were strong indicators of promotion and prevention in this population when slight alterations in the stems of some items were performed (e.g., replacements of unfamiliar/ambiguous expressions as well as of negative wording; changes from question to statement formats). These minor changes did not affect the conceptual meaning of any item. They were performed to accommodate a range of reading abilities that is likely to be broader in this developmentally younger population than in the populations in which the instrument was validated (i.e., students and alumni of an elite U.S. university). In addition, purging the one prevention item that was not a strong indicator of the construct in this population is unlikely to affect the width of the conceptual domain gauged by the RFQ. Moreover, our findings indicate that, consistent with the tenets of regulatory focus theory (Higgins, 1997, 2012b), the RFQ measured promotion and prevention as independent dimensions in this population.

To put these findings into perspective, we situate our results within the broader corpus of investigations analyzing the RFQ and other measures of regulatory focus. To this end, we explore an intriguing question: Why, in similar populations of secondary school students, the GRFM failed to capture promotion and prevention as independent constructs—as Hodis et al. (2016) found—whereas the RFQ did not? To begin answering this question, it is important to note that Summerville and Roese (2008) argued that items comprising the two instruments draw from different conceptual areas of the nomological network of promotion and prevention (defined by the regulatory focus theory). Specifically, the GRFM items tap into a reference-point conceptualization of regulatory focus (Summerville & Roese, 2008). In turn, conceptualizing regulatory focus from a reference-point perspective underlines that a pivotal distinction between promotion and prevention concerns the type of reference point that is used in self-regulation: a positive reference point/end-state for the former and a negative one for the latter (Summerville & Roese, 2008).

In contrast, as noted previously, items from the RFQ reflect a self-guide conceptualization of regulatory focus, wherein promotion is linked to the adoption of an ideal self-guide, whereas prevention is associated with the adoption of an ought self-guide (Summerville & Roese, 2008). Given (a) this conceptual distinction between the respective foundations of the RFQ and GRFM instruments and (b) Summerville and Roese’s (2008) argument that promotion and prevention measured by the RFQ may be different from promotion and prevention gauged by the GRFM, the apparent discrepancy between our findings and those of Hodis et al. (2016) is resolved. Importantly, as it was the case for findings in Hodis et al., results reported by Summerville and Roese did “raise questions about the validity of the reference-point definition” (p. 253) of regulatory focus. Taken together, our findings and those reported by Hodis et al. suggest that employing the RFQ (but not the GRFM) enables measuring promotion and prevention as independent constructs in this population of secondary school students.

Future Directions of Research and Limitations

Regulatory focus theory proposed that the adoption of ideal (vs. ought) self-guides and the employment of positive (vs. negative) reference points are important features distinguishing between promotion and prevention orientations (Higgins, 1997, 2012b). However, a corpus of findings (Hodis et al., 2016; Summerville & Roese, 2008), which also includes the results of this study, raises the possibility that centering the conceptualization of regulatory focus on self-guides (vs. reference points) may result in actually measuring two different sets of constructs (i.e., promotion and prevention), “rather than a single phenomenon” (Summerville & Roese, 2008, p. 253). In this light, a pivotal yet unanswered question is whether an overarching construct of promotion (prevention) could be (a) identified by using concomitantly self-guide and reference-point conceptualizations and (b) gauged by means of the combined sets of indicators provided by the RFQ and GRFM. Answering this question has salient implications for both theoretical developments and substantive work. Hence, this is a potentially productive area for future research.

A notable aspect of our study is that the samples providing data for this investigation differed from those in the RFQ’s validation study in terms of developmental age and country of origin. Despite these differences, the instrument performed well in our research. In particular, the items had strong relationships with the constructs they gauged and, consistent with theoretical expectations, the two constructs measured were unrelated. These findings suggest that future studies could assess whether the RFQ works well across an even broader range of developmental ages and in a host of different (English-speaking) countries.

Other instruments that have been developed to measure distinct aspects of regulatory focus include Haws et al.’s (2010) Composite Regulatory Focus Scale and Ouschan, Boldero, Kashima, Wakimoto, and Kashima’s (2007) Regulatory Focus Strategies Scale. Although a few studies examined the extent to which some measures of promotion and prevention overlap (e.g., Haws et al., 2010; Summerville & Roese, 2008), these investigations used data provided by university students. Thus, it is unclear whether their conclusions are tenable for developmentally younger populations (e.g., adolescents); future research is needed to answer this question. In addition, future studies could assess whether, across developmentally distinct populations, the RFQ and other measures of promotion and prevention have differential predictive validity with regard to pivotal cognitive processes involved in learning (e.g., strategic regulation of comprehension) and other key outcomes (e.g., attainment in work and educational settings, commitment).

Although this study provided important fresh knowledge, three limitations need to be acknowledged. First, the data we collected in this research do not afford any causal interpretations of the findings. Second, without access to data from multiple countries, it is not possible to evaluate to what extent our conclusions generalize to other adolescent populations. Finally, in this research, we only collected data from participants in a relatively narrow developmental stage (i.e., secondary students in School Years 10 through 13). This is why, it cannot be assumed that our findings generalize to other developmental periods/stages, even for New Zealand participants. Further research is needed to clarify these aspects.

Footnotes

Acknowledgements

The first author is grateful for the help that Sara Finney, Kevin Grimm, and Keith Widaman provided in earlier stages of this research.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work of the first author was supported by a Fast Start Marsden Grant from Marsden Fund Council, from Government funding, administered by the Royal Society of New Zealand (Contract VUW1210).

References

Arbuckle

J. L.

(1996). Full information in the presence of incomplete data. In Marcoulides

G. A.

Schumacker

R. E.

(Eds.), Advanced structural equation modeling: Issues and techniques (pp. 243-277). Mahwah, NJ: Lawrence Erlbaum.

Bentler

P. M.

(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246. doi:10.1037//0033-2909.107.2.238

Bollen

K. A.

(2000). Modeling strategies: In search of the Holy Grail. Structural Equation Modeling: A Multidisciplinary Journal, 7, 74-81. doi:10.1207/S15328007SEM0701_03

Clark

L. A.

Watson

(1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309-319. doi:10.1037/1040-3590.7.3.309

Curran

P. J.

West

S. G.

Finch

J. F.

(1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1, 16-29. doi:10.1037/1082-989X.1.1.16

Eccles

J. S.

Wigfield

(2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109-132. doi:10.1146/annurev.psych.53.100901.135153

Fan

Wang

(1998). Effects of potential confounding factors on fit indices and parameter estimates on true and misspecified SEM models. Educational and Psychological Measurement, 58, 701-735. doi:10.1177/0013164498058005001

Gorman

C. A.

Meriac

J. P.

Overstreet

B. L.

Apodaca

McIntyre

A. L.

Park

Godbey

J. N.

(2012). A meta-analysis of the regulatory focus nomological network: Work-related antecedents and consequences. Journal of Vocational Behavior, 80, 160-172. doi:10.1016/j.jvb.2011.07.005

Haws

K. L.

Dholakia

U. M.

Bearden

W. O.

(2010). An assessment of chronic regulatory focus measures. Journal of Marketing Research, 47, 967-982. doi:10.1509/jmkr.47.5.967

10.

Higgins

E. T.

(1987). Self-discrepancy: A theory relating self and affect. Psychological Review, 94, 319-340. doi:10.1037/0033-295X.94.3.319

11.

Higgins

E. T.

(1997). Beyond pleasure and pain. American Psychologist, 52, 1280-1300. doi:10.1037/0003-066X.52.12.1280

12.

Higgins

E. T.

(2012a). Accessibility theory. In Van Lange

P. A. M.

Kruglanski

A. W.

Higgins

E. T.

(Eds.), Handbook of theories of social psychology (pp. 75-96). Thousand Oaks, CA: Sage.

13.

Higgins

E. T.

(2012b). Beyond pleasure and pain: How motivation works. New York, NY: Oxford University Press.

14.

Higgins

E. T.

(2015). Regulatory focus theory. In Scott

R. A.

Kosslyn

S. M.

(Eds.), Emerging trends in the social and behavioral sciences (pp. 1-18). Wiley. doi:10.1002/978111900772

15.

Higgins

E. T.

Friedman

R. S.

Harlow

R. E.

Idson

L. C.

Ayduk

O. N.

Taylor

(2001). Achievement orientations from subjective histories of success: Promotion pride versus prevention pride. European Journal of Social Psychology, 31, 3-23. doi:10.1002/ejsp.27

16.

Hodis

F. A.

Hattie

J. A. C.

Hodis

G. M.

(2016). Measuring promotion and prevention orientations of secondary school students: It is more than meets the eye. Measurement and Evaluation in Counseling and Development, 49, 194-206. doi:10.1177/0748175615625750

17.

Hodis

F. A.

Hodis

G. M.

(2015). Expectancy, value, promotion, and prevention: An integrative account of regulatory fit vs. non-fit with student satisfaction in communicating with teachers. In Cohen

E. L.

(Ed.), Communication Yearbook, 39 (pp. 339-370). New York, NY: Routledge.

18.

L.-T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1-55. doi:10.1080/10705519909540118

19.

Lockwood

Jordan

C. H.

Kunda

(2002). Motivation by positive or negative role models: Regulatory focus determines who will best inspire us. Journal of Personality and Social Psychology, 83, 854-864. doi:10.1037/0022-3514.83.4.854

20.

Miele

D. B.

Molden

D. C.

Gardner

W. L.

(2009). Motivated comprehension regulation: Vigilant versus eager metacognitive control. Memory & Cognition, 37, 779-795. doi:10.3758/MC.37.6.779

21.

Molden

D. C.

(2012). Motivated strategies for judgment: How preferences for particular judgment process can affect judgment outcomes. Social & Personality Compass, 6, 156-169. doi:10.1111/j.1751-9004.2011.00424.x

22.

Mueller

R. O.

Hancock

G. R.

(2008). Best practices in structural equation modeling. In Osborne

J. W.

(Ed.), Best practices in quantitative methods (pp. 488-508). Thousand Oaks, CA: Sage.

23.

Muthen

L. K.

Muthen

B. O.

(2010). Mplus, user’s guide (6th ed.). Los Angeles, CA: Author.

24.

Ouschan

Boldero

J. M.

Kashima

Wakimoto

Kashima

E. S.

(2007). Regulatory focus strategies scale: A measure of individual difference in the endorsement of regulatory strategies. Asian Journal of Social Psychology, 10, 243-257. doi:10.1111/j.1467-839X.2007.00233.x

25.

Rosenzweig

E. Q.

Miele

D. B.

(2016). Do you have an opportunity or an obligation to score well? The influence of regulatory focus on academic test performance. Learning and Individual Differences, 45, 114-127. doi:10.1016/j.lindif.2015.12.005

26.

Steiger

J. H.

(1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173-180. doi:10.1207/s15327906mbr2502_4

27.

Summerville

Roese

N. J.

(2008). Self-report measures of individual differences in regulatory focus: A cautionary note. Journal of Research in Personality, 42, 247-254. doi:10.1016/j.jrp.2007.05.005

28.

Tucker

L. R.

Lewis

(1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1-10. doi:10.1007/BF02291170

29.

Widaman

K. F.

Little

T. D.

Preacher

K. J.

Sawalani

G. M.

(2011). On creating and using short forms of scales in secondary research. In Trzesniewski

K. H.

Donnellan

M. B.

Lucas

R. E.

(Eds.), Secondary data analysis: An introduction for psychologists (pp. 39-61). Washington, DC: American Psychological Association.

30.

Wigfield

Muenks

Rosenzweig

E. Q.

(2015). Children’s achievement motivation in school. In Rubie-Davies

C. M.

Stephenson

J. M.

Watson

(Eds.), Routledge international handbook of social psychology of the classroom (pp. 9-20). New York, NY: Routledge.