Precision in Career Motivation Assessment

Subjective Science Attitude Change Measures

Educators are typically concerned with evaluating the scientific and/or technical skills of their students. This can be done by evaluating the product of his or her work (e.g., speed or precision of an engineered machine or judges’ ratings of a project) or simply evaluating their level of knowledge in a given content area. However, a student’s development of skill does not guarantee that he or she will perceive themselves as interested or even competent in science. Subjective ratings of these outcomes are needed to supplement objective measures. The subjective science attitude change measures (SSACM; Stake & Mares, 2001) represent such a collection of rating scales used to assess the impact of science intervention programs.

The SSACM were originally developed for use with high school students and their parents for the purpose of measuring the impact of science program interventions. The SSACM subscales include (a) increased science motivation (ISM), (b) increased science confidence (ISC), (c) increased science knowledge (ISK), and (e) new social niche (NSN; see Table 1). ISK taps instructional effects and NSN measures perceptions of social cohesion, but neither was included in the present study because the former reflects material that is more pertinent to educational researchers, whereas NSN refers to social situations in high school settings. There are two unique features of the SSACM that distinguish them from other science motivation measures. First, they measure changes in science motivation at a proximal level of specificity because their items are designed to tap intervention-specific effects rather than global attitude change. Most researchers use global measures of science attitudes and make inferences about causal change using pretest–posttest designs; unfortunately, positive gains in student science attitudes are difficult to sustain, as scores often return to pretest levels when measured at follow-up (e.g., Stake & Mares, 2005). Second, the SSACM contain a parent version that provides researchers and educators with more objective data regarding student attitude change. The SSACM are therefore ideal for conducting multitrait, multimethod studies that can in turn offer greater insight into the construct validity of SSACM scores. Along these lines, Stake and Mares (2001) obtained evidence of significant positive correlations between parental and student ratings of ISM (r = .52) and ISC (r = .43). Their results also revealed significant positive correlations between SSACM scores on both ISM and ISC and pretest–posttest difference scores on global measures of ISM and ISC.

OPEN IN VIEWER

Table 1

Subjective Science Attitude Change Measures–Student Version.

Increased science motivation. My experiences in the program…

1. Made science seem more interesting to me.

2. Stimulated my enthusiasm for science.

3. Increased my interest in a science career.

4. Made science seem more fun.

5. Clarified for me what I want to do in a science career.

6. Made me feel more sure that I want a career in science.

Increased science confidence. My experiences in the program…

1. Made me feel more relaxed about learning science.

2. Increased my confidence in my ability to do science.

3. Increased my confidence that I can succeed in science as a career.

4. Made me feel more self-assured as a student of science.

5. Made the idea of a science career for me seem more possible.

6. Increased my confidence that I could handle science courses at the college level.

Increased science knowledge.My experiences in the program…

1. Enhanced my knowledge of science.

2. Allowed me to learn science techniques I didn't know before.

3. Gave me “hands-on” experience that will help me in the future with science projects and activities.

4. Gave me a better understanding of what science is all about.

5. Gave me a better grasp of how to write a research paper.

6. Helped me learn problem-solving skills needed in science.

New social niche. My experiences in the program…

1. Helped me see that many other students like science, just as I do.

2. Made me realize that some other people my age really respect and appreciate other students who are good at science.

3. Gave me an opportunity to meet students from other schools who were more enjoyable to be with than many of the students at my high school.

4. Gave me an opportunity to make friends with people who are a lot like me.

5. Made me realize it is okay to be smart and like science, even if some students in my high school don't think so.

Note. From “Science enrichment programs for gifted high school girls and boys: Predictors of program impact on science confidence and motivation”, by J. E. Stake and K. E. Mares, 2001, Journal of Research in Science Teaching, 38, pp. 1065-1088. Reprinted with permission from John Wiley.

Our review of the extant literature resulted in no other research on the SSACM other than the original Stake and Mares (2001) study. This is unfortunate because the SSACM hold considerable promise for use with college student populations. This is particularly true of college freshmen who may be undecided as to whether to declare science as their major area of study. In a similar vein, older students who may be struggling with a weakening commitment to their science major might benefit from the motivational “boost” that an intervention can provide. SSACM scores can thus be used as indices of such adaptive motivational effects or as predictors of student intentions to change from science to nonscience majors. In short, administering the SSACM in the context of a college science course could yield useful information for students, educators, and counselors alike as important career-related decisions such as these are being considered.

The ISM and ISC scales each purport to have one latent factor underlying their respective items but this is speculative because Stake and Mares (2001) did not report any factor structures. At face value, our reading of the items suggests there may be three distinct factors underlying the 12 items. The ISM scale contains items that appear to measure the instantiation of science interest in that they reflect students’ phenomenological experiences of interest (e.g., “Made science seem more interesting to me” and “Stimulated my enthusiasm for science”). Although individuals often feel excitement or pleasure when they are interested in something, positive affect does not define interest alone. Interest development also involves collating information about the environment and comparing these features against schemas of related experiences (Berlyne, 1960; Durik & Harackiewicz, 2007). Prominent among these features is novelty, as tasks, objects, and experiences that are incongruent with established schemas are thought to stimulate emotional arousal and curiosity (Silvia, 2005). Scherer (2001) referred to this type of appraisal as a novelty check. It is important to highlight the importance of novelty because educational theory and research (e.g., Brophy, 1999, 2008) consistently suggest that teaching material in ways that are novel and creative increases student task engagement. The experience of interest thus involves both cognitive and affective components, directs students’ attention during task engagement, and even facilitates cognitive resource replenishment needed for sustaining motivated task engagement (Sansone & Smith, 2000; Sansone & Thoman, 2005; Silvia, 2008; Thoman, Smith, & Silvia, 2011). Given that various ISM items seem to tap into aspects of the experience of interest, we refer to the factor that is defined by these items as intrinsic science interest (ISI) in the present study.

Several items clearly map onto ISC (e.g., “Increased my confidence in my ability to do science”), which we would subsume under the hypothesized factor science self-efficacy (SSE), ¹ but there are also items from both of the original scales that reflect a mixture of science interest and SSE. Some of these items explicitly measure interest development, whereas others clearly measure SSE, but the element that unifies the ISM and ISC scales appears to be an orienting of the individual to distal outcomes. Items that measure increased career focus (e.g., “Clarified for me what I want to do in a science career”) and goal attainability (e.g., “Made the idea of a science career for me seem more possible”) represent an orientation toward the future, which is thought to be a key determinant of vocational identity (Diemer & Blustein, 2007; Marko & Savickas, 1998; Super, 1980). Short- and long-term (e.g., career) goals are brought into view because skill development leads to the generation of both efficacy percepts and expectancies of positive future outcomes (Lent, Brown, & Hackett, 1994). The synergistic effects of real-time performance feedback, greater confidence in the moment (Nauta & Kahn, 2007), and adaptive expectancies should foster identity development by bridging one’s sense of current self with one’s sense of future self. It is this cognitive consistency in attitudes—from perceptions of present self as a science student to projections of future self as potential scientist—that are thought to contribute to the formation of a firm science identity. We would thus label the latent variable that gives rise to these scores as science career identity (SCI). There are a number of items on the ISM and ISC scales that appear to cohere around this idea, and we explore this possibility in the present study.

Present Study

The primary purpose of the present study was to evaluate the factor structure of the ISM and ISC scales of the SSACM using a college sample of students enrolled in science classes. SSE, interest, and identity are all distinct constructs because they are regulated by slightly different psychological systems (e.g., affective vs. cognitive) and from different temporal orientations. A good science intervention program should address all three by triggering interest and competence perceptions, and initiating the process of integrating interest and self-efficacy into one’s self-concept. We believe that science motivation in its entirety cannot exist independent of these motivational influences. We tested this proposition by conducting a bifactor analysis, whereby ISM and ISC items were allowed to load simultaneously on a general motivation factor and specific latent constructs. Bifactor models are similar to higher order models in that they possess nested structures and are, in fact, mathematically equivalent when proportionality constraints are imposed (Brunner, Nagy, & Wilhelm, 2012). However, bifactor models are thought to be preferable when specific factors are the focus of attention, and the researcher wishes to link both general and specific factors to external variables (Chen, West, & Sousa, 2006). Such is the case in the present study, as we were interested in the extent to which ISI, SSE, and SCI explain variation in relevant outcomes over and above a general factor.

Thus, a second purpose of this study was to evaluate the concurrent validity of the specific factor scores. Four outcomes were chosen for these analyses: (a) absolute task mastery (ATM) motivation for research; (b) intrinsic science motivation; (c) intention to conduct undergraduate research; and (d) science identity. If college students are highly motivated to understand and develop skill in science, then it is reasonable to suspect that they would also be motivated to develop their skills in formal research endeavors. ATM motivation is thought to be a logical outcome of science motivation because involvement in formal research is a necessary step for students to take if they are to pursue scientific careers. Global science motivation was thus hypothesized to be a positive predictor of ATM motivation. Similarly, interest is considered a psychological state resulting from an initial or recurring engagement with an object or activity (Renninger, 2000). The potency of the attention and affect generated by engagement with the task determines whether interest is internalized into the self as a need for competence (science competence in the present case), which is a key component of self-determination theory (SDT; Deci & Ryan, 1985; Ryan & Deci, 2000). ISI was therefore hypothesized to be a significant positive predictor of intrinsic science motivation. Finally, SCI and SSE were hypothesized to be significant positive predictors of general forms of science identity and science confidence, respectively.

The specific factor scores were presumed to demonstrate these concurrent relations with outcomes while statistically controlling for potentially confounding predictors. Because there is a significant association between parental involvement and their children’s math/science career goals (Byars-Winston & Fouad, 2008), we modeled parental occupational type (STEM vs. non-STEM) as a predictor of science motivation. We considered the possibility that variation in the validity analysis outcomes could be accounted for by a general tendency to approach the science lab class with an aim of mastering course content; therefore, mastery motivation was estimated as a statistical control variable. Gender was also included as a covariate, given that women tend to be subject to motivational deficits, particularly lower self-efficacy (e.g., Deemer, Thoman, Chase, & Smith, in press; Rice, Lopez, & Richardson, 2013), owing in large part to environmental factors.

Method

Results

Exploratory Bifactor Analysis

A model with all indicators specified to load on a single first-order factor would represent a logical rival hypothesis to a bifactor model, given that bifactor models are partly defined by general factors, but one-factor models cannot be rotated in EFA. We submitted such a model to CFA using a randomly drawn subset (n = 823) of the overall sample. Although the standardized factor loadings were quite strong, ranging from 0.73 to 0.92, the fit of the model was poor, χ²(54, N = 823) = 1,512.35, p = .00, CFI = 0.88, TLI = 0.86, RMSEA = 0.18 (90% CI: [0.17, 0.19]), SRMR = .04. Having ruled out this model as a viable alternative, we proceeded to perform an EFA using an ESEM approach (Asparouhov & Muthén, 2009). The hypothesized bifactor model was fitted and compared to a number of alternative models: (a) a bifactor model with one general and two specific motivation and confidence factors; (b) a two-factor model reflecting the original SSACM structure; and (c) a three-factor model that included separate SSE, ISI, and SCI factors. A one-factor model could not be tested, given that multiple factors are needed for rotation. Orthogonal rotation methods were used, given that general and specific factors in the bifactor framework are assumed to be uncorrelated (Chen et al., 2006). Geomin rotation was used for the specific-factor models and bigeomin rotation was used for the bifactor models. Fit indices used to assess model fit included the (a) model chi-square test, (b) comparative fit index (CFI), (c) Tucker-Lewis index (TLI), (d) standardized mean square error of approximation (RMSEA), and (d) standardized root mean square residual (SRMR).

Fit statistics are reported in Table 2. Fit index values for the two-factor model were in the acceptable range with the exception of RMSEA, the value for which exceeded the 0.10 threshold for marginal fit (Browne & Cudeck, 1993). The three-factor and alternative bifactor models offered a better fit to the data as indicated by CFI, TLI, and SRMR values, but RMSEA values were still in the marginal range. The alternative bifactor model was statistically equivalent to the three-factor model in terms of the fit statistics but yielded a much different pattern of factor loadings due to the bigeomin rotation (see Table 3). The hypothesized bifactor model produced a more acceptable RMSEA value of 0.06, and standardized loadings for the general factor were slightly higher (mean λ = .865) as compared to general factor loadings in the alternative bifactor model (mean λ = .860). SSACM items were not originally intended to be reverse scored, thus rendering negative factor loadings difficult to interpret. For this reason, only positive factor loadings in the hypothesized model were given consideration to comprise a potential subscale. Specific factor loadings were all fairly low, but this is to be expected as specific factors account for only score variance not accounted for by general factors (Brunner et al., 2012). The following items were therefore identified as producing the highest loadings: (a) Items 1, 2, 3, and 7 for ISI; (b) Items 5, 9, 10, and 11 for SCI; and (c) Items 8 and 12 for SSE. The general factor scale was labeled simply as SSACM Motivation (SSACMM) to refer to a global science motive collectively defined by interest, identity, and self-efficacy. Factor scores were saved from the bifactor analysis and used as predictor variables in the concurrent validity analysis.

OPEN IN VIEWER

Table 2

Summary of ESEM-EFA Model Fit Statistics.

Model	χ2	df	CFI	TLI	RMSEA	90% CI for RMSEA	SRMR
Two factor	594.29	43	.96	.93	.12	[.12, .13]	.02
Three factor	271.23	33	.98	.96	.09	[.08, .10]	.01
Alternative bifactor	271.23	33	.98	.96	.09	[.08, .10]	.01
Hypothesized bifactor	85.61	24	.99	.99	.06	[.04, .07]	.01

Note. CFI = comparative fit index; CI = confidence interval; EFA = exploratory factor analysis (EFA); ESEM = exploratory structural equation modeling; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual; TLI = Tucker-Lewis index.

OPEN IN VIEWER

Table 3

Standardized Factor Loadings for ESEM EFA Model.

	Two-Factor Model		Three-Factor Model			Alternative Bifactor Model			Hypothesized Bifactor Model
Item	MOT	CONF	ISI	SCI	SSE	GF	MOT	CONF	MOT	ISI	SCI	SSE
SSACM1	.87	−.00	1.01	.00	−.15	.83	.38	−.02	.84	.33	−.04	−.00
SSACM2	.96	−.10	.77	−.02	.18	.86	.19	−.13	.86	.15	−.14	.01
SSACM3	.86	.08	.86	.12	−.01	.90	.28	−.01	.91	.24	−.04	−.08
SSACM4	.98	−.08	.54	.01	.46	.91	.01	−.20	.91	−.04	−.22	−.01
SSACM5	.52	.42	.45	.48	.02	.87	.12	.17	.89	.07	.14	−.12
SSACM6	.65	.30	.27	.40	.35	.92	−.06	.03	.92	−.12	.00	−.06
SSACM7	.80	.08	.76	.13	.00	.84	.24	.00	.84	.27	.01	.21
SSACM8	.77	.16	.41	.26	.34	.92	.00	−.03	.92	−.02	−.04	.32
SSACM9	−.01	.83	.00	.88	−.10	.71	.00	.42	.71	−.00	.41	.02
SSACM10	.19	.77	.02	.85	.09	.87	−.06	.34	.87	−.08	.33	−.00
SSACM11	.01	.95	−.01	1.03	−.10	.83	−.01	.49	.83	−.02	.47	−.02
SSACM12	.51	.36	−.00	.49	.48	.85	−.19	.04	.83	−.19	.06	.11

Note. SSACM = subjective science attitude change measure; MOT = motivation; CONF = confidence; ISI = intrinsic science interest; SCI = science career identity; SSE = science self-efficacy; GF = general factor.

Hierarchical Regression Analyses

To evaluate the concurrent validity of the SSACM scores, we conducted a series of hierarchical regression analyses using a subsample of 458 participants. Means, SDs, and correlations for the analysis variables are reported in Table 4. Null correlations among the factor scores (rs ranged from –.06 to .06) reflect the orthogonality of the factors. Gender, parent occupation type, and mastery motivation were included as covariates on Step 1. Only individual factor scores were entered on the second step of the hierarchical regressions in order to maintain consistency with the orthogonal structure of the bifactor model (see Table 5). Regressing ATM motivation on Step 1 of the first analysis resulted in 20% of the variance accounted for by the predictors, F(4, 435) = 27.52, p < .001, while SSACMM explained an additional 12.1% of the variance, ΔF(1, 434) = 77.69, p < .001. Mothers in STEM occupations (β = .10, p = .03) and mastery approach goals (β = .44, p < .001) were significant positive predictors of ATM motivation on Step 1, but the former coefficient was reduced to nonsignificance (β = .06, p = .15) when SSACMM was added to the regression equation (β = .38, p < .001). The Step 1 model of the second analysis was significant, F(4, 453) = 45.07, p < .001, with the collection of predictors accounting for 28.5% of the variance in intrinsic science motivation. Only mastery motivation was found to be a significant predictor of intrinsic science motivation (β = .53, p < .001), whereas mothers in STEM (β = .07, p = .10) and gender (β = −.06, p = .13) were marginally nonsignificant. ISI accounted for a unique 0.8% of the variance in intrinsic science motivation on Step 2, ΔF(1, 452) = 5.10, p = .02 (β = .09).

OPEN IN VIEWER

Table 4

Zero-Order Correlations and Descriptive Statistics for the Regression Analysis Variables.

Variable	1	2	3	4	5	6	7	8	9
1. ATMM	—
2. SCI	.21***	—
3. ISI	.06	−.02	—
4. ISM	.65***	.15**	.12**	—
5. MAP	.44***	.07	.03	.51***	—
6. CLS	.49***	.02	−.07	.68***	.46***	—
7. SI	.55***	.27***	−.01	.67***	.39***	.54***	—
8. SSACMM	.50***	.02	.03	.60***	.40***	.55***	.53***	—
9. SSE	.02	−.06	−.06	.01	−.04	.03	.00	.05	—
10. M	3.79	.00	.00	3.86	4.14	3.88	5.23	.00	.00
11. SD	.76	.88	.83	.73	.76	.71	1.14	.99	.81

Note. ATMM = absolute task mastery motivation; SCI = science career identity; ISI = intrinsic science interest; ISM = intrinsic science motivation; MAP = mastery approach goal; CLS = confidence learning science; SI = science identity; SSACMM = subjective science attitude change measures motivation; SSE = science self-efficacy.

*p < .05. **p < .01. ***p < .001.

OPEN IN VIEWER

Table 5

Results of Hierarchical Regression Analyses Predicting Concurrent Validity Outcomes.

	ATM Motivation					Intrinsic Science Motivation					Science Identity					Confidence Learning Science
Variable	b	SE	β	R 2	Δ R 2	b	SE	β	R 2	ΔR 2	b	SE	β	R 2	ΔR 2	b	SE	β	R 2	ΔR 2
Step 1				.202**					.285**					.159**					.215**
Gender	−.27	.39	−.03			−.45	.29	−.06			−.08	.60	−.01			−.13	.30	−.02
MOCC	.98	.45	.10*			.55	.34	.07			.61	.70	.04			.31	.35	.04
FOCC	−.12	.34	−.02			.16	.25	.03			−.47	.52	−.04			.11	.26	.02
MAP	.86	.08	.45**			.83	.06	.53**			1.18	.13	.40**			.71	.06	.46**
Step 2				.323**	.121**				.293**	.008*				.215**	.056**				.218**	.004
Gender	.00	.36	.00			−.47	.29	−.07			.12	.58	.01			−.11	.30	−.02
MOCC	.60	.42	.06			.52	.34	.06			.52	.68	.03			.29	.35	.04
FOCC	−.03	.31	−.00			.17	.25	.03			−.46	.51	−.04			.11	.26	.02
MAP	.56	.08	.29**			.83	.06	.53**			1.14	.13	.38**			.71	.06	.47**
SSACMM	1.75	.20	.38**			—	—	—			—	—	—			—	—	—
ISI	—	—	—			.38	.17	.09*			—	—	—			—	—	—
SCI	—	—	—			—	—	—			1.80	.32	.24**			—	—	—
SSE	—	—	—			—	—	—			—	—	—			.26	.18	.06

Note. ATM = absolute task mastery; FOCC = father's occupation; ISI = intrinsic science interest; MAP = mastery approach goal; MOCC = mother's occupation; SCI = science career identity; SSACMM = subjective science attitude change measures motivation (general factor); SSE = science self-efficacy.

*p < .05. **p < .001.

In the third analysis, science identity was regressed on the set of statistical control variables on Step 1 and SCI on Step 2. The Step 1 model fit the data well, F(4, 445) = 21.04, p < .001, with the covariates accounting for 15.9% of the variance in science identity. Mastery approach goal was the only significant predictor in the Step 1 model (β = .40, p < .001). The variance explained in science identity increased by 5.6% when SCI was added to the equation on Step 2, ΔF(1, 444) = 31.42, p < .001 (β = .24). Finally, the Step 1 model for predicting CLS was significant, F(4, 454) = 31.02, p < .001, as the coefficient for mastery approach goal was significant (β = .46, p < .001), and the entire set of covariates explained 21.5% of the variance. Contrary to expectation, SSE was found to be a nonsignificant predictor of CLS (β = .06) as the variable accounted for only 0.4% of the variation on Step 2, ΔF(1, 453) = 2.08, p = .15.

Discussion

The current study examined the psychometric properties of the motivation-relevant scales of the SSACM using a college sample of science students. Results revealed that these scales possess a bifactor structure and are predictive of theoretically similar constructs. Results of the ESEM analysis indicated that a bifactor model with one general factor and three specific factors outperformed several alternative models, including a bifactor model consisting of a general factor and the two original scales in question—ISM and ISC. Previous research on the validity of SSACM scores had also been severely lacking; therefore, our study addressed a problematic oversight in the literature. Specifically, the hypothesized general and specific SSACM variables were predictive of such outcomes as mastery motivation for research, intention to conduct undergraduate research, and science identity.

Results of the bifactor analysis indicated that all 12 of the items examined loaded rather impressively onto a general science motivation factor, while separate groups of items demonstrated different patterns of loading onto the three hypothesized specific factors. Stake and Mares’ (2001) original conceptualization of ISM called for the measurement of both science identity and science interest in one scale, but our findings suggest that the two should be decomposed into separate constructs. Interest can be fleeting, particularly situational interest because its formation is dependent upon often mutable factors within the external environment (Hidi & Renninger, 2006; Krapp, 2002). In contrast, our results show that SCI can be thought of as a stable aspect of one’s self-concept that is essentially a by-product of the synthesis of interest, goals, and self-efficacy (Eccles, 2009; Osborne & Jones, 2011). The fact that the specific factor loadings for SCI were stronger than the loadings for ISI and SSE fits with the “by-product” conceptualization of SCI because while SCI was found to be an important contributor to general science motivation, it is an entity that is at the same time more independent of motivation than ISI and SSE are. This would suggest that SCI should be a stronger predictor of theoretically related outcomes than either ISI or SSE, and this was found to be the case as the hierarchical regression analyses indicated that SCI predicted science identity more strongly than ISI and SSE predicted their respective outcomes.

Contrary to expectation, SSE was not a significant predictor of CLS. Recall, however, that in bifactor analysis most of the variance in item scores is explained by the general factor, while the remaining variance is accounted for by specific factors and measurement error. This partitioning of variance is thus not unlike the progressive decline in the magnitude of eigenvalues with each factor extracted in traditional EFA. It seems that most of the variance in SSACM scores was extracted by the SSACMM, SCI, and ISI factors and very little extracted by SSE. Moreover, only 2 items loaded meaningfully on SSE, whereas 2 items that were originally theorized by Stake and Mares (2001) to reflect ISC—“Increased my confidence in my ability to do science” and “Increased my confidence that I can succeed in science as a career”—loaded only on SSACMM. One possible reason why SSE extracted the smallest proportion of variance is that one science laboratory course may not be sufficient to make a pronounced impact on one’s confidence. Students may report more interest in science after having taken a class in it, but this does not mean that they will necessarily feel more confident in their science ability. Indeed, it is more likely that students need repeated exposure to science tasks to render the requisite mastery experiences (Bandura, 1997) needed for self-efficacy to take shape.

Conclusion

The present study yielded evidence that the original ISC and ISM scales of the SSACM possess a bifactor structure with all items demonstrating strong convergence on a general science motivation factor yet loading equally well on specific competence-related constructs. Of course, the SSACM were originally developed with the intent of measuring attitude change in adolescents (Stake & Mares, 2001), therefore, we encourage other researchers to test this bifactor structure in this population as well. As noted at the outset of this article, STEM careers are in high demand and will continue to be for the foreseeable future. Perhaps the process of guiding students into and staying with a STEM career path can be facilitated to the extent that intervention programs are implemented on the basis of precise attitude measurement.