Do Data-Driven Decision-Making Efficacy and Anxiety Inventory Scores Mean the Same Thing for Pre-Service and In-Service Teachers?

Abstract

Teacher data-driven decision making (DDDM) is a professional practice of great prominence in the current K-12 education system. Moreover, teacher self-efficacy and anxiety around DDDM represent important measurement targets in both research and practice. This study consequently examined the validity, reliability, and invariance of data collected via the Data-Driven Decision-Making Efficacy and Anxiety (3D-MEA) Inventory among U.S. in-service (N = 365) and pre-service (N = 457) teachers. The 3D-MEA is intended to measure four dimensions of self-efficacy related to DDDM, as well as anxiety related to DDDM. Multi-group confirmatory factor analyses established structural measurement invariance, which supports meaningful and effectively interchangeable interpretation of 3D-MEA scores with both in-service and pre-service teachers. Reliability estimates were also high in both populations. Limitations and future directions are discussed.

Keywords

data-driven decision making in-service teachers pre-service teachers validity invariance

At the same time as the world’s technological capacity to gather, store, and share information grew exponentially, the federal No Child Left Behind (NCLB) legislation greatly expanded the collection of academic data on students in the U.S. (NCLB, 2002). Both the culture of accountability in education, and increased access to data of many kinds, have proliferated also in response to other federal legislations and initiatives, including the Individuals with Disabilities Education Improvement Act (IDEIA, 2004), the American Recovery and Reinvestment Act (ARRA, 2009), the Race to the Top competition, and the Every Student Succeeds Act (ESSA, 2015). While technology and policy have enabled or prioritized examination and use of education data, the day-to-day responsibility for analyzing, interpreting, and using these data to make decisions that improve teaching and learning has predominantly fallen on classroom teachers (Mandinach & Gummer, 2016).

Unfortunately, teachers’ data literacy and capacity to engage in data-driven decision making (DDDM) has not necessarily kept pace with the growth in education data technology or access to education data themselves (Means et al., 2010, 2011). Online diagnostic assessments, student learning and intervention platforms, extant data warehouses, standards-based official assessments, and the like provide diverse, immense, and sometimes continuous streams of data to educators at different time points and on different time scales. The COVID-19 pandemic, and resultant use of new digital systems and tools for remote learning which often produce their own data as well, have further amplified the need for teachers to be proficient in using data to adapt their teaching to the needs of their students (Dorn et al., 2020; Kaffenberger, 2021).

Prior research indicates that effective and responsible engagement in DDDM is difficult for both in-service teachers (i.e., practicing teachers) as well as pre-service teachers (i.e., those training to become teachers) (Dunn et al., 2013; Wayman & Jimerson, 2014). Two noted factors that can influence how effectively educators use data to make decisions are teacher self-efficacy for DDDM and teacher anxiety for DDDM. Teacher self-efficacy for DDDM is a teacher’s belief in their ability to use DDDM in their classroom, and teacher anxiety for DDDM is a teacher’s stress and anxiety induced by engagement in this practice (Authors, 2016; Dunn et al., 2013). Given the challenges of in-service and pre-service teacher self-efficacy and anxiety around DDDM, and continued emphasis on this practice, proper measurement of these constructs in both of these populations is thus essential in research and practice.

Dunn et al. (2013) developed the Data-Driven Decision-Making Efficacy and Anxiety (3D-MEA) Inventory as a self-report instrument to measure these important constructs and their sub-domains using 20 items and an agreement response format. Past research on the 3D-MEA found strong evidence for the quality of the instrument with each pre-service teachers and in-service teachers (Authors, 2018; 2020; Dunn et al., 2013). While evidence from separate evaluations of the 3D-MEA with these two populations is highly favorable, the current study more formally investigates the functioning of the instrument across pre-service teachers and in-service teachers. It is possible that the 3D-MEA may not function equivalently for in-service teachers and pre-service teachers given that the former as a whole are definitionally more experienced in the education field, whereas the latter are novices by definition; such differences in experience might affect these groups’ respective interpretations of and thus responses to 3D-MEA items. Also, as noted by Authors (2020), several 3D-MEA items reference the respondent’s “district,” which may complicate the item response process and introduce measurement error for respondents who have no district affiliation, including pre-service teachers.

Accordingly, the main purpose of this study is to provide evidence of the reliability, validity, and invariance of 3D-MEA scores among pre-service teachers and in-service teachers. The present study, particularly, addressed the following research questions: (1) To what extent do 3D-MEA inventory scores exhibit the same factorial measurement structure for pre-service teachers and in-service teachers? (2) To what extent do 3D-MEA inventory scores exhibit internal consistency (reliability) among both pre-service teachers and in-service teachers?

The ability to estimate self-efficacy or anxiety related to DDDM accurately among both pre-service and in-service teachers is critical in both research and practice. These estimates can instrumentally inform decisions about the scope or nature of learning opportunities, scaffolds, or supports to avail to either of these populations to advance their DDDM capacity. There are also situations in which a measure of such constructs might be used simultaneously with both pre-service and in-service teachers, necessitating cross-population invariance in the functioning of the instrument. Researchers or teacher educators may be interested in comparing these two populations vis-a-vis their DDDM self-efficacy or anxiety or sub-domains thereof. For example, researchers may ask whether traditionally younger, pre-service teachers have higher self-efficacy with respect to technology-related DDDM tasks than in-service teachers. Should we want to well understand such differences, the instrument needs to be psychometrically invariant for these groups. Others might be interested in tracking developmental changes in DDDM self-efficacy or anxiety during the transition from pre-service teacher to in-service teacher, which also necessitates measurement invariance. The establishment of measurement invariance would help allay concerns in any such studies that observed differences or changes were not actually real but instead artifactual.

Theoretical Framework and Literature Review

The Nature of Self-Efficacy and Anxiety

Research and theory on the nature of self-efficacy and anxiety in general—and self-efficacy and anxiety related to teaching and data-driven decision making in particular—informed our analysis. Bandura’s (1977) social cognitive theory was initial basis for the development of the 3D-MEA instrument (Dunn et al., 2013). Bandura’s (1977, 1994) theory described a cognitive system wherein one’s ability to be successful at new tasks is partly determined by their self-efficacy and anxiety for those tasks.

Teacher self-efficacy beliefs generally were defined by Tschannen-Moran and Hoy (2001, p. 783) as “a judgement of his or her capabilities to bring about desired outcomes of student engagement and learning.” Research indicates that teachers with a strong sense of self-efficacy are more likely to implement new practices (such as DDDM) than teachers with more limited self-efficacy (Guskey, 1988). Conversely, concerns, feelings of anxiety, and stress can hinder a teacher’s ability to take up new innovations or implement new practices. Because of the significant impact they can have on teacher practices, researchers of teacher practices, including DDDM practices, remain invested in exploring task-specific teacher self-efficacy and anxiety.

The earliest research on the 3D-MEA instrument aided researchers to understand the nature—and multi-dimensional structure—of the self-efficacy for DDDM construct. Early factor analytic studies of the 3D-MEA helped theorize DDDM self-efficacy as comprising four task-specific self-efficacy dimensions: self-efficacy for data identification and access, self-efficacy for data technology use, self-efficacy for data analysis and interpretation, and self-efficacy for application of data to instruction. The 3D-MEA also purportedly assesses a stand-alone DDDM anxiety dimension, which is the stress a teacher feels while attempting to engage in DDDM practices (Dunn et al., 2013). As such, the 3D-MEA provides a nuanced profile of psychosocial factors that can influence engagement in DDDM practices. Table 1 summarizes the latent variables for which the 3D-MEA elicits evidence and their definitions.

Table 1.

Definitions of data-driven decision-making efficacy and anxiety latent variables.

Latent variable	Definition
DDDM self-efficacy	—
Self-efficacy for data identification and access	A teacher’s belief in their ability to find and collect quality student data
Self-efficacy for data technology use	A teacher’s belief in their ability to use district data technology resources
Self-efficacy for data analysis and interpretation	A teacher’s belief in their ability to examine data for patterns of performance
Self-efficacy for application of data to instruction	A teacher’s belief in their ability to use data to inform subsequent strategies to help students reach their learning goals
DDDM anxiety	A teacher’s stress or trepidation felt while attempting to engage in DDDM practices

Note. DDDM = data-driven decision making.

Data-Driven Decision-Making Efficacy and Anxiety Reliability and Validity Evidence

Prior evidence on the psychometric properties of the 3D-MEA also informed this research. Four published papers have specifically examined the psychometric properties of the data derived from the 3D-MEA. In general, these three studies emphasized internal consistency-related reliability evidence and internal score structure-based validity evidence. Several smaller-scale substantive studies have also reported data that technically constitutes or may be considered reliability and/or validity evidence (e.g., Authors, 2017; Green et al., 2016), but given their small scale, we emphasize in this section, the four larger-scale studies specifically aimed at investigating the 3D-MEA vis-à-vis score reliability and validity.

Dunn et al. (2013) reported considerable reliability and validity evidence for the 3D-MEA in the paper in which the instrument was first published. In this first psychometric evaluation of the 3D-MEA, the authors used data from 1728 in-service teachers in the U.S. pacific northwest region. They first used exploratory factor analysis, and cross-validated findings in a separate sample using confirmatory factor analysis, concluding that a five-factor model (with four self-efficacy dimensions and one anxiety dimension) well accounted for the observed 3D-MEA data. Authors (2018) then independently conducted a confirmatory factor analysis of 3D-MEA data collected from 365 Illinois in-service teachers, finding a good fit of the same five-factor model and high score reliability.

Two groups of researchers have also formally investigated research questions about 3D-MEA score validity and reliability with pre-service teachers. Dunn et al. (2020) used exploratory factor analysis to examine the internal score structure of 3D-MEA data collected from 602 pre-service teachers over multiple semesters from a course in a single teacher-education institution. The authors hypothesized that given the relative inexperience of pre-service teachers with DDDM would likely mean that self-efficacy would be a unidimensional construct for this population (rather than four separate DDDM self-efficacy dimensions) and this is indeed what they found.

Around the same time, Authors (2020) used confirmatory factor analysis to test the original five-factor structure of 3D-MEA scores with a somewhat smaller, but still adequately sized, and potentially much more heterogeneous sample of pre-service teachers. The authors found an excellent fit of the five-factor model in both training and validation sub-samples in their study. While these findings diverge from those of Dunn et al. (2020), this may be due to differences in sampling. Authors’ (2020) study featured pre-service teachers from at least 38 states and 132 pre-service institutions, and thus may have provided a more heterogeneous sample for testing the 3D-MEAs score structure.

The findings of prior studies have provided favorable evidence for the hypothesized 3D-MEA score structure and score reliability among pre-service teachers and in-service teachers. Despite these separate 3D-MEA psychometric evaluations with these two populations, however, the literature lacks a formal test of the 3D-MEA scores’ invariance across these two important populations with which it might be used jointly (e.g., to compare, for measurement of change). Consequently, the present study aimed to provide evidence of the reliability, validity, and invariance of 3D-MEA scores across both pre-service teachers and in-service teachers.

Method

Data Source

The present study secondarily analyzed data collected from pre-service and in-service teachers between Spring 2014 and Spring 2018. We leveraged data gathered in the context of a research program related to pre- and in-service teachers’ data literacy, including various survey research, intervention, and psychometric studies (e.g., Authors, 2018, 2020). Some of the intervention studies during which data were gathered involved both pretest and posttest data collection, and when data from both time points were available, the relevant pretest data were used.¹ All data were collected via Qualtrics survey software. The authors obtained Institutional Review Board for all studies from which these data were sourced, as well as the present secondary analysis.

Instrumentation

The 3D-MEA instrument (Dunn et al., 2013) consists of 20 items intended to measure five affective dimensions related to DDDM. This instrument intends to measure four subdimensions of self-efficacy related to data-driven decision-making—self-efficacy for data identification access, self-efficacy for data technology use, self-efficacy for data analysis and interpretation, and self-efficacy for the application of data to instruction. Self-efficacy for data identification access, self-efficacy for data technology use, and self-efficacy for data analysis and interpretation are each operationalized with three items, self-efficacy for the application of data to instruction dimension is operationalized with six items, and the anxiety dimension is operationalized with five items. The response format for these 20 items was a 5-point agreement rating scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither disagree nor disagree, 4 = Agree, and 5 = Strongly agree. Table 2 shows all items and Figure 1 provides a graphical representation of the fitted confirmatory factor model with five factors.

Table 2.

Item descriptive statistics.

Items	PST (N = 457)				IST (N = 365)
Items	Mean (M)	Std. Deviation (SD)	Skewness (S₃)	Kurtosis (K₄)	Mean (M)	Std. Deviation (SD)	Skewness (S₃)	Kurtosis (K₄)
I am confident in my ability to access state assessment results for my students	3.21	1.03	−0.25	2.16	3.27	1.16	−0.43	2.35
I am confident that I know what types of data or reports I need to assess group performance	3.18	0.98	−0.21	2.08	3.56	1.00	−0.73	3.08
I am confident that I know what types of data or reports I need to assess student performance	3.37	0.94	−0.43	2.41	3.71	0.98	−0.89	3.43
I am confident I can use the tools provided by my district’s data technology system to retrieve charts, tables or graphs for analysis	3.37	0.97	−0.37	2.52	3.4	1.17	−0.36	2.17
I am confident I can use the tools provided by my district’s data technology system to filter students into different groups for analysis	3.32	0.97	−0.34	2.39	3.23	1.17	−0.23	2.08
I am confident that I can use my district’s data analysis technology to access standard reports	3.26	0.97	−0.22	2.33	3.34	1.12	−0.36	2.23
I am confident in my ability to understand assessment reports	3.58	0.89	−0.62	2.86	3.78	0.96	−0.94	3.63
I am confident in my ability to interpret student performance from a scaled score	3.64	0.84	−0.65	2.89	3.81	0.91	−0.78	3.32
I am confident in my ability to interpret subtest or strand scores to determine student strengths and weaknesses in a content area	3.55	0.85	−0.61	2.83	3.66	1.01	−0.83	3.21
I am confident that I can use data to identify students with special learning needs	3.5	0.92	−0.56	2.75	3.85	0.95	−0.74	2.98
I am confident that I can use data to identify gaps in student understanding of curricular concepts	3.65	0.82	−0.77	3.30	3.81	0.86	−0.79	3.64
I am confident that I can use assessment data to provide targeted feedback to students about their performance or progress	3.75	0.76	−0.78	3.80	3.84	0.87	−1.05	4.44
I am confident I can use assessment data to identify gaps in my instructional curriculum	3.65	0.8	−0.63	2.93	3.77	0.85	−0.86	3.89
I am confident that I can use data to group students with similar learning needs for instruction	3.83	0.75	−0.85	3.97	3.92	0.81	−1.04	4.82
I am confident in my ability to use data to guide my selection of targeted interventions for gaps in student understanding	3.53	0.83	−0.52	2.63	3.71	0.87	−0.67	3.38
I am intimidated by statistics	2.91	1.09	0.10	2.16	2.52	1.10	0.49	2.54
I am intimidated by the task of interpreting students’ state level standardized assessments	3.05	1.01	−0.11	2.23	2.69	1.08	0.25	2.35
I am concerned that I will feel or look ‘‘dumb’’ when it comes to data driven decision-making	3.02	1.11	−0.11	2.11	2.42	1.05	0.34	2.15
I am intimidated by my district’s data retrieval technology	2.94	0.89	−0.09	2.99	2.51	1.01	0.25	2.48
I am intimidated by the process of connecting data analysis to my instructional practice	3.03	0.96	−0.13	2.21	2.48	1.04	0.30	2.24

Notes. PST = pre-service teachers. IST = in-service teachers. Response format was: 1 = Strongly disagree, 2 = Disagree, 3 = Neither disagree nor disagree, 4 = Agree, and 5 = Strongly agree.

Figure 1.

Fitted confirmatory factor analytic model. Notes. Graphical representation of the five-factor model estimated from model 4. The confirmatory factor model featured five inter-correlated latent variables, each inferred by between three to six indicator variables.

Participants

The present study focused on 3D-MEA score properties in two populations: pre-service teachers and in-service teachers. The pre-service teacher sample comprised 457 pre-service teachers from 132 different teacher-education institutions and 38 different states in the U.S. Based on available data, the pre-service teacher sample was 92% female and 8% male. The pre-service participants reported the following race/ethnicity distribution: 87% White, 5% two or more races, 4% Black or African American, 3% Asian, 1% American Indian or Alaska Native, and 0.23% Native Hawaiian or other Pacific Island; and 10% identified as Hispanic/Latino. The mean age of the pre-service teachers was 24.89 years old (SD = 6.78). The pre-service teachers reported that they intended to teach in various school levels: 86% grades K-5 elementary, 5% grades 6–8 middle school, 5% grades 9–12 high school, 4% multiple school levels, and 0.44% pre-kindergarten. A large majority of the pre-service teacher sample reported intending to teach elementary students.

The in-service teacher sample comprised 365 participants from 76 different schools in one Midwestern U.S. state (Illinois). Based on available data, the in-service teachers’ gender distribution was as follows: 79% female, 21% male. The in-service teachers reported their races and ethnicities as 98% White, 1% two or more races, 1% Asian, 0.33% American Indian or Alaska Native, and 0.33% Black or African American; and 3% identified as Hispanic/Latino. The mean age of the in-service teacher sample was 39.78 years old (SD = 11.27). The in-service sample represented teachers of various levels, including pre-kindergarten (1%) teachers, grades K-5 elementary (32%) teachers, grades 6–8 middle school (40%) teachers, grades 9–12 high school (23%) teachers, and teachers of multiple school levels (3%). The average years of teaching experience was 13.40 (SD = 8.97).

The sizes of the per-service and in-service teacher samples were deemed adequate based on evidence available from prior simulation studies related to factor analytic techniques and invariance testing therein. However, the reader will note that precision in such studies is complex and conditional on myriad factors such as the size of item communalities and the extent of factor overdetermination (Cheung & Rensvold, 2002; MacCallum et al., 1999; Meade & Bauer, 2007).

Analytic Approach

The current study sought to investigate whether the 3D-MEA instrument had the same measurement properties between pre-service teachers and in-service teachers. Therefore, multigroup confirmatory factor analysis (MGCFA) was employed as the primary analytic tool. All analyses were conducted in R, primarily using the packages lavaan, semTools, and semPlot (Epskamp et al., 2019; Jorgensen et al., 2016; Rosseel, 2012). We began by examining the nature and scope of missing data, and also assessed univariate and multivariate normality, respectively, using Shapiro–Wilk and Mardia’s tests (Mardia, 1970) to aid in the selection of an estimator.

We then conducted three preliminary analyses, specifically single-group confirmatory factor analyses (CFAs) with: (i) pre-service teacher data only, (ii) in-service teacher data only, and (iii) the combined pre-service teacher and in-service data. To examine the fit of a given model, we examined a variety of fit indices such as the RMSEA, SRMR, and CFI and interpreted them in relation to fitness thresholds specified from the relevant literature (Browne & Cudeck, 1993; Byrne & Stewart, 2006; Hirschfeld & Brachel, 2014; Hu & Bentler, 1998; 1999; Schumacker & Lomax, 1996). We also used chi-square goodness-of-fit tests, though we emphasized fit indices given the sample sizes of the present study.

Next, we formally tested the invariance of model parameters across these two populations using MGCFA (Cheung & Rensvold, 2002; Jöreskog & Sörbom, 1981; Meredith, 1993). This involved comparing the fit of successive multi-group CFA analyses in which parameters were increasingly constrained equal across groups: Model 0 = parameters unconstrained across groups (i.e., configural invariance); Model 1 = factor loadings constrained equal across groups (i.e., metric invariance); Model 2 = factor loadings and item intercepts constrained equal across groups (i.e., scalar invariance); Model 3 = factor loadings, item intercepts, and residual item variances/covariances constrained equal across groups (i.e., strict invariance); and Model 4 = item factor loadings, item intercepts, residual item variances/covariances, and factor variances/covariances constrained equal across groups (i.e., structural invariance). To evaluate invariance, we used both likelihood-ratio difference tests and changes in the comparative fit index (ΔCFIs) to formally compare the fit of the increasingly constrained models (Byrne & Stewart, 2006; Hirschfeld & Brachel, 2014). Given the sensitivity of chi-square tests to sample size, however, we emphasized ΔCFIs in evaluating invariance.

The final analyses were the computation of the coefficient alphas (Cronbach, 1951) and omegas (McDonald, 2013) to examine the reliability of 3D-MEA scores in each group. We also estimated and report latent factor inter-correlations.

Results

Missing Data Analysis

Item-level missing data were negligible, ranging from 0.12% to 0.73% of cases. Little’s (1988) Missing Completely at Random (MCAR) test was used to understand the nature of the missingness. Little’s test results showed that missingness was not related to any other values in this dataset. In other words, there was insufficient evidence to reject the null hypothesis that the missing values are MCAR, $χ^{2} (218, N = 822) = 216.87, p = .51$ . Full information maximum likelihood estimation, which accommodates missing data, was used for all subsequent analyses.

Distributional Assumption Analysis

The data were checked for univariate and multivariate normality (i.e., skewness and kurtosis) before conducting CFAs and MGCFAs. Univariate normality (Shapiro-Wilk) tests for all items (p < .001) and Mardia’s multivariate skewness (b₁ = 5853.26, p < .001) and multivariate kurtosis (b₂ = 77.76, p < .001) tests were statistically significant. Due to this evidence for distributional non-normality, a robust maximum likelihood estimator was used in this present study.

Descriptive Statistical Analysis

The 20 3D-MEA item response means (M), standard deviations (SD), skewness statistics (S₃), and kurtosis statistics (K₄) for both the pre-service and in-service teacher samples are shown in Table 2. Skewness scores were lower than two and kurtosis scores did not exceed the seven for both pre-service and in-service teachers’ responses. Thus, despite statistical evidence of non-normality, skewness and kurtosis did not appear excessive in magnitude (Curran et al., 1996).

Single-Group Confirmatory Factor Analysis

The preliminary results of the single-group confirmatory factor analysis (CFA) conducted with all data combined indicated a good fit of the five-factor model, in the light of Hu and Bentler’s (1999) guidelines (see Table 3). The chi-square test was statistically significant, X² (160, N = 822) = 440.55, p < .001; however, the sample size was large. The CFI = .96 was above the threshold level of .95 (Hu & Bentler, 1999). The RMSEA (RMSEA = 0.055 [90% CI: 0.049–0.061]) showed reasonable fit to the data as it is only slightly over the cut-off point <0.05 (Hu & Bentler, 1999). Finally, the SRMR = 0.04 value indicates good fit as it is below the <0.08 recommended criterion.

Table 3.

Summary of fit for all confirmatory factor models.

Models	χ ^2a	df χ ^2a	Δ χ^2a	df Δ χ^2a	RMSEA [90% CI]^b	RMSEA [90% CI]^c	CFI^b	ΔCFI^b		ΔCFI^c	SRMR
Single-group CFA	—	—	—	—	—	—	—	—	—	—	—
Overall model (N = 822)	440.55***	160	—	—	0.046 [0.042, 0.051]	0.055 [0.049, 0.061]	0.961		0.963		0.037
IST only (N = 365)	317.13***	160	—	—	0.052 [0.045, 0.059]	0.061 [0.051, 0.071]	0.957	—	0.959	—	0.039
PST only (N = 457)	328.17***	160	—	—	0.048 [0.042, 0.054]	0.055 [0.047, 0.064]	0.955		0.957		0.044
Multi-group CFA (N=822)	—	—	—	—	—	—	—	—	—	—	—
Model 0: configural invariance	644.98***	320	—	—	0.050 [0.045, 0.054]	0.058 [0.052, 0.064]	0.956	—	0.958	—	0.042
Model 1: Metric invariance	669.65***	335	22.88	15	0.049 [0.045,0.054]	0.057 [0.051, 0.064]	0.954	−0.002	0.957	−0.001	0.045
Model 2: Scalar invariance	733.64***	350	75.70***	15	0.052 [0.047, 0.056]	0.060 [0.054, 0.066]	0.948	−0.006	0.952	−0.005	0.047
Model 3: Strict invariance	760.79***	370	31.50*	20	0.051 [0.046, 0.055]	0.059 [0.053, 0.065]	0.947	−0.001	0.949	−0.003	0.049
Model 4: Structural invariance	758.91***	385	10.44	15	0.049 [0.045, 0.053]	0.059 [0.053, 0.065]	0.945	−0.002	0.948	−0.001	0.081

Notes. CFA = confirmatory factor analysis. PST = pre-service teachers. IST = in-service teachers. Model 0 = parameters unconstrained across groups. Model 1 = factor loadings constrained equal across groups. Model 2 = factor loadings and item intercepts constrained equal across groups. Model 3 = factor loadings, item intercepts, and residual item variances/covariances constrained equal across groups. Model 4 = item factor loadings, item intercepts, residual item variances/covariances, and factor variances/covariances constrained equal across groups. Individual model chi-square and chi-square difference statistics are robust, using the Satorra-Bentler (2001) adjustment.

^aChi-square statistic adjusted for non-normality using the Satorra-Bentler (2001) method.

^bCalculated using the Satorra–Bentler-adjusted (2001) chi-square statistic.

^cCalculated using the methodology outlined in Brosseau-Liard et al. (2012) and/or Brosseau-Liard and Savalei (2014).

p < .05, **p < .01, ***p < .001.

The fit of the five-factor model was also preliminarily assessed among pre-service teachers and in-service teachers separately, also using single-group CFA. Results showed the five-factor model fit well in both groups. The chi-square test was statistically significant for the pre-service teacher sample data, X² (160, N = 457) = 398.17, p < 0.001; however, the fit indices show that the five-factor model fits the current pre-service teacher sample: CFI = 0.96, RMSEA = 0.055 (90% CI: 0.047–0.064), and SRMR = 0.04. The chi-square test result was statistically significant for in-service teacher sample data too, X² (160, N = 365) = 317.13, p < 0.001, however, fit indices indicated that the five-factor model well fit for in-service teachers: CFI = 0.96, RMSEA = 0.061 (90% CI: 0.051–0.071), and SRMR = 0.04. Given these results, we proceeded with formal invariance testing using multi-group CFA.

Multi-Group Confirmatory Factor Analysis

The configural invariance of the five-factor measurement model between pre-service teachers and in-service teachers was examined first as shown in Table 3. Unsurprisingly, given the results of the single-group CFAs, the results suggested that the five-factor 3D-MEA measurement model fit the data well for both pre-service and in-service teachers according to corresponding CFI, RMSEA and SRMR fit indices. While the chi-square test of fit was significant, X² (320, N = 822) = 644.98, p < .001), fit indices were as follows: CFI =0.96, RMSEA = 0.058 (90% CI: 0.052–0.064), and SRMR = 0.04.

The metric invariance of the five-factor measurement model between pre-service teachers and in-service teachers was then examined as a second step. In this model, the factor loadings were constrained to be equal, meaning that the relationships between latent factors and items for both groups were equal. The results showed good model fit. While again the chi-square fit test was significant, X² (335, N = 822) = 669.65, p < .001), fit indices indicated good model fit: CFI = 0.95, RMSEA = 0.057 (95% CI: 0.051–0.064), and SRMR = 0.05. The non-significant corrected chi-square difference test, Δχ² (15, N = 822) = 22.88, p = .09, indicates that the configural invariance model and metric invariance model fit equally well. In addition, the ΔCFI did not exceed .01 indicating only a trivial degradation in fit relative to the configural invariance model.

For the third step of the MGCFA, scalar invariance was investigated, in which the factor structure, factor loadings, and item intercepts were constrained to be the same for both pre-service and in-service teachers. Overall, the results showed that this model fit well in absolute terms: X² (350, N = 822) = 733.64, p < .001, CFI =0.95, RMSEA = 0.060 (90% CI: 0.054–.066), and SRMR = 0.05. The corrected chi-square differences test Δχ² (15, N = 822) = 75.70, p < .001 indicated possible non-invariance. However, ΔCFI did not exceed the .01 cut-off point again indicating only possibly trivial differences in these parameters across groups.

Next, we examined strict measurement invariance by comparing the fit of the strict invariance model (with factor loadings, item intercepts, and residual item variances/covariances constrained equal across groups) and the scalar invariance model (in which residual item variances/covariances were not equal). While the chi-square difference test between models suggested non-invariance Δχ² (20, N = 822) = 31.50, p < .05, the ΔCFI was trivial implying strict invariance for practical purposes. Finally, structural invariance was examined via likelihood ratio test and ΔCFI value (ΔCFI <.01) for comparison of the structural invariance and strict invariance models. Both the likelihood-ratio test, Δχ² (15, N = 822) = 10.44, p = .79 and ΔCFI (−0.002) implied structural invariance of 3D-MEA scores for pre-service and in-service teachers.

Reliability Analysis

Table 4 reports alpha and omega internal consistency reliability estimates. For in-service teachers, the alpha reliability estimates ranged from 0.83 to 0.91 and the omega reliability estimates ranged from 0.82 to 0.90. For pre-service teachers, the alpha reliability estimates ranged from 0.78 to 0.89 and the omega reliability estimates ranged from 0.82 to 0.90. In general, the coefficient alpha and omega reliability estimates were adequately high per Nunnaly (1978) for all 3D-MEA sub-dimensions and both in-service and pre-service teachers. One exception was the alpha estimate for pre-service teachers’ self-efficacy for data analysis and interpretation scores (α = 0.78), though this was only nominally lower than Nunnaly’s ≥ .80 criterion.

Table 4.

Reliability estimates.

	Access	Technology	Interpretation	Instruction	Anxiety
IST (N = 365)	—	—	—	—	—
Alpha	0.83	0.91	0.85	0.90	0.91
Omega	0.83	0.90	0.82	0.89	0.89
PST (N = 457)	—	—	—	—	—
Alpha	0.82	0.89	0.78	0.88	0.88
Omega	0.83	0.90	0.82	0.89	0.89

Notes. IST = in-service teachers; PST = pre-service teachers; Access = self-efficacy for data identification and access; Technology = self-efficacy for data technology use; Interpretation = self-efficacy for data analysis and interpretation; Instruction = self-efficacy for the application of data to instruction.

Inter-Factor Correlational Analysis

Table 5 shows the estimated inter-factor correlations. A single set of inter-factor correlation estimates are presented here for both pre-service teachers and in-service teachers because our analysis showed structural invariance across populations.

Table 5.

Inter-factor correlations.

-	Access	Technology	Interpretation	Instruction	Anxiety
Access	1	—	—	—	—
Technology	0.71	1	—	—	—
Interpretation	0.78	0.72	1	—	—
Instruction	0.72	0.61	0.82	1	—
Anxiety	−0.40	−0.37	−0.46	−0.42	1

Notes. Access = self-efficacy for data identification and access; Technology = self-efficacy for data technology use; Interpretation = self-efficacy for data analysis and interpretation; Instruction = self-efficacy for the application of data to instruction.

Discussion

In many statistical studies that involve comparing populations, the desired study outcome is to detect a difference in model parameters between those populations (e.g., a mean difference between the two populations). However, in invariance studies the hoped-for outcome is instead no difference in the psychometric model parameters across groups. In this study, this is effectively what we found: no meaningful differences in the psychometric model parameters for pre-service and in-service teachers. Specifically, we found evidence for structural invariance, the highest level of invariance. This means that the 3D-MEA item factor loadings, item intercepts, and item error variances were equal for both in-service and pre-service teachers, as were the latent factor variances/covariances.

With the estimation of Model 4 in which factor variances and covariances were also constrained equal across groups, we noticed a slight uptick in the SRMR fit index (SRMR = .08). The SRMR fit index is known to be particularly sensitive to misspecification of factor variances and/or covariances (Chen, 2007; Hu & Bentler, 1998) and this increase may imply non-equivalence in these parameters. However, this phase of invariance testing pertains to “structural” invariance rather than “measurement” invariance (Dimitrov, 2010) and other model results did imply structural measurement invariance.

In lay terms, these invariance analyses clearly suggest that the 3D-MEA measures the relevant constructs similarly in these two important populations. Another way to think about these findings is that 3D-MEA scores mean the same thing across these two key populations (Byrne & Stewart, 2006). At the same time, we also provided evidence that 3D-MEA score reliability was similarly high for both in-service and pre-service teacher populations, indicating similar levels of random measurement error in these two groups.

The implications of our findings pertain most directly to the measurement concerns of researchers. Structural 3D-MEA invariance across these two populations affords researchers the opportunity to credibly use this instrument with both populations. In particular, this structural invariance evidence helps substantiate the appropriateness of using the 3D-MEA to draw mean comparisons in DDDM self-efficacy and anxiety between pre-service and in-service teachers. It also establishes one necessary condition for well measuring changes in DDDM self-efficacy and anxiety longitudinally over time from the pre-service stage to the in-service stage. This invariance also means that researchers can similarly interpret correlations between this measure and other measures administered to pre-service or in-service populations. In the absence of invariance, observed differences or changes, or differences in correlations with external measures, could be artifacts of measurement error. Such analyses may be necessary in various kinds of studies conducted by researchers involving these populations, including studies that ask developmental, theory-building, or intervention research questions.

These affordances may also benefit practitioners (e.g., higher education personnel, K-12 district staff, and professional development providers) who might use the 3D-MEA for various purposes with both of these populations. These include identifying pre-service or in-service teacher needs, or evaluating professional learning opportunities served up to both pre-service and in-service teachers. High-quality information derived during such efforts may in turn help improve teacher development mechanisms for DDDM, DDDM practices themselves, and potentially student learning and development outcomes.

Our findings about the invariance and reliability of 3D-MEA scores weigh in favor of effectively interchangeable use of this instrument with both U.S. in-service and pre-service teachers. However, the present study had a number of key limitations and delimitations. Chief among the study’s limitations relate to the sampling design, which yielded fairly diverse but not necessarily representative samples of all U.S. in-service and pre-service teachers. For example, our in-service and pre-service samples were limited in terms of the states from which the participants were drawn and/or the school levels in which they work. Additionally, our PST sample was disproportionately female (92% vs. 75% nationally) and white (87% vs. 79% nationally), and our IST sample was disproportionately white (98% vs. 79% nationally) relative to the U.S. teacher population generally (Taie & Goldring, 2020). Our findings should certainly be replicated independently with larger and more representative samples of in-service and pre-service teachers.

Also, our study only provided select forms of reliability and validity evidence concerning the 3D-MEA, generally the forms of validity evidence provided by prior studies of this instrument (i.e., validity evidence based on score internal structure). While sporadic data do exist from substantive studies that could be considered validity evidence (e.g., evidence of correlations with other measures), other forms of validity evidence should be formally gathered as well to more comprehensively validate inferences derived from this instrument. For example, researchers should deploy methodologies such as think-aloud protocols to collect validity evidence based on test-taker response processes; and statistical modeling to collect validly evidence based on score relations with other variables (AERA et al., 2014), especially objective measures of teacher data literacy and implementation of data use practices. These limitations notwithstanding, our findings do present compelling evidence of what the study set out to establish: the invariance of 3D-MEA scores across two critical teacher sub-populations.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Todd D. Reeves

Note

References

American Educational Research Association American Psychological Association & National Council on Measurement in Education . (2014). Standards for Educational and Psychological Testing, American Educational Research Association.

American Recovery and Reinvestment Act of 2009 . (2009), P.L. 111-5, 111th Cong. (2009).

Authors . (2016).

Authors . (2017).

Authors . (2018).

Authors . (2020).

Bandura

(1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191–215. https://doi.org/10.1037//0033-295x.84.2.191

Bandura

(1994). Self-efficacy. In Ramachaudran

V. S.

(Ed.), Encyclopedia of human behavior (Vol. 4, pp. 71–81). Academic Press.

Brosseau-Liard

P. E.

Savalei

(2014). Adjusting incremental fit indices for nonnormality. Multivariate Behavioral Research, 49(5), 460–470. https://doi.org/10.1080/00273171.2014.933697

10.

Brosseau-Liard

P. E.

Savalei

(2012). An investigation of the sample performance of two nonnormality corrections for RMSEA. Multivariate Behavioral Research, 47(6), 904–930. https://doi.org/10.1080/00273171.2012.715252

11.

Browne

M. W.

Cudeck

(1993). Alternative ways of assessing model fit. In Bollen

K. A.

Long

J. S.

(Eds.), Testing structural equation models (pp. 136–162). Sage.

12.

Byrne

B. M.

Stewart

S. M.

(2006). Teacher’s corner: The MACS approach to testing for multigroup invariance of a second-order structure: A walk through the process. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 287–321. https://dx-doi-org-s.web.bisu.edu.cn/10.1207/s15328007sem1302_7

13.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

14.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/s15328007sem0902_5

15.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/bf02310555

16.

Curran

P. J.

West

S. G.

Finch

J. F.

(1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29. https://doi.org/10.1037/1082-989x.1.1.16

17.

Dimitrov

D. M.

(2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43(2), 121–149. https://doi.org/10.1177/0748175610373459

18.

Dorn

Hancock

Sarakatsannis

Viruleg

(2020). COVID-19 and student learning in the United States: The hurt could last a lifetime. https://webtest.childrensinstitute.net/sites/default/files/documents/COVID-19-and-student-learning-in-the-United-States_FINAL.pdf

19.

Dunn

K. E.

Airola

D. T.

Hayawaka

(2020). Pre-service teachers’ efficacy, anxiety, and concerns about data and the new idea of anchored judgment. Current Issues in Education, 21(1), 1–25.

20.

Dunn

K. E.

Airola

D. T.

W.-J.

Garrison

(2013). Becoming data driven: The influence of teachers’ sense of efficacy on concerns related to data-driven decision making. The Journal of Experimental Education, 81(2), 222–241. https://doi.org/10.1080/00220973.2012.699899

21.

Epskamp

Stuber

Nak

Veenman

Jorgensen

T. D.

(2019). Package ‘semplot’. http://cran-r.c3sl.ufpr.br/web/packages/semPlot/semPlot.pdf

22.

Every Student Succeeds Act of 2015 . (2015). P. L. 114-95, S.1177, 114th Cong. (2015).

23.

Green

J. L.

Schmitt-Wilson

Versland

T. M.

(2016). Teachers and data literacy: Ablueprint for professional development to foster data driven decision making. Journal of Continuing Education and Professional Development, 3(1), 14–32. https://doi.org/10.7726/jcepd.2016.1002

24.

Guskey

T. R.

(1988). Teacher efficacy, self-concept, and attitudes toward the implementation of instructional innovation. Teaching and Teacher Education, 4(1), 63–69. https://doi.org/10.1016/0742-051x(88)90025-x

25.

Hirschfeld

Brachel

R.V

(2014). Improving Multiple-Group confirmatory factor analysis in R–A tutorial in measurement invariance with continuous and ordinal indicators. Practical Assessment, Research, and Evaluation, 19(7), 1–12. https://doi.org/10.7275/qazy-2946

26.

L.-T.

Bentler

P. M.

(1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453. https://doi.org/10.1037/1082-989x.3.4.424

27.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

28.

Individuals with Disabilities Education Improvement Act . (2004), H.R. 1350, 108th Congress.

29.

Jöreskog

K. G.

Sörbom

(1981). LISREL V: Analysis of linear structural relationships by the method of maximum likelihood. University of Uppsala.

30.

Jorgensen

T. D.

Pornprasertmanit

Schoemann

A. M.

Rosseel

Miller

Quick

Coffman

(2016). Package ‘semtools’. https://cran.r-project.org/web/packages/semTools/semTools.pdf

31.

Kaffenberger

(2021) Modelling the long-run learning impact of the Covid-19 learning shock: actions to (more than) mitigate loss. International Journal of Educational Development, 81, 102326. https://doi.org/10.1016/j.ijedudev.2020.102326

32.

Little

R. J. A.

(1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 1198–1202. https://doi.org/10.1080/01621459.1988.10478722

33.

MacCallum

R. C.

Widaman

K. F.

Zhang

Hong

(1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. https://doi.org/10.1037/1082-989x.4.1.84

34.

Mandinach

E. B.

Gummer

E. S.

(2016). What does it mean for teachers to be data literate: Laying out the skills, knowledge, and dispositions. Teaching and Teacher Education, 60, 366–376. https://doi.org/0.1016/j.tate.2016.07.011

35.

Mardia

K. V.

(1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3), 519–530. https://doi.org/10.1093/biomet/57.3.519

36.

McDonald

R. P.

(2013). Test theory: A unified treatment. Psychology Press.

37.

Meade

A. W.

Bauer

D. J.

(2007). Power and precision in confirmatory factor analytic tests of measurement invariance. Structural Equation Modeling, 14(4), 611–635. https://doi.org/10.1080/10705510701575461

38.

Means

Chen

DeBarger

Padilla

(2011). Teachers’ ability to use data to inform instruction: Challenges and supports. https://files.eric.ed.gov/fulltext/ED516494.pdf

39.

Means

Padilla

Gallagher

(2010). Use of education data at the local level: From accountability to instructional improvement. U.S. Department of Education.

40.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543. https://doi.org/10.1007/bf02294825

41.

No Child Left Behind Act of 2001 . (2002), P.L. 107-110, 20 U.S.C. § 6319

42.

Nunnally

J. C.

(1978). Psychometric theory (2nd ed.). McGraw-Hill.

43.

Reeves

T. D.

Chiang

J. L.

(2017). Building pre-service teacher capacity to use external assessment data: An intervention study. The Teacher Educator, 52(2), 155–172.

44.

Reeves

T. D.

Onder

Abdi

(2020). Validation of the Data-Driven Decision-Making Efficacy and Anxiety Inventory (3D-MEA) with U.S. pre-service teachers. Mid-Western Educational Researcher, 32(4), 286–303.

45.

Reeves

T. D.

Summers

K. H.

Grove

(2016). Examining the landscape of teacher learning for data use: The case of Illinois. Cogent Education, 3(1).

46.

Rosseel

(2012). Lavaan: An R package for structural equation modeling and more (Version 0.5–12). Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

47.

Satorra

Bentler

P. M.

(2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4), 507–514. https://doi.org/10.1007/bf02296192

48.

Schumacker

R. E.

Lomax

R. G.

(1996). A beginner’s guide to structural equation modeling. Lawrence Erlbaum.

49.

Taie

Goldring

(2020). Characteristics of public and private elementary and secondary school teachers in the United States: Results From the 2017–18 National Teacher and Principal Survey first look (NCES 2020- 142). National Center for Education Statistics, U.S. Department of Education.

50.

Tschannen-Moran

Hoy

A. W.

(2001). Teacher efficacy: Capturing an elusive construct. Teaching and Teacher Education, 17(7), 783–805. https://doi.org/10.1016/s0742-051x(01)00036-1

51.

Walker

D. A.

Reeves

T. D.

Smith

T. J.

(2018). Confirmation of the Data Driven Decision-Making Efficacy and Anxiety Inventory’s (3D-MEA) score factor structure among teachers. Journal of Psychoeducational Assessment, 36(5), 477–491.

52.

Wayman

J. C.

Jimerson

J. B.

(2014). Teacher needs for data-related professional learning. Studies in Educational Evaluation, 42, 25–34. https://doi.org/10.1016/j.stueduc.2013.11.001