Validation of Scores From a New Measure of Preservice Teachers’ Self-efficacy to Teach Statistics in the Middle Grades

Participants

This study was conducted in four different large-sized public institutions of higher education with teacher education programs. Two institutions are located in the Midwest United States with the other two in the south-central United States. The sample consisted of n = 309 preservice teachers enrolled in a mathematics education or introductory statistics course. As institutions do not always offer a statistics course taken only by preservice teachers, participants were recruited from a variety of courses, including mathematics courses taken only by preservice elementary and/or middle grades teachers (e.g., number sense, geometry, algebra), probability and statistics, as well as other courses required in teacher education programs, such as history of mathematics. As such, the preservice teachers were at various stages in their teacher preparation program (see Table 1). Some preservice teachers provided inconsistent or nonstandard grade band intervals to describe their (intended) certifications, but 83.5% reported an interval including one or more of the middle grades (i.e., 5-9), and 11.3% did not respond. Of the 88.7% who responded to this item, 94.2% reported an interval including one or more of the middle grades years. Table 1 summarizes participants’ backgrounds.

The Self-Efficacy to Teach Statistics (SETS) Instrument

The SETS instrument aims to measure levels of self-efficacy to teach statistics, based on the pre-K-12 GAISE report and state standards for teacher knowledge and student learning outcomes that have specific statistics requirements, such as the CCSSM. Groth and Bargagliotti (2012) explore the complementary nature of GAISE and CCSSM. After 15 demographic questions, the SETS instrument consists of 11 items corresponding to GAISE Level A and 15 items to Level B. The instrument has respondents rate their confidence in teaching [specific statistics concepts to] middle school students using a 6-point scale in which higher values correspond to higher confidence. Table 2 presents examples from the SETS and the associated CCSSM Statistics Standard. A response of “confident” (the scale’s 4th point) for every item yields a Level A subscore of 44 (of 66) and Level B subscore of 60 (of 90).

OPEN IN VIEWER

Table 2.

Examples From SETS and CCSSM.

Level	SETS items	CCSSM statistics standards
A	Recognize that there will be natural variability between observations for individuals	Develop understanding of statistical variability:
		• Understand that a set of data has distribution with specific center/variation/shape.
	Create boxplots for summarizing distributions	Summarize and describe distributions:
		• Create boxplots/histograms.
		• Summarize data numerically.
B	Recognize sampling variability in summary statistics such as the sample mean and the sample proportion	Use random sampling to draw inferences about a population:
		• Use data from a random sample to draw inferences about a population with an unknown characteristic of interest.
	Use interquartile range, five-number summaries, and boxplots for comparing distributions	Draw informal comparative inferences about two populations:
		• Use graphs to estimate differences.
		• Use numerical values to assess differences.

Unlike a Structural Equation Modeling (SEM) or Item Response Theory (IRT) framework, which approaches measurement by finding the best model to fit the data, the Rasch framework used focuses on a data generation process that conforms to a specific measurement model. The analysis stage emphasizes exposing data anomalies, not comparing model fit (Andrich, 2004a, 2004b).

Following Wolfe and Smith (2007a), the purpose and frame of reference for the instrument was identified. State mathematics standards for teacher and student statistics content knowledge were obtained from each state’s department of education website and reviewed in conjunction with the items described in GAISE Levels A and B to identify representative behaviors that the SETS instrument would measure. A polytomous scoring model with a 6-point scale was selected to align with SELS and CSSE. To measure two levels of efficacy, the Multidimensional Random Coefficient Multinomial Logit Model, a multidimensional Rasch model, was selected as the scaling model (Adams, Wilson, & Wang, 1997; Briggs & Wilson, 2003).

The item-writing process started in spring 2008. Comments on the item wording were solicited from 12 in-service middle and elementary school teachers enrolled in a mathematics specialist K-8 graduate program at a mid-Atlantic state university. Teaching experience ranged from 2 to 19 years, with a median of 6. About an hour of time was granted during a weekly class meeting taught by the coordinator of the specialist program for teachers to complete the instrument and provide anonymous written comments or suggested changes about item wording. Table 3 illustrates the iterative process of item revision that was completed in fall 2008.

OPEN IN VIEWER

Table 3.

Examples of SETS Iterative Item Revision Process.

	Item 11	Item 12
Stage 0: given to teachers	Recognize that a sample may or may not be representative of a larger population.	Distinguish between a question to be answered with one observation versus a question to be answered based on the interpretation of data that vary.
Stage 1: initial modification based on teacher feedback	Recognize the limitation of scope of inference to the classroom.	Distinguish between a question with an answer based on data that vary versus a question with a deterministic answer.
Stage 2: modified further from authors’ debriefing meetings	Recognize the limitation of making inference (i.e., generalization) from a classroom dataset to any population beyond the classroom.	Distinguish between a question based on data that vary and a question based on a deterministic model (e.g., specific values of rate and time determines a particular value for distance in the model d = r × t).

In spring 2009, the revised instrument was piloted with 23 middle grades preservice teachers and 21 noneducation majors at a doctorate-granting public university in the south-central United States for the basis of an informal exploration of response category usage and item hierarchy. Participants were undergraduates in two sections of an introductory statistics course required for future middle school teachers and other majors such as psychology, biochemistry, industrial technology, and nursing. To explore variability in teaching efficacy levels, stacked bar graphs were generated for each item, showing the percentage of responses in each category level for the entire group as well as for just the education majors (Harrell, Pierce, Sorto, Murphy, & Lesser, 2009). Each of the six rating scale categories was used for each item. Furthermore, descriptive statistics for the items showed that respondents indicated higher efficacy for teaching statistical topics associated with Level A than with those at Level B, confirming the hierarchy intended in the items.

Structural Evidence

Continuing with the Rasch framework, confirmatory factor analysis using the 2009-2010 data (n = 309) was conducted using Conquest. Both AIC and BIC were smaller for the two-dimensional model (AIC = 21541, BIC = 21558) than for the unidimensional model (AIC = 21765, BIC = 21780), thus supporting use of the two subscales in further analysis.

Substantive Evidence

Linacre’s criteria (Linacre, 2004) were used in evaluating the performance of the rating scale. These eight criteria are divided into two groups: “essential” and “helpful” (see last column, Table 4). The 6-point rating scale in the SETS instrument meets all four essential criteria. The Count column of Table 5 shows at least 10 observations for each response category and a unimodal distribution of frequencies. Because each category is being utilized and the distribution of the frequencies follows a bell-shaped curve, the pitfalls of low frequencies for a rating scale category or a bimodal distribution of frequencies do not affect model estimates. The average respondent measure associated with each category increases with the values of the rating scale categories (see Average θ column), indicating teachers with more efficacy overall respond with the higher categories. The unweighted mean-square fit statistics indicate how responses to each rating category match Rasch model expectations. These values are each less than 2.0, indicating a reasonable amount of randomness exhibited in the responses.

OPEN IN VIEWER

Table 4.

Linacre’s Criteria for Rating Scale Evaluation.

	Essential for validity?
	Yes	No
Criterion met?
Yes	N > 10 for each response category;	Category thresholds increase with category;
	Unimodal distribution for each response category;	Category thresholds increase by < 5 logits
	Average measures increase with Category;
	Outfit MNSQ < 2 for each category
No	—	Measure implies category & category implies measure (coherence);
		Category thresholds increase by .81 logits

OPEN IN VIEWER

Table 5.

Rating Scale Category Structure.

							Coherence
Category	Category title	Count	Avg. θ	Infit MNSQ	Outfit MNSQ	Threshold	M → C	C → M
1	Not at all confident	272	−1.02	1.18	1.21	—	80%	15%
2	Only a little confident	557	−0.45	0.89	0.90	−1.43	39%	20%
3	Somewhat confident	1287	0.10	0.97	0.97	−0.98	35%	36%
4	Confident	2065	0.57	0.96	0.94	−0.14	38%	50%
5	Very confident	2094	1.15	0.92	0.93	0.83	40%	55%
6	Completely confident	1607	1.79	1.08	1.05	1.72	71%	27%

Half of the four helpful, but nonessential, criteria were met. The category thresholds increase as categories increase (see Table 5), supporting the current ordering of the categories. For measure-to-category coherence, Categories 2, 3, and 4 are just below the 40% guideline. For category-to-measure coherence, Categories 1, 2, 3, and 6 exhibit less success than the 40% guideline, ranging from 14% to 35%. These category-to-measure coherence percentages indicate that Categories 2 and 3 (“Only a Little Confidence” and “Somewhat Confident”) may be too close in meaning for the respondents to differentiate between them as the model expects. For a 5-point rating scale, the difference in logits between the thresholds of adjacent categories should be at least .81 logits (Wolfe & Smith, 2007b). Although the SETS instrument uses a 6-point scale, the differences between the thresholds are compared to the 5-point criteria. The difference in the first and second thresholds (“Not at all confident” vs. “Only a little confident”) is only .45, while the difference between the remaining adjacent categories is at least .84. Table 5 shows results of the analysis using Linacre’s criteria.

Winsteps estimated the reliability of separation coefficients for the “Reading the Data” and “Reading Between the Data” subscales as .87 and .91, respectively. With subscales’ differing in numbers of items (11 vs. 15, respectively), this observed difference is expected. These estimates are analogous to Cronbach’s alpha, indicating very little measurement error in the scores. Since the estimates exceed .80, decisions based on the SETS scores could be used for either group-based or individual decisions. Conquest estimated the correlation between subscales as .85.

Content Evidence

The item development process itself provides evidence of content validity. The SETS items were derived from the GAISE and states’ standards for teacher and student knowledge of statistics. As previously described, the items were reviewed by in-service teachers enrolled in a K-8 mathematics specialist program to evaluate the wording of the items, which resulted in item revisions. During the final stages of the validation process for the SETS instrument, the CCSSM were released. Informal comparison of the SETS instrument and the CCSSM showed that the items aligned, which was unsurprising given that several GAISE authors helped to develop the CCSSM. Groth and Bargagliotti (2012) discuss connections between GAISE and CCSSM.

Both the item-measure correlations and mean square fit indices are used to evaluate the quality of the items and provide evidence of content validity (Wolfe & Smith, 2007b). The Fisher transformations of the point-polyserial correlations for each item, with the total from its subscale, are shown in the third column of Table 6. The direction and magnitude (all greater than .67) of these correlations indicate a strong, positive relationship between the polytomous item score and the subscale score based on the remaining items. The observed correlations are much greater than the minimum value (.40) suggested by Wolfe and Smith (2007b) for flagging polytomous items.

OPEN IN VIEWER

Table 6.

Item Quality Index Summary.

Subscale structure	Item number	Weighted mean square statistic	Unweighted mean square statistic	Fisher-transformed point-polyserial correlations
“Reading the data (Level A)”	1	1.00	1.06	0.71
	2	1.09	1.07	0.68
	3	1.02	1.10	0.68
	4	1.07	0.95	0.93
	5	0.98	0.95	0.93
	6	1.17	1.13	0.85
	7	1.06	0.99	0.89
	8	0.77	0.83	0.93
	9	0.71	0.78	0.93
	10	1.04	0.94	0.76
	11	1.09	1.18	0.68
“Reading Between the data (Level B)”	12	0.87	0.95	0.81
	13	0.99	0.93	0.78
	14	0.71	0.72	0.91
	15	1.09	1.03	0.89
	16	1.03	0.98	0.91
	17	1.52	1.47	0.74
	18	1.45	1.39	0.78
	19	1.00	1.02	0.81
	20	0.85	0.85	0.89
	21	0.74	0.72	1.02
	22	0.92	0.87	0.83
	23	0.88	0.86	0.93
	24	1.16	1.12	0.73
	25	1.04	1.01	0.81
	26	0.83	0.80	0.91

Winsteps-generated mean square fit indices for the items in each subscale are summarized in the last two columns of Table 6. The unweighted mean-square statistic (outfit) indicates items where unexpected responses are observed, but the responses are close to the respondent’s ability. Weighted mean-square statistic (infit) indicates items where unexpected responses are far from the respondent’s ability. The average for both fit indices was near 1, indicating that, on average, the observed responses differ from the expected responses by roughly the amount predicted by the Rasch model. For almost all items, the mean square statistic was within the acceptable range of 0.5 to 1.5 (Linacre, 2011). The exception is Item 18, whose Infit MNSQ of 1.52 exceeds the suggested 1.5 cutoff by a negligible amount. Based on the correlation and mean square values, there are no items that are flagged as problematic in terms of item quality.