Exploring the Reliability of Curriculum-Based Measurement as a Formative Assessment in a College Classroom

Abstract

The purpose of this preliminary study was to explore the reliability of Curriculum-Based Measurement (CBM) vocabulary-matching forms with students in an introduction to special education course in a college setting. Data from 84 students enrolled in a teacher preparation program across three semesters were examined. Results suggest low to moderate alternate form reliability with adjacent forms (r = .49) compared to the mean of two weekly forms (r = .65). Future directions on form development to strengthen reliability are discussed as well as implications for CBM use in college classrooms as a formative assessment tool.

Keywords

Curriculum-Based Measurement progress monitoring postsecondary

In the collegiate classroom, the expectation is for students to learn content related to a specific subject area (Knight, 2002). Each subject area possesses content-specific vocabulary or academic language that is necessary for acquiring foundational knowledge, prior to moving on to more challenging content (Harmon, Wood, & Hendrick, 2008; Kennedy & Ihle, 2012). College students often possess adequate basic and intermediate literacy skills (i.e., decoding, reading comprehension) but still need to learn large amounts of vocabulary to demonstrate disciplinary literacy or the ability to access and use information in a content area (Kennedy & Ihle, 2012). Research has established that vocabulary knowledge contributes to reading comprehension, high quality writing in content areas, and the ability to meaningfully communicate in the content area (Bravo & Cervetti, 2008). Vocabulary knowledge ranges from no knowledge of a word to deep understanding that allows a person to use it within multiple contexts (Phythian-Sence & Wagner, 2007). To operate in a specific field, then, it is critical one knows the language of that field (Lemke, 1990).

For example, in an introductory biology course, a student is often expected to learn more words in a semester than one would in a foreign language course (Bravo & Cervetti, 2008). In an introductory level course that provides foundational knowledge in special education content, a corresponding textbook can contain upward of 300 vocabulary words (Hallahan, Kauffman, & Pullen, 2015). In a field such as special education, the breadth of knowledge and expertise teacher candidates need is vast (Brownell, Sindelar, Kiely, & Danielson, 2010), and the content knowledge alone is extensive (Scheeler, Budin, & Markelz, 2016). The language in this field is not only complex, but specialized. For instance, teacher candidates are exposed to a set of new terminology, as well as many acronyms that also represent new terms. These future teachers are expected to learn terms to increase their knowledge and skills to implement critical instruction, interventions, and assessments for students with exceptionalities (Kennedy et al., 2016). New terms such as “PBIS,” “accommodations,” “myopia,” or “progress monitoring” may sound familiar to education majors. However, teacher candidates often learn multiple meanings for similar words in different contexts and are expected to apply them in different ways when working with students with disabilities. Thus, acquiring this new vocabulary (i.e., foundational knowledge) contributes to subject mastery for these teacher candidates. This is critical, as before a concept can be applied, it must first be defined (Cuvo, 2003). While many students possess adequate reading comprehension skills at the postsecondary level, it is possible that some may struggle to learn the large amount of vocabulary being introduced in a small amount of time, identify how to use familiar vocabulary in unfamiliar contexts, and apply new constructs in the field of study (Bravo & Cervetti, 2008; Kennedy & Ihle, 2012).

As noted above, the content knowledge in the field of special education is extensive (Scheeler et al., 2016). De Arment, Reed, and Wetzel (2013) posit that taking this first step to ensure teacher candidates have the fundamental knowledge may help them develop and strengthen adaptive expertise to better prepare them prior to entering the field. Adaptive expertise, which was conceptualized by Hatano and Inagaki (1986) is the ability to work efficiently while being innovative. De Arment et al. note this framework could be used within preparation programs for special education preservice teachers, as it could help to examine a candidate’s knowledge, skills, and dispositions. While ensuring that preservice teachers possess adequate content knowledge, this would allow them to develop at least one facet of adaptive expertise, perhaps a first step. Furthermore, this would be one way to ensure that teachers are learning the language of the field, ultimately helping them to be more successful in their career (Brownell et al., 2010; Kennedy & Ilhe, 2012; Kennedy et al., 2016).

Formative Assessment in College Classrooms

To ensure students know, understand, and can apply, when appropriate, new terminology, some type of assessment must occur. Summative assessments, such as exam scores and course grades, are often used to determine whether or not students have mastered content. These provide useful information in that they represent an overall picture of student performance (Salvia, Ysseldyke, & Bolt, 2012). While it is advantageous to incorporate summative assessments, they leave little to no time to intervene. Thus, there is merit to using formative assessment in collegiate classrooms, especially if that feedback assists the learner and the instructor (Glazer, 2014). Formative assessment is often defined as an ongoing task with the purpose of assisting student learning and student performance (Yorke, 2003). However, the definition of “ongoing task” is broad. For example, Glazer (2014) defines formative assessment as “any task that provides feedback to students on their learning achievements during the learning process” (p. 277). She gives examples such as multiple-choice items, essays, or performance tasks that provide feedback during the process of learning instead of at the conclusion of an instructional period. Such a general definition allows much freedom in the college classroom, but it may also be challenging for instructors to give frequent, quality feedback in a timely and efficient manner depending on the type of formative assessment technique (i.e., essay versus multiple-choice; Glazer, 2014).

Yorke (2003) also suggests the use of formative assessments in higher education, as they have a wide utility. For example, this ongoing feedback may also produce self-regulated learners and has the potential to serve as internal motivators to engage students to complete class readings and assignments (Cauley & McMillan, 2010; Nicol & Macfarlane-Dick, 2006). Despite these benefits, the implementation of formative assessment tools in postsecondary settings remains infrequent and is an area that requires further investigation (Glazer, 2014; Knight, 2002; Yorke, 2003). Therefore, frequently monitoring students’ acquisition of vocabulary in introductory special education courses, which presents new terms, definitions, and concepts, seems like a logical first step in assessing learning.

De Arment et al. (2013) not only noted the importance of developing adaptive expertise, they also suggest developing assessments to measure adaptive expertise to provide feedback. Formative assessments would be one way to do this, as this provides feedback to both the instructor and student (Glazer, 2014). This kind of feedback is essential as it enhances student learning, hopefully giving students more breadth and depth of content knowledge in their field of study. It also allows instructors to determine what learners know and what content caused difficulty, and could possibly identify those students at risk of failing (Larson & Ward, 2006). Given that introductory courses in special education can often be taught by many different instructors, having a type of standardized measure could provide consistent and frequent feedback to instructors regarding student outcomes specific to terminology in their courses (Glazer, 2014).

Curriculum-Based Measurement

Curriculum-Based Measurement (CBM; Deno, 1985) is a standardized, formative assessment that provides an instructor with general outcome knowledge and a way to monitor progress in acquisition of a specific academic skill set (Deno, 2003). Over 30 years of research literature reflects that CBM measures are quick academic tasks that can be used as consistent indicators of performance for a variety of K-12 skills in reading (i.e., oral reading fluency, maze), mathematics (i.e., computation), writing (i.e., words written), and content areas (i.e., vocabulary-matching; Hosp, Hosp, & Howell, 2016).

While CBM has been shown to be effective in the K-12 setting, there has been limited exploration of the value of these measures at the postsecondary level. However, given the data supporting CBM, university instructors may be able to employ this technique to objectively assess the progress of learners (Larson & Ward, 2006). Using this type of formative assessment system could provide many potential benefits, as it could allow instructors to check on individual student performance over time and use data to change study habits, instruction, or program emphases (Larson & Ward, 2006). In addition, having an indicator that demonstrates a student’s understanding of the course material might alert the instructor of issues in comprehension of instruction. It could possibly motivate the students to seek help from their instructor, to seek tutoring early on in the semester, or even prevent drop out (Larson & Ward, 2006; Nicol & Macfarlane-Dick, 2006). Specifically, in the field of special education teacher preparation, it could also provide a model of a progress monitoring tool used in K-12 educational settings.

Vocabulary-matching CBM has become more prevalent at the secondary level, as it has the most potential utility for identifying students with difficulties comprehending and acquiring content area knowledge when compared to other measures, such an oral reading fluency task (reading a passage in 1 min and counting total words read), Maze task (reading a passage and having to determine the correct word in a sentence from a multiple-choice item), or Sentence Verification Technique (reading a passage and then indicating whether a group of sentences are congruent with the passages just read; Espin & Foegen, 1996; Mooney & Lastrapes, 2016). More recently, vocabulary-matching CBM measures have been developed as a method to identify and monitor student performance in content areas that include a significant amount of content-specific vocabulary, such as social studies (Beyers, Lembke, & Curs, 2013; Espin, Busch, Shin, & Kruschwitz, 2001; Espin, Shin, & Busch, 2005; Lembke et al., 2017) and science (Borsuk, 2010; Espin et al., 2013).

Vocabulary-Matching Curriculum-Based Measurement

Vocabulary-matching CBM measures require a student to demonstrate vocabulary knowledge by matching a set of 20 words with their respective definitions on a given form, usually within 5 min. Multiple forms are created after identifying a pool of terms in a specific content area. Each form is then given on a regular basis (typically weekly) to assess student comprehension of academic language used within the curriculum (Espin et al., 2001; Espin et al., 2005).

Prior to embedding vocabulary-matching CBM measures into content areas, an important first step is to determine the technical adequacy (i.e., reliability) of these measures (Fuchs, 2004). Previous studies examining middle school samples have reported alternate form reliability coefficients ranging from .64 to .77 for adjacent forms and .84 to .89 for combined adjacent forms (i.e., the mean of 2 weeks; Beyers et al., 2013; Espin et al., 2013; Espin et al., 2001; Lembke et al., 2017).

While vocabulary-matching CBM has been shown to have promise in secondary settings, there has been limited exploration of these measures at the postsecondary level. Since many secondary and postsecondary students need remediation in reading skills (Biancarosa & Snow, 2006), it is possible that many college students come with strong decoding skills but not always adequate comprehension skills. Therefore, vocabulary-matching CBM measures that identify and monitor students for linguistic comprehension may prove beneficial, especially within introductory courses that present new terms, definitions, and concepts (Larson & Ward, 2006). At this time, only one study has expanded using such a tool into postsecondary settings.

Larson and Ward (2006) examined the use of weekly vocabulary-matching CBM in two undergraduate introductory psychology courses with 69 undergraduate students at a historically black university. Student readability was determined using a college reading entrance exam for each of the class sections. Both sections were found to demonstrate comparable reading levels, around the 39th percentile. A series of nine vocabulary-matching forms were created by assigning numbers to pages of the glossary and then to individual definitions in sequential order. A random number generator was used to determine three numbers that would correspond with the page of the definition, the specific term and definition, and the order of the sequence on the form that the definition would appear. Definitions were not repeated on the same form. Each CBM contained 20 terms on the left and 21 definitions on the right with one definition serving as a distractor. Definitions were numbered, and students were required to write the number of the definition next to the correct term. Vocabulary-matching CBMs were administered once per week (i.e., 9 weeks) over the course of the semester. During weekly administrations, students were first given a personal graph containing their previous scores of items correct and then CBMs were administered for 5 min. Upon completion, participants switched with a peer who counted the number of correct matches and then returned the form to the participant. Participants subsequently graphed their own data in an effort to determine progress. These data were then reviewed by the researchers to ensure accuracy (Larson & Ward, 2006).

According to Larson and Ward (2006), a regression trendline was calculated for each participant graph and then, using visual analysis, the direction of the graph (i.e., 1 = ascending, 2 = descending, or 3 = horizontal) was determined. Over the course of the semester, 72% of students displayed ascending, 23% descending, and around 5% horizontal trendlines. The reading scores were then correlated with the trend direction resulting in a weak relationship (r = .05). Slopes were also correlated with participant reading scores and demonstrated not only a weak but inverse relationship (r = −.13). When examining trendlines and final grades, Larson and Ward found that the majority of students demonstrated ascending trends regardless of grade earned. They also determined that horizontal trends were only seen in students who obtained an “A” in the course. Descending trendlines made up around 20% of students earning A and B grades and 33% of students earning a C grade. It is important to note that the majority of the sample earned an A, B, or C and only one student earned an F in the course.

Larson and Ward (2006) reported that while trend lines were not related to reading scores there was some promise when looking at the relationship between overall grades in the course and CBM trend line. Participants’ data also revealed ascending trends indicating growth in performance over the semester. By having students grade and graph their data from vocabulary-matching CBM, Larson and Ward provided an opportunity for self-monitoring comprehension of terms and a way for engagement in learning. Using this type of assessment system provided many potential benefits, as it could allow instructors and students to check on performance over time and use data to change study habits, instruction, or program emphases.

While the findings from this study were promising, regrettably, Larson and Ward (2006) did not investigate the reliability of these measures with an undergraduate population. Determining the reliability of CBMs is an important first and second step in the development of general outcome measures (Fuchs, 2004). The authors called for an extension of this line of inquiry as both instructors and students have a great need for formative indicators similar to those used in K-12 settings to help with increasing student achievement and retention rates (Larson & Ward, 2006).

In summary, vocabulary acquisition continues in higher education (Hennings, 2000) and introductory courses provide foundational knowledge critical for students to acquire prior to moving on to more advanced coursework. This is particularly important for teacher candidates who must learn content knowledge and discipline specific language (Kennedy et al., 2016). Therefore, we view foundational vocabulary knowledge in an introduction to special education course as worthy of assessing with vocabulary-matching CBM. Finally, due to Larson and Ward (2006) being the only investigation of vocabulary-matching CBM in a postsecondary setting along with the absence of examining the reliability of the measures, more research is warranted. As a result, the purpose of the current study was to further this research by exploring the reliability of vocabulary-matching CBM measures in an introduction to special education course. The following research question was addressed: is there evidence of alternate form reliability of vocabulary-matching measures in an Introduction to Special Education course?

Method

Participants

Participants were students enrolled within an initial teacher certification program at a public, liberal arts institution in the Midwest. All students were required, when beginning this program, to take an introduction to special education course that addressed exceptional learners. A total of 125 students participated across three sections of the course offered in three academic semesters. However, only students with full data sets (i.e., completed the CBM each week) were included in the analysis. The second author taught all sections of the course and administered all assessments. The final sample consisted 84 students (68 females) with ages ranging 20 to 33 years (M = 22.1). The ethnicity of the sample was predominately European American and overall grade point average requirement for the initial certification program is 2.75. Participants included students in teacher preparation programs for Elementary (33%), Secondary (26%), Special Education (7%), and Other areas (34%).

Measures

Vocabulary-matching CBM

Vocabulary-matching CBMs were created similarly to previous research (Espin et al., 2001). Totally, 272 terms were collected from the required course textbook glossary and respective definitions were modified to be 15 words or less. The second and fourth authors served as content experts due to familiarity with introductory special education course content and having expertise in teaching. Each content expert reviewed the terms and definitions separately by selecting terms that best addressed course content. Each term was given a rating of 1 to 3, with 1 being highly relevant to the course, 2 being moderately relevant, and 3 being not relevant at all to the content. The Krippendorff’s alpha coefficent was used (Hayes & Krippendorff, 2007) to estimate intercoder reliability. The results demonstrated high intercoder reliability (.86) from the content experts. To be included in the overall list of terms and definitions for the CBMs, terms had to have an average rating of two and a half or lower. The final list of terms meeting these criteria included 200 words and definitions.

As done with previous vocabulary-matching CBM development, using an online number generator, vocabulary terms and definitions were each randomly assigned a weekly number between 1 and 10. As a result, 10 forms were created by randomly selecting 20 terms. Terms were placed on the left side of the form and were organized in alphabetical order (see Figure 1). Twenty definitions were placed on the right side of the form and along with two distracter definitions randomly chosen from other weekly forms that were not directly adjacent to that weekly form (i.e., a randomly chosen definition from form 1 would be placed on form 8 as a distractor item). CBM forms were group administered for 5 min. A raw score of the total number correct for each form was calculated.

Figure 1.

Example vocabulary matching probe.

Procedures

After informed consent was obtained during the first class meeting of each section of the course, the vocabulary CBMs were administered at the beginning of a class session each week for 10 weeks. The first form was given during the second class of the first week of semester after the precontent assessment. Forms two through 10 were given once a week for the remainder of the semester. Administration directions were adapted from the work of Espin (n.d.). All assessments were administered via paper-pencil and collected. The administration process included the following standardized steps:

At the beginning of the class, the researcher passes out the CBM form to the students, face down.

The researcher reads the directions, “When I tell you to begin, you will have five minutes to match as many vocabulary words with their definitions as you can. Write the letter of the definition in the blank next to the vocabulary word to which it refers. There are two more definitions than there are words. If you aren’t sure which definition matches with the word, skip it and go to the next definition. There is no penalty for guessing. At the end of five minutes, I will stop you. When I say stop, please turn your vocabulary sheet over and put your pencil down. If you finish before the five minutes is up, re-check your answers and sit quietly until the five minutes is over. I won’t be able to help you during this time. Do you have any questions before we begin? Please do your best.”

The researcher says, “Begin.” And starts the 5-min timer.

After 5 min, the researcher says, “Stop, thank you.” and collects the forms.

The second author and a graduate student research assistant scored CBMs. After each section of assessments were scored by the graduate assistant, the researcher would randomly choose 10% of the papers to rescore as a measure of fidelity and interrater reliability. These fidelity checks resulted in 100% interrater reliability.

Data Analytic Plan

Data were entered into SPSS v.24.0 for analysis. Descriptive statistics (mean, SD, range, skewness, kurtosis) were calculated for all measures. To examine evidence of alternate form reliability, Pearson’s product-moment correlations for the full sample were calculated with adjacent (1 vs. 2, 2 vs. 3, etc.) and combined adjacent (mean of 1 + 2, mean of 3 + 4) CBM forms to determine alternate form reliability to examine the stability of the measure over time (Shin, Deno, & Espin, 2000). Given the exploratory nature of this study, differences in reliability coefficients between genders and age groups (20-22 and 23-33) were examined. A Fisher’s Z transformation was used to transform correlation coefficients and differences were analyzed using a z test for independent samples (Rosenthal & Rosnow, 1991). Magnitude of coefficients was interpreted using Cohen’s (1988) guidelines (low 0.2-0.4, moderate 0.5-0.7, strong >.70).

Results

Descriptive Data

Descriptive statistics were calculated to determine mean, standard deviation, range, skewness, and kurtosis of the CBMs and the content assessment (see Table 1). Visual inspection of the CBM data histograms and skewness and kurtosis values revealed that the measure for Week 3 was leptokurtic, but not enough to warrant transformations of the data. All CBMs except for form 1 were negatively skewed to some degree, which may indicate a ceiling effect. The mean number of correctly matched items on the CBM task had a positive trend.

Table 1.

Descriptive Data for All Measures.

Measure	N	M	SD	Min	Max	Skewness	Kurtosis
Week 1	84	10.02	3.23	4	18	0.20	−0.32
Week 2	84	15.17	3.62	7	20	−0.37	−0.94
Week 3	84	16.93	2.99	7	20	−1.48	2.66
Week 4	84	16.58	2.80	7	20	−0.89	0.78
Week 5	84	14.83	3.66	6	20	−0.51	−0.38
Week 6	84	17.02	3.10	7	20	−0.95	0.17
Week 7	84	16.08	3.29	6	20	−0.62	−0.22
Week 8	84	16.19	2.91	6	20	−1.08	1.44
Week 9	84	15.96	3.10	8	20	−0.44	−0.68
Week 10	84	16.18	2.89	6	20	−0.79	0.42
Mean 1 + 2	84	12.60	3.01	6	19	−0.02	−0.72
Mean 3 + 4	84	16.76	2.38	9	20	−0.94	0.31
Mean 5 + 6	84	15.93	3.03	7	20	−0.81	0.17
Mean 7 + 8	84	16.13	2.66	9	20	−0.60	−0.56
Mean 9 + 10	84	16.07	2.61	9	20	−0.78	−0.15

Note. SD = Standard Deviation, Min = Minimum, Max = Maximum.

Reliability

Pearson product-moment correlations were conducted with adjacent and combined adjacent CBM forms to determine evidence of reliability across forms (see Tables 2 and 3). For the full sample, adjacent forms had moderate alternate form reliability coefficients (r = .35-.62, p < .01) with a mean of r = .45, while combined adjacent forms (mean of 1 + 2, mean of 3 + 4) had moderate coefficients (r = .57-.71, p < .01) with a mean of r = 65. Combined forms provided larger reliability coefficients than single forms.

Table 2.

Alternate Form Reliability Coefficients for the Full Sample (N = 84).

Adjacent Forms	Coefficient	Combined Adjacent Forms	Coefficient
Week 1 and Week 2	.54**	Weeks 1 + 2 and Weeks 3 + 4	.66**
Week 2 and Week 3	.53**	Weeks 3 + 4 and Weeks 5 + 6	.57**
Week 3 and Week 4	.35**	Weeks 5 + 6 and Weeks 7 + 8	.71**
Week 4 and Week 5	.35**	Weeks 7 + 8 and Weeks 9 + 10	.68**
Week 5 and Week 6	.61**
Week 6 and Week 7	.51**
Week 7 and Week 8	.48**
Week 8 and Week 9	.62**
Week 9 and Week 10	.52**
Overall M	.45	Overall M	.66

p ≤ .05. **p ≤ .01.

Table 3.

Alternate Form Reliability Coefficients for Gender and Age Groups.

CBM	Gender				Age
Forms	Male(n = 16)	Female(n = 68)	z	P	20-22(n = 65)	23-33(n = 19)	Z	p
W1 and W2	.58**	.53**	0.24	.81	.45**	.80**	−2.19	.02*
W2 and W3	.42	.57**	−0.66	.51	.61**	.38	1.1	.27
W3 and W4	.40	.32**	0.30	.76	.41**	.37	0.17	.87
W4 and W5	.59*	.26*	1.35	.17	.29*	.62**	−1.52	.12
W5 and W6	.71**	.53**	0.98	.32	.55**	.78**	−1.52	.13
W6 and W7	.56*	.51**	0.23	.82	.57**	.31	1.17	.24
W7and W8	.50*	.47**	0.13	.90	.60**	.17	1.86	.06
W8 and W9	.56*	.64**	−0.41	.68	.62**	.61**	0.06	.95
W9 and W10	.71**	.47**	1.24	.22	.43**	.75**	−1.83	.07
W1 + 2 and W3 + 4	.58*	.68**	−0.55	.58	.71**	.52*	1.11	.27
W3 + 4 and W5 + 6	.53*	.58**	−0.24	.81	.59**	.50*	0.46	.65
W5 + 6 and W7 + 8	.54*	.76**	−1.29	.19	.70**	.75**	−0.38	.70
W7 + 8 and W9 + 10	.73**	.68**	0.33	.74	.64**	.80**	−1.21	.22

Note. W = Week, & = and. CBM = Curriculum-Based Measurement.

p ≤ .05. **p ≤ .01.

When examining reliability coefficients between subgroups of the sample, no significant differences were found between male and female students. On average, both male and female students produced moderate reliability coefficients (r = .57 and .54, respectively). However, there were two instances where reliability coefficients fell within the low range and were not significant for male students (see Table 3).

When examining differences between age groups, coefficients between forms 1 and 2 demonstrated a statistically significant difference between ages 20 to 22 (r[65] = .45) and ages 23 to 33 (r[19] = .80), z = -2.19, p = .02. On average both age groups produced moderate coefficients (r = .55 and .57, respectively). However, for the older age group, four instances where reliability coefficient fell within the low range and were not significant (see Table 3). Again, the mean of two forms provided larger reliability coefficients than when comparing single forms for all subgroups.

Discussion

The purpose of this study was to explore the use of vocabulary-matching CBM in a college classroom, specifically an introduction course in special education. We sought to determine if these measures demonstrate evidence of alternate form reliability at the postsecondary level. In this section, findings as well as limitations are discussed, along with future directions.

Overall, the alternate form reliability was lower than in previous vocabulary-matching CBM studies conducted at the secondary level. At the postsecondary level, the vocabulary-matching CBMs demonstrated evidence of moderate alternate form reliability and the coefficients for the combined adjacent forms were on average larger (r = .65) than the single adjacent forms (r = .49). These findings are lower overall than in previous research where coefficients of .80 and above were common (Beyers et al., 2013; Espin et al., 2001; Espin et al., 2005). However, this study aligns with previous research finding that combined adjacent forms provide larger reliability coefficients and thus a more stable estimate of student performance than single forms or test scores (Beyers et al., 2013; Lembke et al., 2017). This is important for instructors when considering using formative assessment or data-based instruction at the postsecondary level. While having enough data points to accurately estimate and track student performance is essential, this study adds to the emerging evidence that using averages or combined adjacent forms may provide a clearer and more reliable picture of student performance.

When examining reliability evidence for specific subgroups, it appears that the CBMs had a few instances where groups differed in the magnitude of coefficients but only one instance between older and younger students was found to be statistically significant. We attribute this difference to older students coming into the course with more background knowledge or educational experience. However, this trend does not continue over time. It is likely that the small sample sizes in groups (i.e., males and older students) had an impact on lower reliability coefficients. Given the low to moderate magnitude and inconsistent coefficients across subgroups it appears the measures are not reliable for use within this specific population.

It is possible that the nature of university-level instruction had some impact on participant performance and the reliability results in the current study as compared to student performance in vocabulary matching studies at the secondary level. Middle and high school content classes typically meet daily for an entire school year while college classes meet less frequently across only one semester. In addition, college coursework requires students to be much more independent; students must monitor their own reading, class attendance, and learning. It could be that more stable scores were observed in the secondary level studies due to some combination of instructional frequency, setting, and course content that impacted performance on the CBMs throughout the studies. Perhaps one college semester of 15 to 16 weeks is simply not long enough to obtain reliable scores at the postsecondary level using a vocabulary matching measure.

Limitations and Future Directions

This study is a first step into examining reliability of CBM that expands on the studies conducted by Larson and Ward (2006). It is also exploring the use of CBM as a formative assessment tool within the college classroom. It appears that, based on the results of this study, significant additional research is necessary to investigate the utility of vocabulary-matching CBM in tracking college student’s knowledge in an introductory course. It is important to consider that college classrooms function differently than K-12 classrooms in that students are coming equipped with skills and background knowledge. Since students enrolled in the certification program can either be admitted at the undergraduate or graduate level, our sample was also a significant limitation. Unfortunately, we were unable to access this information and we recognize that differences in this data may be due to varying levels of background knowledge due to the level of experience or number of years in education. As a result, we suggest future studies should be conducted to address measurement concerns by replicating this study with collecting data more frequently (e.g., every week in a semester for a total of 15-16 data points) with a sample of undergraduate students. This should ideally improve the reliability coefficients or at least give a clearer picture of the stability of the data obtained from the CBMs.

In addition, the lack of proximal outcome or criterion measures that align with specific course content limits the ability to validate these measures. At the postsecondary level, while there is the possibility of using teacher candidates’ scores on their certification exams for teaching licensure as an outcome measure, there is no guarantee that the special education content covered in the introduction to special education course will be present on the certification exam, particularly for general education majors or those students not seeking certification in special education. Future studies could investigate the criterion validity evidence of vocabulary-matching CBMs with certification exams to determine if this would be a viable option for examining evidence of validity.

It is also possible that vocabulary-matching CBM on its own is not enough to truly capture content knowledge acquisition that occurs over a college semester course, and perhaps investigating other CBMs that capture related skills like reading comprehension would be appropriate. University educators should note that CBM use at the postsecondary level is not meant to necessarily capture discrete skills, such as lesson planning, but rather is a general outcome measure meant to tap overall content knowledge. Perhaps future studies in teacher education could investigate using a combination of general outcome and discrete performance measures to more precisely capture teacher candidate learning.

Additional Considerations for CBMs in College Settings

While our results do not suggest at this time that vocabulary-matching CBM is a reliable measure of student performance in a college classroom, perhaps we can conceptualize its use in a different way. As discussed previously, formative assessments at the collegiate level that provide ongoing feedback are capable of producing self-regulated learners. In addition, studies have found that they assist in motivating students to complete class readings and assignments (Cauley & McMillan, 2010; Nicol & Macfarlane-Dick, 2006). If using the definition of formative assessment as an ongoing task that provides feedback on student learning and performance (Yorke, 2003), then a tool such as vocabulary-matching that is used throughout the semester may be helpful for self-regulation or self-monitoring of learning. In the future, it would be helpful to monitor the fidelity of administration of the CBM measures, as well as the examination of the social validity of measures to gain instructors’ perceived utility of assisting students in monitoring their progress in college courses. While this was not addressed within the scope of this study, soliciting instructor feedback about the utility of this information could provide insight as to whether or not instructors or students would actually use this type of feedback to make changes in the course or study patterns, respectively.

In addition, vocabulary-matching CBM in college courses, specifically in the area of education and teacher preparation, could serve not only as a self-monitoring tool in learning new content but also operate as a model for collecting formative assessment data and their involvement in data-based decision-making. The investigation of teacher candidate data-based decision skills based on their involvement in, and the modeling of, formative assessment procedures during their courses may also allow for additional consideration of the utility of CBM at the collegiate level.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Beyers

S. J.

Lembke

E. S.

Curs

(2013). Social studies progress monitoring and intervention for middle school students. Assessment for Effective Intervention, 38, 224–235. doi:10.1177/1534508413489162

Biancarosa

Snow

C. E.

(2006). A vision for action and research in middle and high school literacy: A report to Carnegie Corporation of New York (2nd ed.). Washington, DC: Alliance for Excellent Education.

Borsuk

E. R.

(2010). Examination of an administrator-read vocabulary-matching measures as an indicator of science achievement. Assessment for Effective Intervention, 35, 168–177. doi:10.1177/1534508410372081

Bravo

M. A.

Cervetti

G. N.

(2008). Teaching vocabulary through text and experience in content areas. In Farstrup

A. E.

Samuels

S. J.

(Eds.), What research has to say about vocabulary instruction (pp. 130–149). Newark, DE: International Reading Association.

Brownell

M. T.

Sindelar

P. T.

Kiely

M. T.

Danielson

L. C.

(2010). Special education teacher quality and preparation: Exposing foundations, constructing a new model. Exceptional Children, 76, 357–377. doi:10.1177/001440291007600307

Cauley

K. M.

McMillan

J. H.

(2010). Formative assessment techniques to support student motivation and achievement. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 83, 1–6. doi:10.1080/00098650903267784

Cohen

(1988). Statistical power analysis for the behavioural sciences. Hillside, NJ: Lawrence Erlbaum.

Cuvo

A. J.

(2003). On stimulus generalization and stimulus classes. Journal of Behavioral Education, 12, 77–83. doi:1053–0819/03/0300–0077/0

De Arment

S. T.

Reed

Wetzel

A. P

. (2013). Promoting adaptive expertise: A conceptual framework for special educator preparation. Teacher Education and Special Education, 36, 317–230. doi:10.1177/0888406413489578

10.

Deno

(1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52, 219–232. doi:10.1177/001440298505200303

11.

Deno

S. L.

(2003). Developments in curriculum-based measurement. The Journal of Special Education, 37, 184–192. doi:10.1177/00224669030370030801

12.

Espin

C. A.

Busch

Lembke

E. S.

Hampton

D. D.

Seo

Zukowski

B. A.

(2013). Curriculum-based measurement in science learning: Vocabulary-matching as an indicator of performance and progress. Assessment for Effective Intervention, 38, 203–213. doi:10.1177/1534508413489724

13.

Espin

C. A.

Busch

Shin

Kruschwitz

(2001). Curriculum-based measures in the content areas: Validity of vocabulary-matching measures as indicators of performance in social studies. Learning Disabilities Research & Practice, 16, 142–151. doi:10.1111/0938–8982.00015

14.

Espin

C. A.

Foegen

(1996). Validity of general outcome measures for predicting secondary students’ performance on content-area tasks. Exceptional Children, 62, 497–514.

15.

Espin

C. A.

Shin

Busch

T. W.

(2005). Curriculum-based measurement in the content areas: Using vocabulary matching as an indicator of progress in social studies learning. Journal of Learning Disabilities, 38, 353–363. doi:10.1177/00222194050380041301

16.

Espin

C. A.

(n.d.). Expert connection: Curriculum-based measures: Are there ways to use CBM in content areas? Retrieved from TeachingLD website: http://teachingld.org/questions/12

17.

Fuchs

L. S.

(2004). The past, present, and future of curriculum-based measurement research. School Psychology Review, 33(2), 188–193.

18.

Glazer

(2014). Formative plus summative assessment in large undergraduate courses: Why both? International Journal of Teaching & Learning in Higher Education, 26, 276–286.

19.

Hallahan

D. P.

Kauffman

J. M.

Pullen

P. C.

(2015). Exceptional learners: Introduction to special education (13th ed.). Boston, MA: Allyn and Bacon.

20.

Harmon

J. M.

Wood

K. D.

Hendrick

W. B.

(2008). Vocabulary instruction in middle and secondary content classrooms: Understandings and direction from research. In Farstrup

A. E.

Samuels

S. J.

(Eds.), What research has to say about vocabulary instruction (pp. 150–181). Newark, DE: International Reading Association.

21.

Hatano

Inagaki

(1986). Two courses of expertise. In Stevenson

Azuma

Hakuta

(Eds.), Child development and education in Japan (pp. 262–272). New York, NY: Freeman.

22.

Hayes

A. F.

Krippendorff

(2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77–89.

23.

Hennings

D. G.

(2000). Contextually relevant word study: Adolescent vocabulary development across the curriculum. Journal of Adolescent & Adult Literacy, 44, 268–279.

24.

Hosp

M. K.

Hosp

J. L.

Howell

K. W.

(2016). The ABCs of CBM: A practical guide to curriculum-based measurement (2nd ed.). New York, NY: Guilford.

25.

Kennedy

M. J.

Ihle

F. M.

(2012). The old man and the sea: Navigating the gulf between special educators and the content area classroom. Learning Disabilities Research & Practice, 27, 44–54.

26.

Kennedy

M. J.

Wagner

Stegall

Lembke

Miciak

Alves

K. D.

. . . Hirsch

S. E.

(2016). Using content acquisition podcasts to improve teacher candidate knowledge of curriculum-based measurement. Exceptional Children, 82, 303–320.

27.

Knight

P. T.

(2002). Summative assessment in higher education: Practices in disarray. Studies in Higher Education, 27, 275–286. doi:10.1080/03075070220000662

28.

Larson

W. C.

Ward

(2006). Developing a predictor of university students’ course grades using curriculum-based measurement: An initial investigation. Journal of College Reading and Learning, 37, 45–60. doi:10.1080/10790195.2006.10850192

29.

Lembke

E. S.

Allen

Cohen

Hubbuch

Landon

Bess

Bruns

(2017). Progress monitoring in social studies using vocabulary matching curriculum-based measurement. Learning Disabilities Research & Practice, 32, 112–120. doi:10.1111/ldrp.12130

30.

Lemke

J. L.

(1990). Talking science: Language, learning, and values. Norwood, NJ: Ablex.

31.

Mooney

Lastrapes

R. E.

(2016). The benchmarking capacity of a general outcome measure of academic language in science and social studies. Assessment for Effective Intervention, 41, 209–219. doi:10.1177/1534508415624648

32.

Nicol

D. J.

Macfarlane-Dick

(2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31, 199–218. doi:10.1080/03075070600572090

33.

Phythian-Sence

Wagner

R. K.

(2007). Vocabulary acquisition: A primer. In Wagner

R. K.

Muse

A. E.

Tannenbaum

K. R.

(Eds.), Vocabulary acquisition: Implications for reading comprehension (pp. 1–14). New York, NY: Guilford Press.

34.

Rosenthal

Rosnow

R. L.

(1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York, NY: McGraw-Hill.

35.

Salvia

Ysseldyke

Bolt

(2012). Assessment: In special and inclusive education. Cengage Learning. Boston, MA.

36.

Scheeler

M. C.

Budin

Markelz

(2016). The role of teacher preparation in promoting evidence-based practice in schools. Learning Disabilities: A Contemporary Journal, 14(2), 171–187.

37.

Shin

Deno

S. L.

Espin

(2000). Technical adequacy of the maze task for curriculum-based measurement of reading growth. The Journal of Special Education, 34, 164–172. doi:10.1177/002246690003400305

38.

Yorke

(2003). Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. Higher Education, 45, 477–501. doi:10.1023/A:1023967026413