An Examination of Differential Item Functioning in a Rubric to Assess Solo Music Performance

Introduction

Solo and ensemble festivals are an important part of the music education experience for many secondary-level music students across the USA. From a student perspective, participation in adjudicated events, such as solo and ensemble festivals, can increase quality of musicianship, improve motivation, and increase self-efficacy (Austin, 1988; Banister, 1992; Franklin, 1979; Howard, 1994; Hurst, 1994). From a music program perspective, the results of student participation in adjudicated events can impact teaching decisions related to curricular content, learning objectives, and educational goals (Abeles, Hoffer, & Klottman, 1994; Crochet, 2006; Howard, 2002). From a community perspective, student achievement in adjudicated events not only affects perceptions of teacher and program quality, but these perceptions are often a catalyst for a program’s continued participation with a particular focus on achieving high-level results and related accolades (Boyle, 1992; Kirchhoff, 1988).

The significance and importance of these adjudicated events in the landscape of secondary-level music education yields a broad and significant amount of research investigating the psychometric properties of their scoring systems as well as extraneous variables that may affect scoring outcomes. Historically, these psychometric inquiries most often include reliability considerations using some form of inter- or intra-rater reliability coefficients in the context of true score (i.e., Classical Test Theory) testing traditions (Bergee, 2003; Bergee, 2007; Bergee & McWhirter, 2005; Bergee & Westfall, 2005; Ciorba & Smith, 2009; Conrad, 2003; Fiske, 1975; Fiske, 1983; Hash, 2012; King & Burnsed, 2009; Latimer, Bergee, & Cohen, 2010; Norris & Borst, 2007; Saunders & Holahan, 1997; Silvey, 2009, for example). Investigations into extraneous variables that may have an overall effect on rating results historically fall into two broad categories: performer-centered variables and contextual variables (Wesolowski, Wind, & Engelhard, Jr., 2015). Examples of performer-centered variables include expressive communication via body movement (Davidson, 1993, 1994, 1995; Davidson & Coimbra, 2001); physical attractiveness (Bermingham, 2000; Davidson & Coimbra, 2001; Wapnick, Darrow, Kovacs, & Dalrymple, 1997; Wapnick, Mazza, & Darrow, 1998, 2000); musical expression (Schubert, 2002); stage entrance and facial expressions (Waddell & Williamon, 2017); and performing from memory (Kopiez, Wolf, & Platz, 2017). Examples of contextual variables include the environmental effect on acoustics (Ando, 1988), and social factors (McPherson & Thompson, 1998). Additionally, more recent investigations into rater behavior have suggested that, from a cognitive perspective, variations in audio and visual presentation cues can affect performance assessment outcome scores (Griffiths & Reay, 2018; Krath, Hahn, & Whitney, 2015; Platz & Kopiez, 2012; Tsay, 2015).

Fairness and Differential Item Functioning

According to the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014, p. 50) (herein referred to as the Standards), fairness is defined as “responsiveness to individual characteristics and testing contexts so that test scores will yield valid interpretations for intended uses.” More broadly, fairness is a validity issue concerned with the degree to which measurement procedures result in accurate estimates of student ability. Notably, measurement equivalence, or “lack of measurement bias,” is one of the four general views of fairness highlighted by the Standards (2014, pp. 51–52):

Characteristics of the test itself that are not related to the construct being measured, or the manner in which the test is used, may sometimes result in different meanings for scores earned by members of different identifiable groups. For example, differential item functioning (DIF) is said to occur when equally able test takers differ in their probabilities of answering a test item correctly as a function of group membership.

Differential item functioning (DIF) is a statistical framework that aids in the detection of systematic patterning of responses based on students’ subgroup affiliations (Holland & Thayer, 1988). More specifically, DIF is said to exist if students with the same estimated ability level from different group affiliations demonstrate different probabilities of success on a specific item.

In the case of music performance assessment, the type of musical instrument a student plays is an important grouping characteristic that may potentially account for systematic differences in outcome scores and violate the principle of test fairness as prescribed by the Standards. Each musical instrument comes with its own unique set of foundational skills, problems, and challenges. As Colwell and Hewitt (2011, p. 1) note:

Mixed-instrument classrooms present unique challenges . . . many of which are unique to each instrument. Poor hand position on the clarinet is not poor hand position on the violin. Carefully monitoring each student on all of the foundational skills . . . requires knowledge of each instrument.

Duke and Byo (2011, p. 9–10) acknowledge that “Double reeds, horns, tubas . . . all present unique problems . . . double reeds are overly finicky for a variety of reasons (e.g., reeds, instrument adjustments) . . . horn . . . is problematic because of its register . . . Tubas simply require a great deal of physical capacity . . .”

In many solo and ensemble performance assessment contexts, the same evaluation/scoring form is used for all students, regardless of the instrument they are playing. Furthermore, students of heterogeneous instrument types are evaluated on the same performance standard of superior, excellent, good, fair, or poor (National Association for Music Education, 2016). From a feedback and evaluation perspective, Duke and Byo (2011, p. 4) noted, “Consider that, in all likelihood . . . feedback about a group’s performance is almost always the wrong feedback for one or more members of the group.” With these considerations in mind, systematic differences in estimates of student ability, due to the interaction between item difficulties and musical instrument, may exist specifically because of the unique performance characteristics and challenges associated with specified musical instruments. Therefore, an investigation into possible DIF effects related to musical instrument is warranted.

In the case of music performance assessment research, DIF related to students’ musical instrument grouping has not been examined in the music assessment research literature. The purpose of this study was to examine DIF in a rubric used to assess middle-school solo and ensemble performances. In particular, this study was guided by the following research questions:

Method

Psychometric Considerations for the MFR-PC Model

The MFR-PC measurement model (Linacre, 1989; Masters, 1982) was used in this study to convert the raters’ unadjusted raw scores to estimated linear measures. More specifically, the probability of raw score responses was modeled as a function of student ability, item difficulty, rater severity, and music instrument classification parameters (Linacre, 1989). The Rasch model was used for two specific purposes. First, the properties of the Rasch family of measurement models are fixed in their invariant measurement requirements for estimating data-to-model fit, making it particularly useful in contexts where raters mediate the assessment (Engelhard & Wind, 2019). The five requirements for rater-mediated invariance include: (a) rater invariant measurement of persons (i.e., the measurement of students must be independent of the particular raters that happen to be used for the measuring); (b) non-crossing person response functions (i.e., a higher-ability student must always have a better chance of obtaining higher ratings from raters than a lower-ability student); (c) person-invariant calibration of raters (i.e., the calibration of the raters must be independent of the particular students used for calibration); (d) non-crossing rater response functions (i.e., any student must have a better chance of obtaining a higher rating from lenient raters than from more severe raters); and (e) variable map (i.e., students and raters must be simultaneously located on a single underlying latent variable) (Engelhard, 2013). Second, as Penfield and Camilli (2007) note, approaches to DIF detection all share common theoretical interpretations rooted in the modeling capabilities of Item Response Theory (IRT). The Rasch family of measurement models, considered part of the same modeling (i.e., scaling) tradition as IRT, was thus a suitable measurement model for this study (Wesolowski, 2019).

The partial credit (PC) version of the measurement model (Masters, 1982) was used to allow the rating scales to vary by each item. Because the rubric was built in a manner where the rating scale categories are empirically unequal in the number of categories and substantively unequal in the difficulty of their content, the PC version of the model was a preferred choice. The PC version of the model allows for the empirical investigation of item differences across each of the item’s individual rating scale structures (i.e., differences in step difficulty across rating scale categories), thereby providing a more detailed true score estimation of model-data fit.

The MFR-PC measurement model used in this study included facets for students, items, raters, and instrument classification. The model is specified as follows:

l n [ P n i j m k P n i j m k − 1 ] = θ n − λ i − δ j − γ m − τ i k ,

(1)

where

ln[P_nijmk/P_nijmk-1] = natural log of the probability that Student n rated by Rater i on Item j for Musical Instrument m receives a rating in category k rather than category k-1,

θ n = the logit-scale location (e.g., ability) of Student n,

λ i = the logit-scale location (e.g., severity) of Rater i,

δ j = the logit-scale location (e.g., difficulty) of Item j,

γ m = the logit-scale location (e.g., difficulty) of Musical Instrument m, and

τ i k = the logit-scale location where rating categories k and k-1 are equally probable for Student n.

Psychometric Considerations for DIF Interaction Model

A post hoc DIF analysis was carried out by adding an additional interaction parameter to the MFR-PC measurement model as specified in Equation 1. In the case of this study, DIF analysis identifies any potential significant differences in student scores across students’ instrument classification at the same estimated ability level. In other words, DIF tested the null hypothesis that students of the same ability did not differ significantly according to the music instrument they happen to play. In the case of this study, the interaction parameter included crossing the item facet with the instrument classification facet. The interaction term ( δ j γ m ) was added to Equation 1 as follows:

l n [ P n i j m k P n i j m k − 1 ] = ( θ n − λ i − δ j − γ m − τ i k ) − δ j γ m ,

(2)

where δ j γ m is the interaction term between item difficulty and music instrument difficulty.

DIF analyses consist of two parts. The first is an omnibus test of the null hypothesis that the overall set of interaction terms between the item facet and the instrument classification facet do not differ significantly from zero. The test statistic that confirms or disconfirms the null hypothesis is a chi-square. If the omnibus test yields a significant chi-square, post hoc testing of individual interactions is conducted to explore meaningful patterns of interactions between individual elements (i.e., pairwise interactions between every item and every instrument grouping to identify where the significant interactions occur). The interaction terms are calculated as t-statistics with a mean of zero and standard deviation of 1 (i.e., a Z-score that approximates a normal distribution). In this study, significant interactions (|Z| ⩾ 2.00) indicate that an item is statistically significantly more difficult or less difficult for a student based on the instrument they play. All data analyses in this study were conducted using the FACETS computer program (Linacre, 2014).

Results

DIF Interaction Model Results

To determine whether measurement equivalence for all items exists when used to measure subgroups of students based upon their musical instrument (research question 1), a DIF omnibus analysis was conducted. The DIF analysis was conducted by crossing the item and instrument classification facets to test the null hypothesis that the overall set of interaction terms between the item facet and instrument classification facet do not significantly differ from zero. The analysis indicated an overall statistically significant item performance based on instrument classification ( χ ² ₍₂₅₂₎ = 634.00, p < .01), enabling the null hypothesis to be rejected. This suggests that measurement equivalence for all items does not exist when used to measure subgroups of students based upon their musical instrument. Table 1 illustrates the mean pairwise differences in average ability scores between instrument groupings.

OPEN IN VIEWER

Table 1.

Mean Pairwise Differences Between Instrument Achievement Scores (In Logit Units).

Instrument	Average measure	Mean differences in achievement
		Flute	Oboe	Clarinet	Saxophone	Trumpet	French horn	Trombone	Euphonium	Tuba
Flute	0.38	–	0.38	0.33	0.38	0.61	0.03	0.88	0.51	0.29
Oboe	0.00		–	−0.05	0.00	0.23	−0.35	0.50	0.13	−0.09
Clarinet	0.05			–	0.05	0.28	−0.30	0.55	0.18	−0.04
Saxophone	0.00				–	0.23	−0.35	0.50	0.13	−0.09
Trumpet	−0.23					–	−0.58	0.27	−0.10	−0.32
French horn	0.35						–	0.85	0.48	0.26
Trombone	−0.50							–	−0.37	−0.59
Euphonium	−0.13								–	−0.22
Tuba	0.09									–

All differences are statistically significant ( χ ² ₍₈₎ = 791.70, p < .01). Substantive differences ⩾ .30 logits (Engelhard & Myford, 2003).

To find out if any patterns of DIF effects exist for items when used to measure subgroups of students based on their musical instrument (research question 2), a post hoc pairwise interaction analysis between all items (n = 28) and all instrument classifications (n = 9) was conducted. Of the 252 possible interaction terms, a total of 57 (22.62%) interactions were found to be statistically significant (|Z| ⩾ 2.00). Supplementary Table 5 provides the individual significant pairwise interactions between items and instrument classification, as indicated by |Z| ⩾ 2.00, presented in order of positive to negative bias index values. The infit mean square (Infit MSQ) column shows how much misfit to the model remains after accounting for the bias. Infit MSQ is expected to be less than 1.00. Any value greater than 1.00 can be interpreted as having further unknown causes of misfit beyond that of the identified DIF effect. The observed minus expected values column (Obs.-Exp.) represents the total observed raw scores minus the total expected raw scores divided by the total observed count of raw scores. Positive values indicate the item was systematically less difficult for the identified instrument classification than expected. Negative values indicate the item was systematically more difficult for the identified instrument classification than expected. The bias index values (Bias) indicate the size of the DIF effect in logit units. Directionality and interpretation of bias index values correspond directly to the observed minus expected values. The expected value of bias indices is 0.00 logits, indicating there was no systematic differential item behavior exhibited. Positive values indicate the item was systematically less difficult for the identified instrument classification than expected. Negative values indicate the item was systematically more difficult for the identified instrument classification than expected. The standard error (SE) represents the standard model error of the bias estimate. The Z-score (Z) is the statistic that tests the null hypothesis that there is no DIF effect other than the standard measurement error. Any value above 2.00 or below -2.00 indicates a significant DIF interaction effect. Note that only the 57 (of a possible 252) significant interactions (|Z| ⩾ 2.00) are reported in the table.

In the context of assessments involving comparatively small sample sizes, it is more advantageous and relevant to evaluate bias index values for items demonstrating statistically significant DIF effects (Linacre, 2018). To identify the size of DIF effects for items used to measure subgroups of students based on their musical instrument (research question 3), bias index values were examined to make substantive interpretations of their effect. Based on the guidelines of Zwick, Thayer, and Lewis (1999), bias indices where |DIF| >/= .64 logits are considered to represent a moderate to large effect, bias indices where |DIF| is in the range of .43 to .63 logits are considered to represent a slight to moderate effect, and bias indices where |DIF| < .43 logits are considered to represent negligible effects. A total of 26 (10.3%) significant interaction terms demonstrated a moderate to large effect where |DIF| >/= .63 logits; nine (3.6%) significant interaction terms demonstrated a slight to moderate effect, where |DIF| = (.43, .63 logits); and 22 (8.7%) significant interaction terms demonstrated a negligible effect, where |DIF| < .43 logits. Figure 1 provides descriptive statistics for the significant pairwise interactions between items and instruments by DIF effect size. Effects represented by |DIF| >/= .43 are highly relevant to considerations of the fairness of assessment outcomes.

OPEN IN VIEWER

Figure 1.

Frequency of significant pairwise interactions between items and musical instruments by differential item functioning (DIF) effect size.

Discussion

The purpose of this study was to examine DIF in a rubric used to assess middle-school solo and ensemble performances. To answer the first research question (does measurement equivalence for all items exist when used to measure subgroups of students based on their musical instrument?), an omnibus test of DIF was used to test the null hypothesis that the overall set of interaction terms between all items and all instruments do not differ significantly from zero. An overall statistically significant interaction between item and instrument facets ( χ ² ₍₂₅₂₎ = 634.00, p < .01) was found, indicating that measurement equivalence did not exist for all items when used to measure subgroups of students based on the particular instrument they happened to play, so the null hypothesis was rejected.

To answer the second research question (what patterns of DIF effects exist for items when used to measure subgroups of students based on their musical instrument?), a post hoc analysis of pairwise interactions between all item elements (n = 28) and all instrument classification elements (n = 9) was conducted. Of the 252 possible interaction terms, a total of 57 (22.62%) of the interactions were found to be statistically significant (|Z| ⩾ 2.00).

To answer the third research question (what size of DIF effects exist for items when used to measure subgroups of students based on their musical instrument?), bias index values were examined to make substantive interpretations of the DIF effect. A total of 26 (10.3%) significant interaction terms demonstrated a moderate to large effect where |DIF| >/= .63 logits, nine (3.6%) significant interaction terms demonstrated a slight to moderate effect, where |DIF| = (.43, .63 logits), and 22 (8.7%) significant interaction terms demonstrated a negligible effect, where |DIF| < .43 logits.

It is important to consider assessment practices in music not just as a mechanism for justifying scoring and validating inferences but, more importantly, as a method for building theories and better understanding psychological constructs. As noted eloquently by Bond & Fox (2015, p. 44):

. . . [We] can take the conventional approach and see this as a test of whether our data correspond with our substantive theory. However, it is more useful if we complement this use with the idea of whether our construct, as expressed in developmental or other theory, fits with our data. Ideally, this is just one part of an epistemologically iterative research program in which theory informs practice and practice informs theory dialectically.

It is clear the unique challenges and pedagogies for both teaching and learning a musical instrument affect the fairness of scoring outcomes when attempting to measure music performances multilaterally. Interestingly, the results demonstrated adequate data-to-model fit as evidenced by the acceptable fit statistics of the MFR-PC model. However, post hoc DIF analyses indicated a multitude of DIF effects. An attempt to understand the significant instrument-by-item interactions substantively is far beyond the scope of this study, and even the broad-based considerations mentioned here are speculative, at best. Follow-up phenomenological investigations into exemplar performances, related raw scores, and rater insights coupled with pedagogical considerations by experts of instrument-specific content would be one step in the right direction for explaining the particular results from both scholarly and pedagogical perspectives.

Solo and ensemble festivals, such as the one examined in this study, take place regularly and frequently each year throughout the USA. On the evidence of the challenges to fairness revealed by the findings of this study, it may well be that similar challenges pervade other similar assessment contexts. According to current frameworks whereby raw scores are used instead of estimated measures, performance standard classifications are assigned on the basis of summed raw scores equilaterally across all instrument types, judges mediate the assessment environment without controlling for judge severity, and DIF is not a consideration, fairness cannot be guaranteed. Such frameworks do not currently permit investigations into the functionality of measurement tools. These could be improved by examining data-to-model fit and conducting DIF analyses, important methods for monitoring and maintaining the overall stability of measurement tools. Wright and Masters (1982, p. 114) note that “[when] a variable is used with different groups of persons or to measure the same persons on different occasions, it is essential that the identity of the variable be maintained from one occasion to the next” and, as Wright (1999, p. 704) argues, “It is up to us [the test developers] to construct and maintain [tests].”It is hoped that this vision may be realized in future.

A major consideration for the field of music assessment is how best to move forward in providing high-quality formal assessments for students. It is clear, from a pedagogical perspective, that different instruments make different demands on those who play them and, from a psychometric perspective, performances on different instruments are scored differently. These differences need to be acknowledged and accounted for in assessment practice. It is suggested that researchers and assessors work together to examine and reflect on current assessment paradigms and consider implementing unique measurement tools designed to evaluate individual instruments. A more focused approach whereby the challenges and idiosyncrasies of assessing performance on specific instruments are considered empirically may produce not only fairer assessment procedures but also better, more relevant formative feedback that is directly transferable to improved teaching and learning in relation to each instrument.

When considering improved approaches to formative assessment, one psychometric solution might be to implement a diagnostic classification modeling (DCM) approach to large-scale music assessment processes (Rupp, Templin, & Henson, 2010). With the use of Rasch measurement theory or other families of measurement models under the umbrella of IRT, students are assigned a single score along a ‘continuum’ representative of the particular latent construct being measured (e.g., from less to more music performance ability). This approach is arguably summative in nature. DCMs, by contrast, are a class of psychometric models that can characterize the traits of student performances as categorical latent constructs (Bradshaw, 2016). DCM approaches are thus more diagnostic than summative, as students are provided with a profile of proficiency attributes representing a meaningful pathway toward mastery of skill content rather than a single score. Such approaches may be better suited to the expectations of students participating in solo and ensemble festivals, in relation to formative feedback.

Lastly, it is important that fairness guidelines should be taken into consideration when conducting formal music assessments. The Educational Testing Service’s (ETS) Guidelines for Fairness Review of Assessments (2009) and International Principles for the Fairness of Assessments (2016) are just two of many such sets of guidelines that can be helpful in promoting fairness when designing and implementing formal assessment measures. Most fairness guidelines are written from a high-stakes, cognitive-exam perspective, where cognitive considerations (i.e., related to topics and content), affective considerations (i.e., related to eliciting strong and distracting emotions), and physical considerations (i.e., related to physical barriers in items and stimulus materials) are needed when an examinee takes an exam. Many music assessments are performance based, therefore considerations of the rater’s interaction with the items/criteria may be more relevant when considering fairness in the design of an assessment. Therefore, some ETS guideline considerations, specifically for writing assessment items, may be most relevant and appropriate to the context of music performance assessment. Some examples include:

Considering fairness, both qualitatively and quantitatively, and taking action to overcome fairness challenges in protocols for assessing music performance are important ways of enhancing the quality of programs and overall validity of inferences regarding student performances. Indeed they may be considered essential for improving current music assessment research and practice. From a scholarly perspective, continued investigations and improved training in graduate programs is a fundamental step towards improving the science of psychometrics as it relates to the field of music assessment. Graduate programs regularly train students in statistics, but often gloss over or completely ignore training in measurement theory. From the perspective of a practitioner, it is vital to hold discussions with teachers and music administrators about the impact of validity, reliability, and fairness on inferences of student ability if current practice is to change for the better.

Supplemental Material

Supplemental Material

Supplemental Material

Supplemental Material

Supplemental Material