The development of formative assessment rubrics for enhancing students’ performance on Thai percussion instruments

Abstract

This research aimed to develop formative assessment rubrics for enhancing students’ performance on Thai percussion instruments using the Many-Facet Rasch Measurement Partial Credit model (MFRM-PCM). Samples were high-school students playing four types of Thai instruments and the raters who were qualified and properly trained. The research instrument was rubrics used for assessing students’ Thai percussion instruments skills. Each item contained five quality levels. The research methodology was divided into three phases, that is, the synthesis and verification of content validity, pre-testing of the psychometric properties and the assessment design, and the verification of the rubrics’ psychometric properties. Four facets of the MFRM-PCM including students, raters, musical instruments, and items facet were employed to analyze the psychometric properties of the developed rubrics. It was found that the rubrics developed for assessing performances of Thai percussion instruments with the MFRM-PCM contained eight criteria and 12 items. The quality levels of the items after the category’s effectiveness analysis were reduced ranging from two to three quality levels.

Keywords

Formative assessment Many-Facet Rasch Measurement Partial Credit Model music performance assessment rubrics Thai music performance

Introduction

Nowadays, the assessment of students’ performances of Thai percussion instruments is based on the opinions of experts or a committee of judges and basic criteria determined by the Ministry of University Affairs (2001). However, the basic criteria outlined by the Ministry of University Affairs in 2001 were narratively written and only described the appropriate skills without providing specific steps for each performer’s level, which are crucial for the users to determine their current skill levels (Moskal, 2000).

Although efforts have been made to develop an assessment framework that aligns with the basic criteria outlined, there was lacking an appropriate development process, which hindered a broader acceptance of the framework (Chuppunnarat et al., 2018).

Based on additional research related to the development of assessment tools for Thai musical instrument skills, some showed a deficiency in the appropriate process for evaluating psychometric properties such as validity and reliability, in accordance with established principles of measurement and evaluation (Lertsiri, 2014; Lertwiriyapiti, 2020), while some limited to Classical Test Theory (CTT) (Boonlue, 2004). CTT may not be suitable for developing standard assessment tools, as its psychometric properties can vary depending on the sample groups being assessed due to estimating by the mean of samples (Fan, 1998). So, it is important to ensure the invariant psychometric properties of the standard rubrics since it confirms the validity and consistency of skills assessment and is applicable across diverse samples. The researcher, thus, saw the importance of developing assessment tools with invariant psychometric properties to assess Thai students’ performances that are fitting and provides specific steps as helpful feedback.

Tools for assessing musical skills

A music skill is categorized as a performance skill linked to cognitive and mental abilities. Assessing a musical skill is significantly challenging for music educators because music performance is a practical skill that requires assessing both objective skills, such as instrumental techniques, and subjective skills, such as aesthetic expression during the performance (DeLuca & Bolden, 2014). Additionally, the methods, tools, and raters of the musical skills are unique from any other practical skill (Wesolowski et al., 2017).

One of the suitable tools for assessing musical skills is a rating scale, which provides multiple rating scales for each item (polytomous scale). However, a limitation of the rating scale is that it only provides the levels of the skills assessed, but still lacks the information and feedback on each level for users (Wesolowski, 2012). It is probably because rating scales only provide quantifiers or frequency of a person’s skills, which may cause vagueness. Information or feedback obtained from the assessment is essential to instructors because it helps them determine how to improve the teaching techniques’ quality, assess the teaching design’s effectiveness, and inform areas of improvement for learners (Edwards et al., 2019).

Rubrics, sometimes also referred to as the analytic rating scale or the holistic rating scale (Carr, 2000), are the tools that can identify the performers’ skill levels and provide explicit descriptions for each level, offering insight into why a specific level was assigned by providing information and feedback from assessing practical skills or complex performances, which are difficult to assess (Jonsson & Svingby, 2007; Wesolowski, 2012). These features make rubrics particularly conducive to formative assessment (Brookhart, 2013; Mertler, 2001) even though they can be designed to suit formative, summative, or self-assessment (Jonsson & Svingby, 2007; Panadero & Jonsson, 2013). Using rubrics for assessment can motivate learning, self-directed, and self-perception in the learners themselves (Panadero & Jonsson, 2013), and also help encourage learners to reflect on themselves and set the goals they want to achieve in the future (Pancorbo et al., 2020). In addition, rubrics are suitable for assessing musical skills because they can be designed to objectively assess both the technical and aesthetic expressions, and improve the assessment’s consistency. Learners and instructors can use their results to improve the aesthetic aspect, reflect on their skills, and control themselves during the rehearsal (DeLuca & Bolden, 2014).

Rubrics are a type of educational assessment tool that requires examining the psychometric property, particularly validity and reliability, to ensure their quality. There are three types of validity that are most used for developing rubrics, that is, content, construct, and external validity (Brookhart & Chen, 2014; Jonsson & Svingby, 2007). Content validity means rubrics containing criteria, items, and descriptions are comprehensive and representative of skills. Construct validity means rubrics can assess skills in a way that aligns with the quantitative evidence obtained from the measurement model analysis. External validity means the rubrics assess the performance correlated to the other similar-target instruments. However, because of lacking other Thai music-assessing instruments to compare, this validity would not be included in this study. Reliability means the rubrics can maintain their construct validity when applied to other target groups.

The concept of rubrics development

For the formative purpose, the analytics rubrics are more suitable than the holistic rubrics because it separates overall performance into each skill. Users can get information or feedback on each skill and use it to identify strengths and areas of improvement intensively (Brookhart, 2018; Jonsson & Svingby, 2007; Mertler, 2001).

The essential elements of rubrics include criteria, levels or categories, and descriptions (Moskal, 2000). Each skill is separated into criteria, and each criterion may have more than 1 item. However, as noted by DeLuca and Bolden (2014), even though separating skills into subordinate criteria helps assess the skills easier and objectively, performing music involves integrating multiple skills at once. For example, correctly hand position on an instrument can enhance the natural expression of sound. Separating these skills into different criteria may hinder students from understanding how they fit together and hinder their ability to use assessment results for self-assessment and improvement purposes.

The category or level section of rubrics indicate the expected skill level for each criterion. The number of levels is subjective and can vary based on what is to be assessed. Even though there is no universal rule to determine the appropriate number of them, some suggest that it depends on how much distinction of skills developers desire or how much that skills can be distinguished (Moskal, 2000). However, according to Brookhart (2018), the common number of levels is three to five. Therefore, the researcher used the five levels, the highest number, for developing the initial rubrics in this study. Then the researcher investigates the appropriate number of them after examining their application.

The description of the rubrics’ level is a key aspect of providing feedback (Brookhart, 2018; Moskal, 2000). It contains information on expected skills for each level ranging from difficulty to easiness. However, there is a conflict regarding how to develop these descriptions. One method is to use a set of adjective words to describe each level. Tierney and Simon (2004) suggest using a set of sequential quantifiers or adverbs of frequency to describe the difference in skill levels and provide a positive sense of feedback to promote learning motivation. Sequential words, such as few, some, most, and all, may be used for each skill level. Some studies on developing musical assessing tools, such as Wesolowski et al. (2017) and Edwards et al. (2019), used this approach. For example, the change in tone quality between registers is described as a serious problem, a moderate problem, and not a problem during performance.

Although consistency can promote agreement among raters (Airasian & Russell, 2008), DeLuca and Bolden (2014) argue that developing descriptions with a set of sequential quantifiers or adverbs of frequency can cause subjectivity and ambiguity. The use of such feedback can result in different interpretations of information between students and teachers, limiting the clear understanding and application of assessment results and hindering the useful guidelines for self-improvement.

Many-Facet Rasch Measurement Partial Credit Model (MFRM-PCM)

Although the Classical Test Theory is widely used because it is easier to be applied and analyzed in any test situation, it is sample dependent, which means whether it is applied to other sets of samples, its psychometric properties (including items’ difficulty and discrimination) will change (Fan, 1998). This problem will hinder the purpose of the development of the standard tools.

To ensure the invariant psychometric properties and determine the appropriate number of rubrics levels, the most suitable measurement model for this study would be the Many-Facet Rasch Measurement Partial Credit Model (MFRM-PCM). Based on the study (Bond et al., 2021; Linacre, 1989, 2022a), the MFRM-PCM is suitable for analyzing the psychometric properties of rubrics, as it can analyze the multi-scale rubrics, which involve two or more facets. This model explains a person’s ability, the difficulty of an item, and the severity of a rater by probability calculated from a raw score and allows all elements related to the assessment to be investigated together. In other words, the student’s ability is estimated by the probability of being assessed by raters using items. Moreover, including the rater facet in the model’s analysis ensure that the psychometric properties will not vary when different raters use it. The analysis includes the facets related to this study, that is, the student, rater, musical instrument, and item facet. Each facet functions to verify construct validity that supports the quality of the rubrics’ validity.

Additionally, this model allows each item to have its own levels of difficulty, so an item’s level can vary within the same rubrics and provides psychometric properties in terms of construct validity and reliability. It can also investigate the effectiveness of each quality level, which helps the researcher determine the appropriate number of levels based on statistical evidence. Furthermore, the model is flexible for incomplete assessing design applications. Lastly, when used with other target groups, the rubrics that undergo a construct revision with the MFRM-PCM will contain stable psychometric properties. The MFRM-PCM used in this research has the following equation (revised from Linacre, 1989).

l n (\frac{P_{n i k j m}}{1 - P_{n i k j m}}) = B_{n} - D_{i k} - C_{j} - A_{m}

$l n (P_{n i k j m} / 1 - P_{n i k j m})$ is the probability of student n that plays instrument m and is assessed in category k of item i by rater j, $B_{n}$ is the ability of Student n, $D_{i k}$ is the (difficulty) pertaining to the score levels in item i, $C_{j}$ is the severity/leniency of rater j, and $A_{m}$ is the influence of the results of the assessment on musical instrument m.

The targeted instruments for the development of the rubrics

The study aims to develop rubrics for assessing the solo playing of four Thai musical instruments; that is, Thai xylophone (Ranat Ek), Thai alto xylophone (Ranat Thum), Thai large Gong circle (Khong Wong Yai), and Thai small Gong circle (Khong Wong Lek). These instruments are typically played in a Thai percussion ensemble (Piphat) (Miller, 1992). They are all percussion instruments that can also be played solo to demonstrate the player’s skills and required similar assessment criteria (Ministry of University Affairs, 2001), so they rule out differential item functioning. The rubrics developed using the MFRM-PCM will help assess learners’ skills and enable instructors to improve their lessons. Developing the rubrics is necessary because Thai musical instrument skills are typically learned as an extracurricular activity (Lithsomphot & Sithiamorn, 2017), and learners can benefit from a reliable assessment tool to evaluate their progress and identify areas for improvement.

Method

The study consisted of three stages of research and development.

Stage 1

The synthesis and verification of content validity. Synthesizing documents and related research were used to develop rubrics in the preliminary stage. Then, five experts reviewed the rubrics and scored each item based on the relationship between the rubric’s content and the musical performance it intended to assess. The experts consisted of experts in Thai Music performance in higher education and secondary school and an expert in educational measurement and evaluation.

Stage 2

Pre-testing the psychometric properties and the assessment design. The developed rubrics in Stage 1 were used to assess 40 high-school students who played four types of instruments by five raters. The researcher used the linking design (Edwards et al., 2019; Engelhard, 1997; Linacre, 2022a), recommended for MFRM-PCM analysis, to ensure that data were linked. This design reduced the number of tasks for each rater, as compared to the complete design where every student and every item would be assessed by every rater. This design was also done to prevent fatigue, which could affect the consistency of the raters (Johnson et al., 2009), and still maintained the model’s psychometric properties estimation within each facet (Linacre, 1989; Uto, 2021). To clarify, the design was one rater would assess 16 students using every assessment item, and one student would be assessed by two different raters. And every rater would assess the same student that had been assessed by eight previous raters and would be assessed by the next eight. The number of students was higher than the minimum required for the exploratory analysis using the MFRM-PCM at 30 (Chen et al., 2014; Linacre, 2022a).

Stage 3

The verification of the rubrics’ psychometric properties. The developed rubrics underwent pre-testing and improved quality levels were used to assess 84 high-school students who played four types of instruments by six raters. The linking design was also used, which was one rater would assess 28 students and every rater would assess the same student that had been assessed by the previous 14 raters and would be assessed by the next 14. Therefore, each assessment item would be rated at least 168 times (one student per two raters), which was higher than the minimum of 100 times required for improving the quality levels’ analysis (Linacre, 1994). After the analysis, the rubrics with improved effectiveness of the quality levels were given to the raters, who were actual users, for content analysis and analytic induction. Therefore, the qualitative analysis would be used to improve the rubrics’ quality once more.

In Stages 2 and 3, video records of student performances were collected as research data from organizations that held Thai musical instrument competitions. Purposive sampling was used to select the samples, with criteria including a similar type of musical instrument, the age of the performance, the solo songs, and the quality of the audio and visual recordings. Six raters, meanwhile, were chosen to assess the performances based on at least 2-year of experience teaching Thai instruments, and in sending students to competitions or judging competitions. It was also important that the raters had not judged the sample group.

The researcher provided training to each rater to cover the targeted skills being assessed, the quality levels described in the rubrics, and the potential errors that raters may make. Randomly selected samples of performances were then given to each rater, with an equal distribution of the four targeted instruments.

Results

Initial rubrics

After synthesizing the basic criteria outlined, ten equally separated criteria were found for each instrument. However, criterion 10, which involved instrument maintenance after a performance, was not conducted immediately during the performance. Therefore, it was removed from the initial rubrics. Additionally, the researcher found that criteria 5 and 8, which assessed the accuracy of key melodies and arrangement (Thang), should be assessed together because they are interdependent when arranging an instrument’s solo version (Miller, 1992). Thus, these criteria were combined. Moreover, some criteria were subdivided due to their extensive details. Consequently, the initial rubrics consisted of 8 criteria and 12 items.

To convert the narrative description from the basic criteria outlined into the rubrics format, the researcher, firstly, placed the description in the highest level (level 5) because the basic criteria outlined only contained appropriate skills. Therefore, it was reasonable to place it at the highest level (Dickinson & Adams, 2017). To generate the lower levels (levels 1 to 4), the researcher sequenced the explicit descriptions of the extent of what must be demonstrated in the same order for each score level.

After developing the description at each level for each item, the initial rubrics would be sent to five experts to examine the content validity. To illustrate, criterion 4, item 7 was used to represent the rubrics’ development process in this paper. The example of the initial item in rubrics as shown in Figure 1.

Figure 1.

Example of the initial rubrics (translated from Thai to English).

The results of content validity and pre-testing analysis

The results of the content validity analysis showed that the agreement value of all 12 items ranged from 0.6 to 0.10, which was above the acceptable value of 0.5. However, there was advice from five experts that emphasized unclear descriptions, which were concerned that they probably confused raters in some cases.

As a result, the researcher improved the descriptions of the quality levels of each item using the experts’ suggestions and the structure of the quality levels using the concept of monotonicity for each item, with the high quality indicating the full elements and the lower quality indicating elements that were missing as the quality declines (as seen in the bold sentence of each level in Figure 2). The development aligned with the rating scale concept, which promoted agreement among raters and maintained all the information obtained from the assessment.

Figure 2.

Example of the rubrics after content validity and pre-testing analysis (translated from Thai to English).

The rubrics improved in Stage 1 were used in pre-testing with would be analyzed by MFRM-PCM. The reliability of separation indicated differences of the elements in the facets ranged from 0 to 1. The higher value confirms the reliability and supports construct validity. The value of the Fit Mean Square was intended to be between 0.5 and 1.5 to indicate the model fit. If the value was below 0.5, the data were deemed too predictable. If the value was above 1.5, then the data were deemed unpredictable.

The analysis results suggested that the rubrics’ item, rater, and musical instrument facet passed all psychometric properties criteria (Item Rel = 0.90, Item infit/outfit = 0.70 – 1.29, Rater Rel = 0.95, Rater infit/outfit = 0.73 – 1.31, Musical Instrument Rel = 0.00, Musical Instrument infit/outfit = 0.82 – 1.27), while student facet showed the model misfit of eight students (Student Rel = 0.76, Student infit/outfit = 0.43 – 2.30). However, the researcher determined to maintain the misfit student in the model analysis because this result was not from students directly. It was a consequence of raters who used the items to assess the student. Therefore, the researcher would subsequently investigate the misfit student from the rater and item facet (Linacre, 2010; Meijer & Sitsma, 2001; Rudner & Wright, 1995).

Following the analysis, the researcher collected and synthesized relevant questions about the rubrics of each rater during the rater training and then used them as guidelines for the rubrics’ improvement to ensure the understanding, harmony, and feasibility of the levels’ descriptions. The revision covered the following areas; that is, criteria title and the item title to ensure they were consistent with the skills assessed, operational definition of the criteria and the item to ensure clarity, with examples for the observation included, and misspellings and the description for the details of some of the items. The example of the item in rubrics after pre-testing as shown in Figure 2.

The results of the psychometric properties analysis in the final stage using MFRM-PCM

The psychometric properties analysis in the final stage showed that the fixed Chi-Square test indicated a significant difference in the student facet ( $χ^{2}$ = 279.2, p < .01), the rater facet ( $χ^{2}$ 307, p < .01), and the Item facet ( $χ^{2}$ = 81.8, p < .01). However, there was no significant difference in the musical instrument facet ( $χ^{2}$ = 1.6, p > .01), which was reasonable because there should be non-difference of assessment influence among the four types of musical instruments. The analysis results of the random Chi-Square for the student and rater facets found that the samples that are students ( $χ^{2}$ = 56.0, p > .01) and the raters ( $χ^{2}$ = 4.9, p > .01) showed the abilities and score assessment that was not different from the population.

The reliability of separation in the student facet (Rel = 0.68) indicated the difference in students’ skills. The rater facet (Rel = 0.98) and the musical instrument facet (Rel = 0.88) indicate a difference that was sufficient to confirm construct validity. The musical instrument facet (Rel = 0.00) also indicated the non-difference of assessment influence among instruments. The results of Fit Mean Square in the Rater facet (infit/outfit = 0.78 – 1.22) and the Musical Instrument facet (infit/outfit = 0.97 – 1.10) showed the model fit, except the Student facet (infit/outfit = 0.43 – 2.30) There were eight students that show a lack of the model fit, and the Item facet (infit/outfit = 0.58 – 1.56) contained Item 10 (infit/outfit = 1.12 – 1.56) that was not fit with the model, which needed an investigation (the summary statistical results as shown in Table 1).

Table 1.

Summary statistical results.

Item	Measure	Fit MS		$P T_{c o r}$	Categories usage					Average measure					Outfit MS
Item	Measure	Infit	Outfit	$P T_{c o r}$	1	2	3	4	5	1	2	3	4	5	1	2	3	4	5
1	−0.86	1.08	0.93	0.32	0^a	0^a	2^a	24	132	–	–	2.11	1.98^a	2.99	–	–	1.6	0.8	1.1
2	−0.58	0.96	0.90	0.43	0^a	1^a	10	17	130	–	0.78	1.32	1.62	2.77	–	0.6	0.8	1.0	0.9
3	−0.39	0.99	0.97	0.36	0^a	2^a	1^a	24	131	–	0.96	0.97	1.49	2.55	–	1.2	0.6	0.9	1.0
4	0.06	1.12	1.07	0.54	1^a	5^a	30	37	85	−0.33	0.93	0.91^a	1.8	2.39	0.6	1.4	0.6	1.4	1.2
5	0.10	1.17	1.43	0.41	2^a	2^a	15	46	93	0.39	−0.12^a	1.13	1.56	2.21	1.1	0.3	1.8	1.7	1.1
6	0.24	1.09	0.98	0.49	0^a	5^a	15	37	101	–	0.27	0.66	1.31	2.11	–	0.8	0.8	1.1	1.1
7	0.07	1.04	1.01	0.47	0^a	3^a	9^a	52	94	–	0.32	0.74	1.46	2.29	–	0.8	0.9	1.1	1.0
8	0.62	0.89	0.84	0.61	0^a	6^a	16	71	65	–	−0.09	0.05	1.11	1.06	–	1.0	0.4	0.9	1.0
9	0.20	0.90	1.03	0.49	0^a	0^a	10	38	110	–	–	0.33	1.14	2.11	–	–	0.8	1.2	0.9
10	−0.15	1.12	1.56^a	0.37	1^a	1^a	12	45	99	−0.27	0.71	0.98	1.98	2.36	0.3	0.9	0.8	2.5^a	1.2
11	0.56	0.93	0.90	0.54	0^a	0^a	12	60	86	–	–	–0.15	1.02	1.90	–	–	0.6	1.0	1.0
12	0.13	0.63	0.58	0.75	1^a	3^a	20	69	65	−0.55	0.36	0.60	1.38	2.82	0.3	0.7	0.6	0.4	0.7

Means that the item should be investigated.

The analysis results of the category’s effectiveness

From the statistical analysis, it was found that every item needed to be improved in terms of the quality level. The quality level effectiveness index must be considered, with the Point-Measure Correlation value of each item being positive, the category usage for each quality level of each item being used at least 10 times, the average measure of each quality level being monotonic from the lower level to the high level, and the Outfit MS value for each quality level being under 2.0 (Linacre, 2002). It was found that the quality level 1, 2, 3, and 4 of Item 1; the quality level 1 and 2 of Item 6, 8, 9, 11, and 12; the quality level 1, 2, and 3 of Item 3, 4, and 7; as well as the quality level 1, 2, and 4 of Item 10 should be combined or revised (the summary statistical results as shown in Table 1).

To maintain the rubrics’ important information, the researcher turned quality level 1 of Items 3, 4, and 9, which should be combined with other quality levels, into a tentative agreement for the player to follow before the rubrics was used. The tentative agreement prior to the rubrics’ usage can be summarized as follows; that is, the correct type of mallets for playing instruments, sitting in the meditation position or on the floor with legs on the side (Thai sitting style), and the correctness of solo Thai musical pieces.

The results of the critique by the raters

After the quantitative analysis, the researcher asked three raters, who were the actual users, for a critique to provide their views on the explanation of the quality levels of the rubrics for the final round of assessment. When comparing the rubrics used for the assessment of the skills in the final stage, with five quality levels for each item, the critique results revealed that all three raters’ comments were agreeable when it came to 1) the final rubrics’ descriptions were significantly improved, in terms of showing more clarity and clearly differentiate various players’ skills, 2) the reduction of the quality level numbers helped the raters make an assessment easier while their feedback was maintained, and it was consistent with the skill types when it came to two quality levels, readiness and unreadiness.

I also agreed that some quality levels for some items should be part of the tentative agreement before the preliminary usage because it’s necessary to do so. For example, the mallets must be used with the correct type of instrument. If they are not, we would not give a score. The three items added are considered appropriate. If the actual performance falls in line with one of the three items, scoring won’t be done, or we can make void. But that rarely happened. So, it should be consistent with the research in that it would be very unlikely that the performance would fall into quality level 1 in the previous rubrics. I quite agree the items should contain two quality levels, like Item 1 (Instruments) and Item 2 (Mallets), which concern readiness, because these items are about readiness, so it’s about whether you’re ready or not. It’s not about being very ready or barely ready. So, it’s correct that there are only 2 quality levels, i.e., readiness and unreadiness. It’s consistent with the research results. (Rater 1)

The rubrics whose quality level numbers were adjusted make it easier for us to assess, even if it already provides enough clarity. When put in the classroom context, I think the new rubrics are more appropriate because students will receive better and correct feedback. (Rater 2)

The rubrics can be used to assess skills and I quite agree that it can be used as a standard. (Rater 3)

Item 7 (Basic technical performance) and Item 10 (Accuracy of solo performance and fit with the melody and rhythm) should have two quality levels based on the analysis results. However, when considering the skill types, it was found that these skills are not about readiness, so they should have at least three quality levels. For Item 7, the researcher considered combining quality levels 1 and 2 with quality level 3. This was because when quality level 2 (3 times) was combined with quality level 3 (9 times), the sum was over 10 times, so it was considered appropriate that the quality levels were combined and reduced to 3, which was equal to the other items in the same criterion.

Item 10 did not fit the model due to its extremely overuse category (Outfit MS > 1.5) (Linacre, 2022b). The analysis results showed that Level 4 contained assessment results that could not be explained, came off as accidental, or were random. The researcher decided to give Item 10 only three quality levels by combining problematic quality level 4 with level 3 from the old rubrics and combining quality level 1 with level 2 from the previous rubrics to quality level 1. The rubrics’ quality levels for each performance stage are summarized in Table 2. The example of the final rubrics is shown in Figure 3.

Table 2.

The summary of the number of quality levels of the rubrics in each stage.

Criteria	Items	Number of quality levels
Criteria	Items	Pre-analysis	Post-analysis	Post-critique
Aspect 1 Preparation of instruments	1. Instruments	5	2	2
	2. Mallets	5	2	2
	3. Instruments’ tuning	5	3	3
Aspect 2 Sitting posture and body gesture	4. Sitting posture and body gesture	5	3	3
Aspect 3 Mallet holding	5. Mallet holding	5	3	3
Aspect 4 Fundamental and technical performance	6. Fundamental performance	5	3	3
	7. Basic technical performance	5	2	3
	8. Special technical performance	5	3	3
Aspect 5: Accuracy of key melodies and appropriate arrangement for each player’s skills	9. Accuracy of key melodies and appropriate arrangement for each player’s skills	5	3	3
Aspect 6 Accuracy of solo performance and fit with the melody and rhythm	10. Accuracy of solo performance and fit with the melody and rhythm	5	2	3
Aspect 7 Tone quality throughout the performance.	11. Tone quality throughout the performance.	5	3	3
Aspect 8 Connection of each skills performance	12. Connection of each skills performance	5	3	3

Figure 3.

Example of the final rubrics (translated from Thai to English).

Discussion

The results of rubrics development, which contain two to three levels analyzed by quality levels’ effectiveness, likely provide quality feedback to help students improve their skills on Thai percussion instruments because the final rubrics underwent a psychometric property examination involving students with a broad range of abilities. The fixed and random Chi-Square and the reliability of separation also revealed that students had varying abilities that are close to those resulting from the population. Therefore, the improvement of the number of quality levels of each item would help maintain consistency with a large group of students’ skill levels. If the rubrics are used to assess students or students use them to assess themselves, the assessment results from the rubrics will be comparable to other students’ skills with more accuracy. Moreover, according to Jonsson and Panadero (2017), it is beneficial for rubrics to have each item with a different number of quality levels in terms of eliminating the rater effect.

Items with two quality levels were in accordance with the types of practice skills (Johnson et al., 2009). When it comes to preparation, which is the initial part of the complicated rehearsal, there should be only practice and non-practice. In this case, the rubrics contain two levels (i.e., readiness and non-readiness) only for the parts related to the preparation of musical instruments. In line with the raters’ critique, there should only be two levels regarding readiness: being ready and unready. That is consistent with Linacre (2002), which also found that some variables have only two skill levels or two responses in nature. In addition, based on other studies related to the assessment of musical instruments that also employed the MFRM-PCM, it was also found that some items that underwent the quality levels’ effectiveness revision were reduced the number of quality levels to two (Edwards et al., 2019; Wesolowski et al., 2017).

As for accuracy in identifying the skills’ quality levels, students would receive accurate and precise results from rubrics’ information that will stimulate students learning and practice increasingly (DeLuca & Bolden, 2014; Edwards et al., 2019; Panadero & Jonsson, 2013). They also help students learn about their abilities and become aware of which skills they need to improve and how (Asmus, 1999; Pancorbo et al., 2020). Therefore, the rubrics, which underwent a revision to lower the quality levels and were analyzed with samples that possess varied skills, could accurately reflect students’ skills, which will affect their motivation and reflection. This claim is also in line with the raters’ critique during the rubrics’ improvement in the final stage highlighted the two types of Thai musical instruments learners, that is, those who prefer challenges and those who don’t. Rubrics may cause different effects on the student’s motivation.

To explain, if both types of students are assessed to be middle (quality level 3) in the rubrics containing five quality levels, students who like challenges would feel the need to improve but will find time to practice more later while students that don’t like challenges would feel that their moderate skills are enough. In contrast, when compared with the rubrics that contain two to three quality levels, the students would be assessed at the lower end of the quality level (quality level 1), compared to the same skills in the rubrics containing five quality levels. Students who like challenges would feel motivated to practice more because they feel the need to go rise above the lowest quality level. Students who don’t like challenges may feel discouraged. The rubrics that underwent a revision to lower the number of quality levels, thus, will provide a significant impetus for both types of students more than the rubric with five quality levels.

As for feedback obtained from the rubrics’ assessment results, since the rubrics underwent a revision to lower the number of quality levels and did not undergo revision to the descriptions of the skills in each level, only the completeness of each element was improved. The feedback, thus, obtained from the descriptions in each level of the items is not different in the rubrics that contain five quality levels and the ones that contain two to three quality levels. The difference is the reflection on students’ quality levels, which will affect students’ motivation to get additional practice. The rubrics users that are instructors can also use the information obtained from the feedback similarly. Therefore, even though the developed rubrics contain a reduced number of quality levels, the efficiency of providing feedback becomes more accurate, which is the objective of this research.

Moreover, the description development concept used in this study may compromise the conflicting views about using a set of sequential quantifiers or adverbs of frequency stated in this article. These rubrics contain descriptions to provide useful information in which high quality indicates the full elements and lower quality indicates elements that were missing as the quality declines to ensure objectivity and unambiguity, as well as using complete or incomplete elements to maintain consistency. However, it is important to make sure that there is communication between users. To get the most out of these rubrics, the instructors must identify which element students are missing in each item.

In addition, the improvement of the rubrics’ quality for this research regarding the assessment of Thai musical instrument skills is verified in terms of quality and quantity. The quality involves the criteria and item synthesis, the examination of content validity by experts, the revision of definitions of items, and the descriptions of the rubrics’ quality levels from the raters’ queries during the raters’ training, as well as the critique by the raters, who were the actual users. The quantity includes an examination of the psychometric properties by MFRM-PCM, both during the pre-testing and the final stage, which confirmed the quality of the rubrics through elements that concern four facets, as well as the analysis to decide on improving the quality levels’ effectiveness for the items. The researcher considers that the initial rubrics, five quality levels developed by synthesis and the revision based on the experts’ or the raters’ suggestions, are to develop in terms of quality only. The fact that the researcher made an additional improvement based on the analysis results using MFRM-PCM and the quality levels’ effectiveness analysis, is to develop in terms of quantity based on statistical evidence. It can, thus, be concluded that the rubrics for assessing Thai musical instrument skills are the result of the development of both quantity and quality, in line with the approach by van der Linden (2017), who suggested that there should be a common ground between the content the experts agree on and what fits the measurement model.

For future studies, there should be studies to consider adding skills related to performances involving other Thai musical instruments, such as Thai musical aesthetics, that were not identified clearly in any related documents or research and include them in other developed rubrics to increase the challenges in the assessment. New skills might be added to these rubrics or created as a new tool. However, if new skills are included in these rubrics, one should consider whether the new and existing skills are unidimensional. If they are multidimensional, other models that can analyze multidimensionality should be considered.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Purin Thepsathit

References

Airasian

P. W.

Russell

M. K.

(2008). Classroom assessment: Concepts and application (6th ed.). McGraw-Hill.

Asmus

E. P.

(1999). Music assessment concepts. Music Educators Journal, 86(2), 19–24. https://doi.org/10.2307/3399585

Bond

T. G.

Yan

Heene

(2021). Applying the Rasch model: Fundamental measurement in human sciences (4th ed.). Routledge.

Boonlue

(2004). A construction of an evaluation form for basic skill performance of “Khong-Wong-Yai” [Master’s thesis]. Srinakharinwirot University Institutional Repository (SWU IR). http://thesis.swu.ac.th/swuthesis/Ed_Mea/Ronnachai_B.pdf

Brookhart

S. M.

(2013). How to create and use rubrics for formative assessment and grading. ASCD.

Brookhart

S. M.

(2018). Appropriate criteria: Key to effective rubrics. Assessment, Testing and Applied Measurement, 3(22), 1–12. https://doi.org/10.3389/feduc.2018.00022

Brookhart

S. M.

Chen

(2014). The quality and effectiveness of descriptive rubrics. Educational Review, 67(3), 1–26. https://doi.org/10.1080/00131911.2014.929565

Carr

N. T.

(2000). A comparison of the effects of analytic and holistic rating scale types in the context of composition tests. Issues in Applied Linguistics, 11(2), 207–241.

Chen

W. H.

Lenderking

Jin

Wyrwich

K. W.

Gelhorn

Revicki

D. A.

(2014). Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Quality of Life Research, 23(2), 485–493. https://doi.org/10.1007/s11136-013-0487-5

10.

Chuppunnarat

Laovanich

Srisuttiyakorn

(2018). The development of assessment tools and quality judgment system for Mahori ensemble competition. Journal of Education Studies, 46(4), 300–317.

11.

Deluca

Bolden

(2014). Music performance assessment: Exploring three approaches for quality rubric construction. Music Educators Journal, 101(1), 70–76.

12.

Dickinson

Adams

(2017). Value in evaluation – the use of rubrics. Evaluation and Program Planning, 65(1), 113–116. https://doi.org/10.1016/j.evalprogplan.2017.07.005

13.

Edwards

A. S.

Edwards

K. E.

Wesolowski

B. C.

(2019). The psychometric evaluation of a wind band performance rubric using the Multifaceted Rasch Partial Credit Measurement Model. Research Studies in Music Education, 41(3), 343–367.

14.

Engelhard

(1997). Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1(1), 19–33.

15.

Fan

(1998). Item response theory and classical test theory: An empirical comparison of their item/person statistic. Educational and Psychological Measurement, 58(3), 357–381. https://doi.org/10.1177/0013164498058003001

16.

Johnson

R. L.

Penny

J. A.

Gordon

(2009). Assessing performance: Designing, scoring and validating performance tasks. The Guilford Press.

17.

Jonsson

Panadero

(2017). The use and design of rubrics to support assessment for learning. In Carless

Bridges

S. M.

Chan

C. K. Y.

Glofcheski

(Eds.), Scaling up assessment for learning in higher education (pp. 99–111). Springer.

18.

Jonsson

Svingby

(2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144.

19.

Lertsiri

(2014). Knowledge management of Thai traditional music skills: case study of Ra Nad-ek [Doctoral dissertation]. Chulalongkorn University Intellectual Repository (CUIR). http://cuir.car.chula.ac.th/handle/123456789/46368

20.

Lertwiriyapiti

(2020). Construction of Gong Wongyai skills: A case study local wisdom of Thai Auttagrit music school [Master’s thesis]. Srinakharinwirot University Institutional Repository (SWU IR). http://ir-ithesis.swu.ac.th/dspace/bitstream/123456789/1057/1/gs611130170.pdf

21.

Linacre

J. M.

(1989). Many-faceted Rasch measurement. MESA Press.

22.

Linacre

J. M.

(1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328. https://www.rasch.org/rmt/rmt74m.htm

23.

Linacre

J. M.

(2002). Understanding Rasch measurement: Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85–106.

24.

Linacre

J. M.

(2010). When to stop removing items and persons in Rasch misfit analysis? Rasch Measurement Transactions, 23(4), 1241. https://www.rasch.org/rmt/rmt234g.htm

25.

Linacre

J. M.

(2022a). A user guide to FACETS (Version 3.84.0) [Computer software]. https://www.winsteps.com/a/Facets-Manual.pdf

26.

Linacre

J. M.

(2022b). A user guide to WINSTEPS (Version 5.2.3) [Computer software]. https://www.winsteps.com/a/Winsteps-Manual.pdf

27.

Lithsomphot

Sithiamorn

(2017). A model of Thai music learning and instruction management of secondary schools under the office of the basic education commission Bangkok metropolitan, Academic Journal Phranakorn Rajabhat University, 8(2), 196–203.

28.

Meijer

R. R.

Sitsma

(2001). Person fit statistic - What is their purpose. Rasch Measurement Transaction, 15(2), 823.

29.

Mertler

C. A.

(2001). Designing scoring rubrics for your classroom. Practical Assessment, Research, and Evaluation, 7(7), 1–8. https://doi.org/10.7275/gcy8-0w24

30.

Miller

T. E.

(1992). The theory and practice of thai musical notations. Ethnomusicology, 36(2), 197–221. https://doi.org/10.2307/851914

31.

Ministry of University Affairs. (2001). Standard criteria for Thai music and criteria of assessment. Parbpim Limited Partnership.

32.

Moskal

B. M.

(2000). Scoring rubrics: What, when and how? Practical Assessment, Research, and Evaluation, 7(7), 1–5. https://doi.org/10.7275/a5vq-7q66

33.

Panadero

Jonsson

(2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 1(9), 129–144.

34.

Pancorbo

Primi

John

O. J.

Santos

Abrahams

Fruyt

F. D.

(2020). Development and psychometric properties of rubrics for assessing social-emotional skills in youth. Studies in Educational Evaluation, 67(1), 100938.

35.

Rudner

Wright

B. D.

(1995). Diagnosing person misfit. Rasch Measurement Transaction, 9(2), 430. https://www.rasch.org/rmt/rmt92h.htm

36.

Tierney

Simon

(2004). What’s still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2), 1–7. https://doi.org/10.7275/jtvt-wg68

37.

Uto

(2021). Accuracy of performance-test linking based on a many-facet Rasch model. Behavior Research Methods, 53(4), 1440–1454.

38.

Van der Linden

W. J

. (2017, 31 January). IRT-van-der-Linden-2. https://www.rasch.org/audio/IRT-van-der-Linden-2.mp3

39.

Wesolowski

B. C.

(2012). Understanding and developing rubrics for music performance assessment. Music Educators Journal, 98(3), 36–42. https://doi.org/10.1177/0027432111432524

40.

Wesolowski

B. C.

Amend

R. M.

Barnstead

T. S.

Edwards

A. S.

Everhart

Goins

Grogan

R. J.

III Herceg

A. M.

Jenkins

S. I.

Johns

P. M.

McCarver

C. J.

Schaps

R E.

Sorrell

G. W.

Williams

J. D.

(2017). The development of a secondary-level solo wind instrument performance rubric using the Multifaceted Rasch Partial Credit Measurement Model. Journal of Research in Music Education, 65, 95–119.