On the quality of professional development in the United States: Examining music educators’ experiences,sentiments,and emotions

Abstract

The purpose of this study was to develop a valid and reliable assessment tool to measure music educators’ participation in and perceptions of their high-quality professional development (PD) experiences. A sample of 450 in-service music educators across the United States completed a 78-item rating scale embedded within 10 domains reflecting high-quality PD criteria: (a) content, (b) evidence/research, (c) coherence, (d) relevance, (e) active learning, (f) application, (g) collaboration, (h) reflection, (i) feedback, and (j) duration. Participants also answered the open-ended question, “What are some adjectives you would use to describe your overall professional development experiences?” Data were analyzed using Rasch measurement analyses and sentiment analyses. Results indicated an overall acceptable data-to-model fit, with the content domain reflecting most participation and the duration domain reflecting least participation. Bias analyses suggested that music educators received systematically different quality PD experiences based upon their teaching area and/or grade-level emphases. Participants had an overall positive sentiment of their PD experiences, which positively correlates with the level of quality PD they experienced (r = 0.42, p < .05). The most frequent emotions of the participants were trust and joy. Implications for the development and programming of PD in the field of music education are discussed.

Keywords

In-service teaching professional development Rasch sentiment analysis survey

Professional development (PD) of educators is essential to the improvement of teaching and learning within and across schools (Guskey, 2014; Kyriakides et al., 2009; Reeves, 2010; Wells, 2014). Research literature in music education supports the value of PD and its positive impact on music educators and their students (Koner & Eros, 2019). This is the case both in the United States and internationally. For example, Pellegrino et al. (2018) identified PD communities as having a positive impact on music teacher educators’ sense of empowerment and self-understanding in the United States. In Korea, Shin and Seog (2018) suggested that collaborative PD positively affects music educators’ content knowledge and confidence in teaching. Andrews (2016) found that community PD experiences play a significant role in arts educators’ personal development in Canada.

Researchers investigating PD in music education primarily focus on educators’ needs (Bauer, 2007; Bautista et al., 2018; Bowles, 2002; Conway, 2007, 2011) preferences (Ferrara, 2009), and motivations (Bush, 2007). Although there is clear support that music educators’ engagement in PD positively affects a variety of variables in music teaching and learning, there is little evidence as to (a) what defines PD as being a “quality” experience, (b) what extent music educators experience “quality” PD, and (c) how the “quality” of PD affects music educators’ perceptions of its effect. In the United States, not all music educators experience the same amount or quality of PD. The first problem, as noted by McCaffrey and Lovins (2019) based upon a State Education Agency Directors of Arts Education (2016) study on arts assessment, is that “. . . there is no vetting system to ensure the quality of professional development offerings” (p. 41). The second problem, they continue, is that “educators are often left to their own devices to find opportunities. . . and they often bear the cost for such professional growth opportunities themselves” (p. 41). Bauer (2007) identified several limitations for programming and music educator access to quality PD, including geographic areas, time allowances, technological considerations, and the role of individual experiences and practices. Conway (2008) identified similar limitations, including challenges related to communication, observation, time management, technological interaction, and mentor training. Although there are many limitations to overcome for defining, programming, and accessing quality PD, its continuing need for improvement is continually recognized by the field of music education (Bowles, 2002).

These limitations are not only music-specific but also true for the field of education, broadly. As Buysse et al. (2009) noted, there is no agreed-upon definition of what constitutes “quality” PD in education or related fields. Guskey (2000), also recognizing this disparity, suggested three reasons for this lack of consensus: (a) research literature contains narrow and disjointed representations of criteria that describe quality PD; (b) there is a premature focus on the effects of PD rather than on the components or processes of PD; and (c) there is a focus on the quantity of PD experiences and neglect toward quality.

Characteristics of high-quality professional development

Outside of a review of literature by Bautista et al. (2017), who described a five-feature framework (content focus, active learning opportunities, coherence, collective participation, duration) of quality PD based upon the work of Desimone (2009), there is little research as to what characterizes “quality professional development (PD)” specific to the field of music education. Therefore, similar to Bautista et al. and expounding upon their work, we revisited educational research and policy literature to identify and extrapolate more detailed characteristics of high-quality PD suitable for developing a comprehensive rating scale. To systematically and validly identify themes in the research literature that are characteristic of high-quality professional development experiences, we used the thematic coding process prescribed by Gibbs (2007). Based upon this methodology, we identified 10 key themes. We used these ten themes as domains in the scale: (a) content, (b) evidence/research, (c) coherence, (d) relevance, (e) active learning, (f) application, (g) collaboration, (h) reflection, (i) feedback, and (j) duration.

Content

Content-driven PD supports the improvement of content knowledge related to instruction and the student (Desimone, Porter, Garet, et al., 2002; Garet et al., 2001). PD that is anchored in content appropriately aligned to the age and ability of students can support educators in clearly defining educational goals and objectives (Hiebert, 1999). PD that focuses on the appropriate alignment between the rigor of content and levels of student achievement may better support student learning and achievement (Kennedy, 1998).

Evidence/research

Evidence/research-based PD reflects the best available research and practice in teaching, learning, and leadership (Peixotto & Fager, 1998). Teaching-specific PD is often not adequately supported by up-to-date educational research, which may yield inherent uncertainties in trustworthiness, verifiability, and replicability (Guskey & Yoon, 2009). It is important that PD developers have access to relevant research and knowledge of best practice, that it is applied to the structure and design of PD, and that PD participants are informed of its origins (Guskey, 2000).

Coherence

Coherent PD is consistent with school, district, regional, and/or national reforms and policies (Desimone, 2011). Loucks-Horsley, (2010) recommended that school administrators and other stakeholders support educators’ PD programs to create consistency for teachers across schools, districts, and regions. Input from stakeholders and administrators can help ensure PD programs align with regulations, initiatives, reforms, and policies at all levels.

Relevance

Relevant PD is designed in direct response to teachers’ needs (Bautista et al., 2018). Bush (2007) suggested that high-quality PD be aligned to teachers’ educational and instructional objectives. Teachers are more likely to affect positive change in their teaching practices if they regard PD as relevant to their day-to-day teaching contexts and classroom practices (P. Little & Hayes, 2003).

Active learning

PD that includes active learning processes can facilitate engagement, interaction, and decision-making (Wei et al., 2009). Desimone (2011) suggested that teachers should have a participatory role, as opposed to a passive role, in their PD activities. PD that necessitates active participation such as discussion, observation, and networking, has been found to be more effective than programs that does not support active learning processes (Blank, 2013). Active learning opportunities may increase the effectiveness of teachers’ instruction (Desimone, Porter, Garet, et al., 2002).

Application

Applied PD is embedded in authentic contexts, such as real-world teaching and learning environments (Leong & Tan, 2019). Applied PD activities facilitate opportunities for teachers to apply new knowledge to ongoing teaching activities and enact long-term change in teaching strategies, fostering a more reflective stance toward evaluating student learning (Darling-Hammond & McLaughlin, 2011). Examples of applied PD activities include participating in mentoring, coaching, or observation (Garet et al., 2001), collecting data (Desimone, Porter, Birman, et al., 2002), action research (Bauer, 2007; Desimone, 2009), or other activities that take place during everyday teaching.

Collaboration

Collaborative PD involves interaction and discourse within a community of stakeholders (Stanley et al., 2014). Opportunities for collaboration should take place early in PD design and involve decisions among teachers, administrators, and PD facilitators (Helterbran & Fennimore, 2004; Peixotto & Fager, 1998). Collaboration focuses on teaching communities by invoking the sharing of knowledge (Darling-Hammond & McLaughlin, 2011 ) and promoting the spirit of positive change beyond individual teaching contexts and classrooms (Darling-Hammond & Richardson, 2009). Collaboration may also lead to improved depth-of-knowledge of subject matter, improved instructional strategies, and greater collegiality (Feiman-Nemser, 2001).

Reflection

Reflective PD promotes self-evaluation of teaching and learning (Forde et al., 2006; Ragland, 2017). It involves inquiry, experimentation, and critical reflection on teacher practice and the effectiveness of PD activities (Darling-Hammond & McLaughlin, 2011; Loucks-Horsley, 2010). Desimone (2009) suggested that because teacher reflection is a warranted component of teacher effectiveness frameworks, it should also be used to evaluate the quality of prescribed PD.

Feedback

PD should include effective feedback from a variety of stakeholders that is timely, appropriate, satisfactory, and useful (Bautista et al., 2014; Peixotto & Fager, 1998). It can stem from a variety of sources, including PD facilitators (Bautista et al., 2014), school administrators (Gallo, 2018), colleagues (Garet et al., 2001), and students (Mandouit, 2018). Receiving feedback is important and necessary for facilitating ongoing high-quality PD, particularly when it is longitudinal in nature (Desimone, 2011).

Duration

PD with extended duration is characterized as longitudinal and sequential in nature. PD should be of adequate duration to best support quality learning (Bautista et al., 2014; Boyle et al., 2005; Hiebert, 1999; Lydon & King, 2009; Wells, 2014). Varying lengths and durations for effective PD have been prescribed, including an average time of 6 months or 91 total hours (Blank, 2013), spread out over a semester including 20 hours or more of contact time (Desimone, 2011), or between 30 and 100 hours over 6 to 12 months (Darling-Hammond & Richardson, 2009). Although there is no consensus regarding the efficiency of contact time, researchers support the notion that longitudinal and sequential PD may be more effective than single-day experiences (Adey, 2004; Garet et al., 2001; Wei et al., 2009).

Purpose and research questions

As Guskey (2000) and Buysse et al. (2009) suggested in education, broadly, and Bautista et al. (2017) noted in music education, specifically, there is a need to focus more on the components and characteristics of quality PD as one way to improve educators’ experiences. The field of music education is in need of a mechanism to outline characteristics of quality PD in a way that also identifies and measures the extent to which music educators engage in quality PD experiences. Furthermore, little is known as to the relationship between the quality of music educators’ PD experiences and their perceptions of their experiences. The purpose of this study was to develop a valid and reliable assessment tool to measure music educators’ participation in and perceptions of high-quality PD experiences in the United States. The research questions that guided this study include:

What is the ordering and fit of items/domains representing participation levels of music educators’ high-quality PD experiences?

To what extent does bias exist in music educators’ PD experiences based upon their teaching area (e.g., primary, secondary, both) and/or grade level (e.g., pre-k/elementary, middle school, high school).

Is there a relationship between the quality of PD music educators experiences and their sentiments and emotions toward those experiences?

Method

Participants

In-service music educators across the United States were randomly solicited via the National Association for Music Education’s (NAfME) Research Survey Assistance portal. The selection criteria included all K-12, full-time, in-service music educators that are active members of NAfME. Prospective participants were contacted via two emails (one initial invitation email and one follow-up email). A total of 450 music educators completed the survey, which is an adequate sample size based upon recommendations prescribed by Linacre (1994b), Smith et al. (2008), and Wright and Tennant (1996). Demographic information collected from the participating in-service music educators included (a) state, (b) grade-level, (c) music content area taught, (d) full-time years taught, (e) level of education completed, and (f) music degrees completed (see Figure 1).

Figure 1.

Demographic data by grade level and teaching area. Panel A: National sample of respondents (N = 450). Panel B: Frequency (%) of PD experiences by grade level. Panel C: Frequency (%) of PD experiences by teaching area. Panel D: Frequency (%) of content area taught by grade level. Panel E: Frequency (%) of content area taught by teaching area. Panel F: Frequency (%) of degrees obtained by grade level. Panel G: Frequency (%) of degrees obtained by teaching area.

Survey items, domains, and response categories

The survey consisted of 78 items embedded within 10 domains described in the literature review: (a) content (n = 21), (b) evidence/research (n = 3), (c) coherence (n = 6), (d) relevance (n = 4), (e) active learning (n = 11), (f) application (n = 3), (g) collaboration (n = 9), (h) reflection (n = 4), (i) feedback (n = 8), and (j) duration (n = 9) (see Appendix 1). The response categories were based upon a dichotomous response format (e.g., no/yes) or a polytomous, 4-point Likert-type scale format with two possible anchor-types: (a) frequency (e.g., never/rarely, occasionally, frequently, usually/always), or (b) agreeability (e.g., strongly disagree, disagree, agree, strongly agree).

Psychometric considerations and data analysis

Rasch measurement

To answer the first and second research questions, Rasch Measurement analyses were used. Rasch measurement models (Rasch, 1960/1980) are part of the family of Item Response Theory models and are particularly effective for measuring latent constructs in the behavioral, social, and psychological sciences (Wesolowski, 2019). All surveys contain psychosocial information related to a latent construct under investigation (Wright, 2000; Wright & Masters, 1982). In the case of this study, the latent construct under investigation was defined as “high-quality professional development.” In survey research, the goal of Rasch measurement is to simultaneously and independently estimate respondent measures and survey item measures, thereby allowing an opportunity to make meaningful inferences about the usefulness and suitability of the items’ patterns of responses (e.g., construct validity; Bond, 2003; Linacre, 2004) and respondents’ patterns of responses (e.g., predictive validity; Linacre, 2004). The most meaningful application of Rasch analyses in survey research is to detect respondent and item idiosyncrasies through the analysis of fit statistics and related standard errors of measurement. Fit indices falling outside the range of 0.60–1.40 identify items and respondents that demonstrate idiosyncrasies in patterns of responses expected by the model (Linacre & Wright, 1994). When fit indices are outside of the acceptable range, they are considered not acceptable for making meaningful inferences. When fit indices are inside of the acceptable range, construct validity is supported for item measures and predictive validity is supported for respondent measures. The respondent measures can be interpreted as the respondents’ amount of participation in high-quality PD (herein referred to as “respondents,” or “respondent measures”). The item measures can be interpreted as the amount of participation in the criterion described in the item (herein referred to as “item,” or “item measures”). The FACETS computer program (Linacre, 2014) was used for all Rasch analyses.

Sentiment analysis

To answer the third research question, a sentiment analysis was conducted. Sentiment analysis is a computational process for extracting and classifying affect (e.g., sentiments and emotions) from natural language text (Liu, 2015; Mäntylä et al., 2018). The survey response options in this study were all forced-choice; therefore, we wanted to provide the respondents an opportunity to freely describe perceptions of their PD experiences. Respondents were posed the open-ended question, “What are some adjectives you would use to describe your overall professional development experiences?” We hypothesized that the more quality PD experiences respondents participate in, the higher the aggregated sentiment index score they would demonstrate. This relationship between the amount of quality participant experiences and their level of positive valence also served as a type of concurrent validity for the results of the Rasch analysis.

To examine affect, a corpus of descriptors was built from the raw responses. The responses were cleaned and normalized using the R (R Core Team, 2019) text-mining package, tm (Feinerer & Hornik, 2018). To identify sentiments and emotions, the R packages sentiment (Rinker, 2019) and syuzhet (Jockers, 2017) were used. First, the response data were restructured into a single character vector and aggregated sentiment scores were obtained for each respondent. Aggregated sentiment scores were assigned to each respondent based upon the positive, neutral, or negative valence of descriptors they provided. To further explore the lexical diversity of the polarized text, the distribution of emotions for the descriptors was also examined. After excluding all neutral-classified words, sentiments were classified into two categories (positive or negative) and emotions were classified into eight categories (anger, anticipation, disgust, fear, joy, sadness, surprise, or trust) (Ekman, 1992; Pluchik, 1980) using two lexicons developed by the National Research Council (NRC): (a) the Word-Emotion Association Lexicon, and (b) the Hashtag Sentiment Lexicon (Mohammad, 2012; Mohammad & Turney, 2010). The NRC lexicons have consistently demonstrated high reliability (alpha = .91–.98) (Kiritchenko and Mohammad, 2014, 2016).

Results

Table 1 provides the summary statistics of the Many-Facet Rasch Partial Credit (MFR-PC) model for the respondent facet, item facet, teaching area facet, and grade-level facet. Tests of chi-square significance answer the substantive question, “Are the overall differences between logit-locations for the elements (i.e., each respondent, each item, each teaching area, or each grade-level) statistically significant?” Results indicated overall statistically significant differences for the respondents (= 5634.40, p < .01), items (= 5070.80, p < .01), teaching areas (= 14.30, p < .01), and grade-levels (= 16.40, p < .01). Reliability of separation (Rel) statistics answer the substantive question, “How spread out are the element locations on the logit scale?” For respondents, Rel is interpreted similarly to Cronbach’s alpha. For the three remaining facets, Rel is interpreted as the degree to which the elements can be reliability differentiated from each other. Results indicated moderate to high reliabilities of separation between respondents (Rel = 0.95), items (Rel = 0.99), teaching area (Rel = 0.89), and grade level (Rel = 0.88). Mean square error (MSE) answers the substantive question, “How consistently have the patterns of responses been interpreted?” Acceptable fit is demonstrated by MSE ranges between 0.60 and 1.40 (Wright & Linacre, 1994). Values below 0.60 indicate underfit (i.e., too similar) patterns and values above 1.40 indicate overfit (i.e., too sporadic) patterns to make meaningful inferences. Results indicate that all facets demonstrated an acceptable range of data-to-model fit.

Table 1.

Summary Statistics for the Rasch Measurement Model.

	Facets
	Respondents	Items	Teaching area	Grade level
Measure (logits)
Mean	0.02	0.00	0.00	0.00
SD	0.85	1.02	0.03	0.11
N	450	78	3	3
Infit MSE
Mean	0.99	1.00	1.00	0.95
SD	−0.36	0.14	0.04	0.07
Std. infit MSE
Mean	−0.30	0.10	−0.70	−1.30
SD	2.00	1.90	1.50	2.30
Outfit MSE
Mean	1.05	1.05	1.05	1.03
SD	0.46	0.26	0.05	0.07
Std. outfit MSE
Mean	0.00	0.30	2.30	1.60
SD	1.80	2.30	1.80	2.70
Separation statistics
Reliability of separation	0.95	0.99	0.89	0.88
Chi-square	* 5634.40	* 5070.80	* 14.30	* 16.40
Degrees of freedom	449	77	2	2

SD: standard deviation; MSE: mean square error.

p < .01.

Item and domain calibrations (Research question 1)

The measurement calibrations and ordering for the items and domains are found in Table 2. Higher logit scores (i.e., measure) represent higher levels of participation in quality PD and lower logit scores represent lower levels of participation in quality PD. For the items facet, the least participation was demonstrated by item 10.3_04 (2.10 logits; I have participated in PD experiences that consisted of multiple, related sessions spread over 1 month) and the most participation was demonstrated by item 1.03_02 (–3.16 logits; I have experienced PD that addressed instructional strategies in teaching methods). One item demonstrated overfit (3.01_06, infit MSE = 1.57; I have experienced PD that has responded to district/state standardized testing results). For the domain groupings, the least participation was demonstrated in the Duration domain (1.02 logits) and the most participation was demonstrated in the Content-driven domain (–0.96 logits). All domains demonstrated acceptable data-to-model fit.

Table 2.

Calibration of Domains and Items for the Professional Development Quality Scale.

	Observed Average	Measure	Standard Error	InfitMSE	Std.Infit	Outfit MSE	Std.Outfit
Domain
Duration	1.41	1.02	0.14	1.15	2.10	1.34	2.70
Active learning	2.05	0.63	0.08	0.90	−1.40	0.93	−1.00
Feedback	2.19	0.54	0.08	1.00	0.00	1.10	0.70
Collaboration	2.19	0.43	0.08	0.98	−0.20	1.00	−0.10
Reflection	2.62	−0.25	0.07	0.88	−1.70	0.87	−1.70
Relevance	2.62	−0.26	0.08	0.81	−2.60	0.81	−2.70
Coherence	2.62	−0.31	0.08	1.09	1.00	1.16	1.30
Application	2.72	−0.44	0.08	0.97	−0.30	0.96	−0.50
Evidence/research	1.63	−0.77	0.15	1.04	0.80	1.08	0.90
Content	2.98	−0.96	0.09	0.92	−0.90	0.92	−1.00
Item
10.03_04	1.13	2.10	0.18	1.07	0.61	1.26	1.18
10.03_07	1.16	1.81	0.17	1.10	0.94	1.28	1.49
10.03_05	1.17	1.77	0.17	1.16	1.55	1.34	1.81
10.03_06	1.22	1.42	0.15	1.16	1.88	1.25	1.68
05.01_11	1.54	1.41	0.09	1.15	1.41	1.40	2.63
09.01_04	1.61	1.34	0.09	1.06	0.62	1.32	2.38
05.01_09	1.57	1.29	0.09	0.80	−2.10	0.80	−1.52
09.01_01	1.61	1.28	0.08	0.82	−2.01	0.81	−1.48
10.03_03	1.24	1.28	0.15	1.06	0.80	1.08	0.63
07.01_09	1.77	1.19	0.08	0.96	−0.44	0.98	−0.20
01.01_02	1.26	1.18	0.14	1.05	0.67	1.01	0.15
05.01_03	1.78	1.16	0.09	0.77	−2.87	0.77	−2.58
05.01_10	1.68	1.14	0.08	1.11	1.20	1.13	1.07
07.01_03	1.82	0.95	0.08	1.04	0.45	1.03	0.31
09.01_03	1.89	0.91	0.08	1.01	0.16	0.98	−0.20
05.01_04	1.96	0.82	0.08	0.88	−1.54	0.87	−1.46
07.01_06	1.93	0.76	0.07	0.98	−0.18	1.02	0.25
01.02_01	1.34	0.74	0.13	1.08	1.51	1.08	0.90
01.01_01	1.34	0.72	0.13	1.02	0.43	1.01	0.15
10.03_01	1.34	0.72	0.13	1.45	7.55	1.91	7.89
01.02_02	1.36	0.63	0.13	1.05	0.98	1.03	0.34
10.03_02	1.36	0.61	0.13	1.23	4.36	1.94	8.66
05.01_08	2.07	0.54	0.07	1.00	0.00	1.05	0.59
09.01_02	2.03	0.53	0.07	1.30	3.49	1.89	6.71
03.01_06	2.09	0.52	0.07	1.56	6.31	1.97	8.18
05.01_06	2.19	0.49	0.08	0.81	−2.61	0.82	−2.35
07.01_05	2.16	0.48	0.07	1.05	0.64	1.04	0.54
05.01_07	2.22	0.45	0.07	0.81	−2.83	0.84	−2.02
07.01_04	2.19	0.42	0.07	1.05	0.76	1.16	1.78
01.01_03	1.41	0.41	0.13	1.08	1.79	1.07	0.97
07.01_08	2.24	0.41	0.08	1.10	1.27	1.12	1.44
01.02_03	1.41	0.38	0.13	1.09	1.94	1.09	1.20
07.01_07	2.33	0.28	0.08	0.95	−0.69	0.94	−0.71
05.01_05	2.31	0.25	0.08	0.81	−2.51	0.81	−2.52
04.01_04	2.37	0.17	0.08	0.82	−2.48	0.82	−2.43
09.02_04	2.55	0.14	0.09	0.92	−1.01	0.92	−0.93
10.01	2.44	0.14	0.08	1.06	0.75	1.06	0.81
09.02_01	2.58	0.09	0.08	1.01	0.12	0.99	−0.13
08.01_04	2.43	0.03	0.07	1.01	0.19	1.02	0.21
09.02_03	2.61	0.03	0.09	0.95	−0.62	0.95	−0.57
09.02_02	2.66	−0.01	0.09	0.94	−0.66	0.97	−0.31
07.01_02	2.53	−0.06	0.08	0.88	−1.67	0.88	−1.65
03.01_04	2.51	−0.11	0.08	1.02	0.23	1.01	0.19
08.01_03	2.52	−0.12	0.07	0.85	−2.17	0.83	−2.32
06.02	2.58	−0.21	0.08	0.96	−0.56	0.94	−0.72
05.01_01	2.58	−0.25	0.08	0.79	−2.94	0.79	−2.86
01.02_04	1.55	−0.27	0.13	0.98	−0.55	0.98	−0.29
03.01_05	2.67	−0.29	0.08	0.98	−0.18	0.97	−0.38
01.01_04	1.56	−0.30	0.13	1.03	0.63	1.03	0.46
04.01_03	2.63	−0.30	0.08	0.79	−2.96	0.79	−2.98
05.01_02	2.65	−0.35	0.08	0.98	−0.24	0.97	−0.30
08.01_02	2.71	−0.35	0.07	0.82	−2.61	0.79	−2.83
01.02_07	1.58	−0.39	0.13	1.10	2.39	1.17	2.49
04.01_01	2.70	−0.39	0.08	0.80	−2.83	0.79	−2.92
01.01_07	1.59	−0.45	0.13	1.10	2.29	1.16	2.27
03.01_01	2.66	−0.46	0.09	0.87	−1.82	0.87	−1.75
03.01_02	2.75	−0.52	0.08	1.11	1.39	1.10	1.26
06.01	2.79	−0.54	0.08	1.02	0.34	1.02	0.30
04.01_02	2.78	−0.55	0.08	0.85	−2.06	0.83	−2.30
07.01_01	2.79	−0.56	0.08	0.83	−2.37	0.82	−2.51
06.03	2.77	−0.57	0.08	0.94	−0.79	0.92	−1.09
08.01_01	2.81	−0.57	0.08	0.84	−2.27	0.86	−1.84
01.02_05	1.63	−0.60	0.13	1.05	1.11	1.09	1.25
02.02	2.88	−0.68	0.08	0.90	−1.35	0.88	−1.59
10.02	1.65	−0.71	0.13	1.01	0.22	0.98	−0.23
02.01_01	2.82	−0.77	0.08	0.95	−0.66	0.94	−0.83
01.03_03	1.67	−0.85	0.13	1.07	1.34	1.17	1.90
01.01_05	1.68	−0.89	0.14	1.07	1.25	1.17	1.86
01.03_05	1.69	−0.92	0.14	1.04	0.75	1.13	1.47
03.01_03	3.06	−1.00	0.08	1.02	0.27	1.01	0.20
02.01_02	3.25	−1.42	0.10	0.93	−0.80	0.94	−0.66
01.03_04	1.78	−1.46	0.15	1.03	0.35	1.11	0.93
01.03_07	1.84	−1.90	0.17	1.12	1.10	1.51	2.76
01.03_01	1.84	−1.93	0.17	0.98	−0.16	0.91	−0.51
01.03_06	1.86	−2.02	0.18	1.00	0.00	1.15	0.89
01.02_06	1.89	−2.33	0.20	1.01	0.13	1.13	0.66
01.01_06	1.92	−2.78	0.23	0.96	−0.18	1.03	0.19
01.03_02	1.94	−3.16	0.27	0.95	−0.17	0.67	−1.02

Note. Domains and items are arranged in measure order from the least amount of participation to the most amount of participation. Observed Average refers to the average raw score (1–4) of all responses. Measure refers to the estimated linear measure based upon the transformation of observed average to logit units of measurement. Standard Error refers to the standard error of the measure estimate. Fit statistics answer the question, “How well do the data fit the model?” Infit MSE refers to the information-related fit and Outfit MSE refers to the outlier sensitive fit. Std. fit is the standardized version of MSE fit reported as a Z-score. MSE: mean square error.

Bias analysis (Research question 2)

Two bias analyses were conducted: (a) item-by-grade-level interaction analysis; and (b) item-by-teaching area interaction analysis.

Item-by-grade-level interaction

Overall, the analysis indicated a statistically significant omnibus test based upon grade level (= 333.20, p < .01), suggesting that bias occurs between items and grade level. A post hoc pairwise interaction analysis between all items (N = 78) and grade-levels (N = 3) indicated that of the total 234 interaction terms, 24 (10.3%) were found to be statistically significant (see Table 3). Of the 24 statistically significant interactions, four interactions had a moderate effect (|DIF| 0.43:0.64) and 11 interactions had a large effect (|DIF| >/= 0.64) (Zwick et al., 1999). Any bias sizes below 0.43 are considered to be negligible.

Table 3.

Significant Pairwise Interactions Between Items, Grade Level, and Teaching Area (|Z| ⩾ 2.00).

	Item Code	Item	Infit MSQ	Obs.-Exp.	Bias	SE	Z
Grade-Level
PreK-Elementary	PD_Q01.02_05	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in music teaching approaches (e.g., Kodaly, Orff, Dalcroze, etc.).	1.00	0.17	0.97	0.25	3.85
PreK-Elementary	PD_Q01.01_05	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in music teaching approaches (e.g., Kodaly, Orff, Dalcroze, etc.).	1.00	0.16	1.04	0.28	3.73
High School	PD_Q03.01_06	I have experienced professional development that has responded to district/state standardized testing results.	1.70	0.28	0.38	0.13	2.83
PreK-Elementary	PD_Q05.01_02	In the context of my professional development experiences, I have spent time engaging in hands-on activities.	1.10	0.19	0.35	0.13	2.74
PreK-Elementary	PD_Q03.01_03	I have experienced professional development that has aligned with national and/or state standards.	1.00	0.16	0.33	0.14	2.38
Middle School	PD_Q01.03_07	I have experienced professional development that addressed instructional strategies in instructional technology.	1.10	0.09	0.96	0.41	2.35
PreK-Elementary	PD_Q03.01_01	I have experienced professional development that has aligned with my planned student learning outcomes.	0.80	0.15	0.32	0.14	2.35
Middle School	PD_Q01.03_03	I have experienced professional development that addressed instructional strategies in classroom data collection.	1.00	0.10	0.54	0.26	2.08
PreK-Elementary	PD_Q04.01_03	I have experienced professional development that addressed my professional needs.	0.80	0.14	0.27	0.13	2.08
High School	PD_Q03.01_02	I have experienced professional development that has aligned with my school’s initiatives and goals.	1.00	0.17	0.35	0.17	2.06
PreK-Elementary	PD_Q04.01_02	I have experienced professional development that aligned with my teaching goals.	0.80	0.14	0.28	0.14	2.05
PreK-Elementary	PD_Q06.02	My professional development experiences have encouraged the application of newly acquired content knowledge (e.g., non-pedagogical knowledge) in my classroom.	1.10	0.14	0.26	0.13	2.04
PreK-Elementary	PD_Q04.01_01	I have experienced professional development that provided relevant resources that positively impacted my students’ learning.	0.90	0.14	0.27	0.13	2.03
PreK-Elementary	PD_Q09.01_01	As a part of my professional development experiences, I have received formal feedback from professional development facilitators.	0.20	0.75	1.00	0.50	2.01
High School	PD_Q04.01_03	I have experienced professional development that addressed my professional needs.	0.80	−0.19	−0.33	0.16	−2.09
High School	PD_Q05.01_02	In the context of my professional development experiences, I have spent time engaging in hands-on activities.	0.80	−0.19	−0.33	0.16	−2.09
PreK-Elementary	PD_Q01.02_07	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in any secondary teaching area in music not listed above	1.10	−0.10	−0.45	0.20	−2.25
High School	PD_Q05.01_07	In the context of my professional development experiences, I have spent time implementing lessons and/or curricula.	0.90	−0.23	−0.36	0.15	−2.36
PreK-Elementary	PD_Q03.01_06	I have experienced professional development that has responded to district/state standardized testing results.	1.50	−0.20	−0.30	0.12	−2.56
PreK-Elementary	PD_Q01.01_07	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in another secondary-teaching area not listed above	1.10	−0.11	−0.52	0.20	−2.58
PreK-Elementary	PD_Q01.03_03	I have experienced professional development that addressed instructional strategies in classroom data collection	1.10	−0.11	−0.55	0.20	−2.70
PreK-Elementary	PD_Q01.03_07	I have experienced professional development that addressed instructional strategies in instructional technology	1.20	−0.09	−0.67	0.23	−2.88
High School	PD_Q01.02_05	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in music teaching approaches (e.g., Kodaly, Orff, Dalcroze, etc.)	0.90	−0.17	−0.75	0.25	−2.99
High School	PD_Q01.01_05	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in music teaching approaches (e.g., Kodaly, Orff, Dalcroze, etc.)	1.10	−0.24	−1.10	0.25	−4.36
Teaching Area
Secondary	PD_Q06.01	My professional development experiences have encouraged the application of newly acquired pedagogical knowledge (e.g., how to teach) in my classroom.	1.40	0.54	1.04	0.31	3.33
Secondary	PD_Q06.02	My professional development experiences have encouraged the application of newly acquired content knowledge (e.g., non-pedagogical knowledge) in my classroom.	1.20	0.50	0.88	0.28	3.11
Both	PD_Q10.03_02	I have participated in professional development experiences that consisted of multiple, related sessions spread over less than 1 week.	1.30	−0.12	−0.65	0.30	−2.18
Secondary	PD_Q07.01_03	In the context of my professional development experiences, I have spent time collaborating with colleagues online.	1.40	−0.36	−0.85	0.38	−2.26

Note. Items listed in order of Z size. Shaded areas indicate bias >/= |.43| logits, which suggest a moderate to large effect on estimations (Zwick, Thayer, and Lewis, 1999). Positive Z-scores indicate group patterns significantly higher than expected by the model. Negative Z-scores indicate group patterns significantly lower than expected by the model.

Item-by-teaching area interaction

Overall, the analysis indicated a statistically significant omnibus test based upon teaching area (= 342.10, p < .01), suggesting bias occurs between items and teaching area. A post hoc pairwise interaction analysis between all items (N = 78) and teaching areas (N = 3) indicated that of the total 234 interaction terms, four (1.70%) were found to be statistically significant (See Table 3). All four interactions had a large effect (|DIF| >/= 0.64) (Zwick et al., 1999).

Sentiment analysis (Research question 3)

The sentiment analysis resulted in an overall positive valence for respondents, with a mean aggregated sentiment index score of 0.18 (SD = 0.69). A Pearson’s r correlation between respondents’ aggregated sentiment scores and respondents’ Rasch measures was 0.42 (p < .05), indicating a statistically significant, moderately positive relationship. This suggests that as respondents’ experiences with high-quality PD increase, sentiments toward a positive valence also increases.

A total of 482 positive and negative descriptor responses were identified. 324 (67.22%) descriptors were identified as positive valence and 158 (32.78%) descriptors were identified as negative valence. Of the 482 total descriptor responses, 86 (56.20%) unique descriptors were identified as positive valence and 67 (43.80%) unique descriptors were identified as negative valence.

Upon examining the emotional distribution of responses, all eight emotions of the lexicon were represented: trust (33.60%), joy (27.5%), anticipation (9.20%), sadness (8.10%), anger (7.50%), disgust (7.10%), surprise (6.10%), and fear (1.00%). Figure 2 provides a breakdown of the most frequently used positive and negative descriptors with a minimum of three use cases as well as the overall distribution of emotions underscoring the sentiments.

Figure 2.

Descriptive statistics for the sentiment analysis. Panel A: Counts of Most Frequently Used Descriptors: Positive Valence. Panel B: Counts of Most Frequently Used Descriptors: Negative Valence. Panel C: Frequency (%) of Emotional Distribution for All Descriptors.

Conclusion and discussion

The purpose of this study was to develop a valid and reliable assessment tool to measure music educators’ participation in and perceptions of high-quality PD experiences in the United States. In answer to research question one (What is the ordering and fit of items/domains representing participation levels of music educators’ high-quality PD experiences?), results suggest a clear and reliable ordering of domains, items, and respondents. The significant and high reliability of separation for the items and domains provide empirical support for the strong construct validity of the measure. The significant and high reliability of separation for the respondents provide empirical support for respondents’ shared understanding of the construct (Lopez, 1998; Wright & Linacre, 1992). The ordering of domains from least participation to most participation is: (a) duration, (b) active learning, (c) feedback, (d) collaboration, (e) reflection, (f) relevance, (g) coherence, (h) application, (i) evidence/research, and (j) content.

In answer to research question two (To what extent does bias exist in music educators’ PD experiences based upon their teaching area and/or grade level?), results suggest that music educators experience different participation levels of high-quality PD experiences based upon teaching area and grade level. A total of 28 significant interactions were identified across grade-level and teaching area, with 11 interactions demonstrating moderate to large effect sizes for grade-level and 4 interactions demonstrating moderate to large effect sizes for teaching area.

In answer to research question three (Is there a relationship between the quality of PD music educators experiences and their sentiments and emotions toward those experiences?), results suggest that music educators’ sentiments were moderately and positively correlated with their experiences of high-quality PD. This suggests that the higher quality PD music educators experience, the more positive their sentiment toward those experiences.

The results of this study suggest that each of the 10 domains adequately describes the quality of PD in the field of music education. While the teachers’ needs, perceptions, preferences, and motivations of PD are important when designing and evaluating PD programs, there is a need for continued evaluation, consideration, and inclusion of the criteria that define “high-quality PD.” Results suggest that music educators are experiencing an abundance of music-specific PD that is content-rich, evidence-based, embedded, and coherent to policy, standards, teacher needs, and preferences. These findings corroborate with previous research regarding music-specific PD that is focused on content needs due to the broad music teaching certificate and alternative music classrooms where teachers may be teaching unfamiliar musical content (Grieser & Hendricks, 2018; May et al., 2017; West, 2020).

The results of this study also suggest that music educators are less likely to be experiencing music-specific PD that (a) is longitudinal in nature, (b) has active learning opportunities, (c) provides adequate feedback, and (d) is collaborative. These findings support research regarding teachers’ experiences in music-specific PD. In comparing PD activities of music educators with those in other disciplines, Gallo (2018) suggested that music educators participate in content-focused programs, but that the lengths of programs fall short of hours recommended by researchers to change instructional practices and improve student achievement. Gallo additionally reported that there is not enough information regarding collaboration within music education PD to make concrete statements. Results from this study suggest that music educators are less likely to experience collaborative activities and engagement in their PD experiences. Secondary content area teachers are even less likely to experience collaborative PD activities than their peers who teach only a primary content area. Research suggests that many music educators feel isolated in their school environment, which may inhibit effective collaboration within their local school community (Bautista et al., 2017; Conkling & Henry, 2002; Grieser & Hendricks, 2018). In addition, lack of professional learning communities that are readily accessible within local schools for music educators, the prevalence of “one-size-fits-all” PD, and the trend of focusing on general pedagogical strategies for an entire school faculty may contribute to the decreased music PD experiences that incorporate collaboration, feedback, active learning opportunities, and adequate duration (Bautista et al., 2018).

The results to the second question highlight the systematic differences of educators’ experiences of quality PD across content and grade levels. Elementary educators experienced more content-specific PD and received more formal feedback from PD facilitators than middle and high school music educators. However, elementary educators experienced less PD focusing on instructional technology, data collection, and content or pedagogical knowledge regarding secondary teaching areas. Middle school music educators received more PD that includes instructional technology and data collection. High school music educators received less PD that includes content and pedagogical knowledge in music teaching approaches. Specifically identifying Kodaly, Orff, and Dalcroze in the items, however, may influence these findings. We suggest more specific research examining and comparing PD programs as they relate to differing needs of grade levels and content areas.

Findings regarding differences between teaching areas suggest that music educators teaching a secondary content area experience more PD that encourages the application of newly acquired pedagogical knowledge and content knowledge than others. Music educators teaching both primary and secondary content areas experience less longitudinal PD than their counterparts who teach one content area. Perhaps the challenges inherent in managing two content areas inhibit teachers’ abilities to experience more continuous and longitudinal applications of PD in their classrooms.

From a policy perspective, data evaluation has been identified as an important part of implementing and assessing PD effectiveness with respect to the allocation of funding and other resources (Eraut, 1994). However, inadequate data available due to a lack of systematic PD assessments often leaves stakeholders unable to examine PD effectiveness and impact (Bredeson, 2002). Government is now where many crucial decisions about policy implementation are being made, and it is important that district leaders support meaningful policies (Johnson & Uline, 2005). Unfortunately, little attention has been given to identifying PD opportunities as they relate to the allocation of appropriate funding. Current research literature highlights the importance of using valid assessment tools to evaluate the quality of PD utilized by local school districts across the United States (Bredeson, 2002; Eraut, 1994; Guskey, 2000). We hope that the survey developed in this study may be used to provide valid and reliable data as to music educators’ participation in high-quality PD experiences.

Footnotes

Appendix

Appendix 1.

Quality of professional development survey: Items, domains, and response category codings.

ID	Domain	Item	1	2	3	4
01.01_01	Content	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in music theory	No	Yes
01.01_02	Content	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in music history	No	Yes
01.01_03	Content	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in music composition	No	Yes
01.01_04	Content	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in world music	No	Yes
01.01_05	Content	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in music teaching approaches (e.g., Kodaly, Orff, Dalcroze, etc.)	No	Yes
01.01_06	Content	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in performance/pedagogical techniques specific to my area (e.g. band, orchestra, choir, general music, etc.)	No	Yes
01.01_07	Content	I have experienced professional development that increased my pedagogical knowledge (e.g., how to teach) in another secondary-teaching area not listed above	No	Yes
01.02_01	Content	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in music theory	No	Yes
01.02_02	Content	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in music history	No	Yes
01.02_03	Content	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in music composition	No	Yes
01.02_04	Content	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in world music	No	Yes
01.02_05	Content	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in music teaching approaches (e.g., Kodaly, Orff, Dalcroze, etc.)	No	Yes
01.02_06	Content	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in performance/pedagogical techniques specific to my primary teaching area (e.g., band, orchestra, choir, general music, etc.)	No	Yes
01.02_07	Content	I have experienced professional development that increased my content knowledge (e.g., non-pedagogical knowledge) in any secondary teaching area in music not listed above	No	Yes
01.03_01	Content	I have experienced professional development that addressed instructional strategies in assessment methods	No	Yes
01.03_02	Content	I have experienced professional development that addressed instructional strategies in teaching methods	No	Yes
01.03_03	Content	I have experienced professional development that addressed instructional strategies in classroom data collection	No	Yes
01.03_04	Content	I have experienced professional development that addressed instructional strategies in classroom management	No	Yes
01.03_05	Content	I have experienced professional development that addressed instructional strategies in learning theories	No	Yes
01.03_06	Content	I have experienced professional development that addressed instructional strategies in differentiated instruction	No	Yes
01.03_07	Content	I have experienced professional development that addressed instructional strategies in instructional technology	No	Yes
02.01_01	Evidence/research	I have had professional development experiences that were informed by research.	Never/Rarely	Occasionally	Frequently	Usually/Always
02.01_02	Evidence/research	I have had professional development experiences that were informed by best practices.	Never/Rarely	Occasionally	Frequently	Usually/Always
2.02	Evidence/research	My professional development facilitators introduced instructional methods for which there is evidence of effectiveness.	Never/Rarely	Occasionally	Frequently	Usually/Always
03.01_01	Coherence	I have experienced professional development that has aligned with my planned student learning outcomes.	Never/Rarely	Occasionally	Frequently	Usually/Always
03.01_02	Coherence	I have experienced professional development that has aligned with my school’s initiatives and goals.	Never/Rarely	Occasionally	Frequently	Usually/Always
03.01_03	Coherence	I have experienced professional development that has aligned with national and/or state standards.	Never/Rarely	Occasionally	Frequently	Usually/Always
03.01_04	Coherence	I have experienced professional development that has informed me of national, state, and/or district policies.	Never/Rarely	Occasionally	Frequently	Usually/Always
03.01_05	Coherence	I have experienced professional development that has reflected current trends in national and/or state policies.	Never/Rarely	Occasionally	Frequently	Usually/Always
03.01_06	Coherence	I have experienced professional development that has responded to district/state standardized testing results.	Never/Rarely	Occasionally	Frequently	Usually/Always
04.01_01	Relevance	I have experienced professional development that provided relevant resources that positively impacted my students’ learning.	Never/Rarely	Occasionally	Frequently	Usually/Always
04.01_02	Relevance	I have experienced professional development that aligned with my teaching goals.	Never/Rarely	Occasionally	Frequently	Usually/Always
04.01_03	Relevance	I have experienced professional development that addressed my professional needs.	Never/Rarely	Occasionally	Frequently	Usually/Always
04.01_04	Relevance	I have experienced professional development that was directly influenced by participating teachers’ input.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_01	Active learning	In the context of my professional development experiences, I have spent time participating in meaningful discussions.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_02	Active learning	In the context of my professional development experiences, I have spent time engaging in hands-on activities.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_03	Active learning	In the context of my professional development experiences, I have spent time analyzing examples of student work.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_04	Active learning	In the context of my professional development experiences, I have spent time analyzing my own students’ learning and progress.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_05	Active learning	In the context of my professional development experiences, I have spent time planning to implement professional development resources.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_06	Active learning	In the context of my professional development experiences, I have spent time planning lessons and/or curricula.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_07	Active learning	In the context of my professional development experiences, I have spent time implementing lessons and/or curricula.	Never/Rarely	Occasionally	Frequently	Usually/Always
050.1_08	Active learning	In the context of my professional development experiences, I have spent time observing expert teachers.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_09	Active learning	In the context of my professional development experiences, I have spent time being observed by expert teachers.	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_10	Active learning	In the context of my professional development experiences, I have spent time presenting (e.g., conferences, in-service meetings, etc.).	Never/Rarely	Occasionally	Frequently	Usually/Always
05.01_11	Active learning	In the context of my professional development experiences, I have spent time producing written work (e.g., written articles, action research reports, standards/assessment development, etc.).	Never/Rarely	Occasionally	Frequently	Usually/Always
6.01	Application	My professional development experiences have encouraged the application of newly acquired pedagogical knowledge (e.g., how to teach) in my classroom.	Never/Rarely	Occasionally	Frequently	Usually/Always
6.02	Application	My professional development experiences have encouraged the application of newly acquired content knowledge (e.g., non-pedagogical knowledge) in my classroom.	Never/Rarely	Occasionally	Frequently	Usually/Always
6.03	Application	My professional development experiences have encouraged the application of newly acquired instructional strategies in my classroom.	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_01	Collaboration	In the context of my professional development experiences, I have spent time discussing teaching concepts and skills with others.	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_02	Collaboration	In the context of my professional development experiences, I have spent time solving problems with my colleagues.	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_03	Collaboration	In the context of my professional development experiences, I have spent time collaborating with colleagues online.	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_04	Collaboration	In the context of my professional development experiences, I have spent time engaging with others in Professional Learning Communities (PLCs).	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_05	Collaboration	In the context of my professional development experiences, I have spent time acting as a mentor and/or mentee.	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_06	Collaboration	In the context of my professional development experiences, I have spent time working jointly with students, parents, and/or community members.	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_07	Collaboration	In the context of my professional development experiences, I have spent time working with more experienced music content experts.	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_08	Collaboration	In the context of my professional development experiences, I have spent time working with more experienced education (e.g., non-music) content experts.	Never/Rarely	Occasionally	Frequently	Usually/Always
07.01_09	Collaboration	In the context of my professional development experiences, I have spent time partnering with independent institutions (e.g., higher education, non-profit organizations, etc.).	Never/Rarely	Occasionally	Frequently	Usually/Always
08.01_01	Reflection	As part of my professional development, I have been encouraged to analyze the effectiveness of my teaching.	Never/Rarely	Occasionally	Frequently	Usually/Always
08.01_02	Reflection	As part of my professional development, I have been encouraged to evaluate a variety of teaching practices and strategies.	Never/Rarely	Occasionally	Frequently	Usually/Always
08.01_03	Reflection	As part of my professional development, I have been encouraged to participate in reflective discussion about my teaching.	Never/Rarely	Occasionally	Frequently	Usually/Always
08.01_04	Reflection	As part of my professional development, I have been encouraged to assess the effectiveness of the professional development content.	Never/Rarely	Occasionally	Frequently	Usually/Always
09.01_01	Feedback	As a part of my professional development experiences, I have received formal feedback from professional development facilitators.	Never/Rarely	Occasionally	Frequently	Usually/Always
09.01_02	Feedback	As a part of my professional development experiences, I have received formal feedback from school administrators.	Never/Rarely	Occasionally	Frequently	Usually/Always
09.01_03	Feedback	As a part of my professional development experiences, I have received formal feedback from colleagues.	Never/Rarely	Occasionally	Frequently	Usually/Always
09.01_04	Feedback	As a part of my professional development experiences, I have received formal feedback from students.	Never/Rarely	Occasionally	Frequently	Usually/Always
09.02_01	Feedback	Any formal feedback I have received regarding the implementation of my professional development has been timely.	Never/Rarely	Occasionally	Frequently	Usually/Always
09.02_02	Feedback	Any formal feedback I have received regarding the implementation of my professional development has been appropriate.	Strongly Disagree	Disagree	Agree	Strongly Agree
09.02_03	Feedback	Any formal feedback I have received regarding the implementation of my professional development has been satisfactory.	Strongly Disagree	Disagree	Agree	Strongly Agree
09.02_04	Feedback	Any formal feedback I have received regarding the implementation of my professional development has been useful.	Strongly Disagree	Disagree	Agree	Strongly Agree
10.01	Duration	During my professional development experiences, ample time was provided to achieve the planned objectives.	Never/Rarely	Occasionally	Frequently	Usually/Always
10.02	Duration	I have participated in a single professional development experience that consisted of multiple, related sessions spread over a period of time (i.e., non-consecutive days).	No	Yes
10.03_01	Duration	I have participated in professional development experiences that consisted of multiple, related sessions spread over 1 day	No	Yes
10.03_02	Duration	I have participated in professional development experiences that consisted of multiple, related sessions spread over less than 1 week	No	Yes
10.03_03	Duration	I have participated in professional development experiences that consisted of multiple, related sessions spread over 1 week	No	Yes
10.03_04	Duration	I have participated in professional development experiences that consisted of multiple, related sessions spread over 1 month	No	Yes
10.03_05	Duration	I have participated in professional development experiences that consisted of multiple, related sessions spread over 3 months	No	Yes
10.03_06	Duration	I have participated in professional development experiences that consisted of multiple, related sessions spread over 4–12 months	No	Yes
10.03_07	Duration	I have participated in professional development experiences that consisted of multiple, related sessions spread over 1 year or longer	No	Yes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Brian C Wesolowski

Michael A Alsop

References

Adey

(2004). The professional development of teachers: Practice and theory. Kluwer Academic Publishers.

Andrews

B. W.

(2016). Towards the future: Teachers’ vision of professional development in the arts. International Journal of Music Education, 34(4), 391–402. https://doi.org/10.1177/0255761415590361

Bauer

W. I.

(2007). Research on professional development for experienced music teachers. Journal of Music Teacher Education, 17(1), 12–21. https://doi.org/10.1177/10570837070170010105

Bautista

Brizuela

B. M.

Glennie

C. R.

Caddle

M. C.

(2014). Mathematics teachers attending and responding to students’ thinking: Diverse paths across diverse assignments. International Journal for Mathematics Teaching & Learning, July, 1–28.

Bautista

Toh

G. Z.

Wong

(2018). Primary school music teachers’ professional development motivations, needs, and preferences: Does specialization make a difference? Musicae Scientiae, 22(2), 196–223. https://doi.org/10.1177/1029864916678654

Bautista

Yau

Wong

(2017). High-quality music teacher professional development: A review of the literature. Music Education Research, 19(4), 455–469. https://doi.org/10.1080/14613808.2016.1249357

Blank

R. K.

(2013). What research tells us: Common characteristics of professional learning that leads to student achievement. Journal of Staff Development, 34(1), 50–53.

Bond

T. G.

(2003). Validity and assessment: A Rasch measurement perspective. Metodología de las Ciencias del Comportamiento, 5(2), 179–194.

Bowles

(2002). The self-expressed professional development needs of music educators. Update, 21(1), 24–28. https://doi.org/10.1177/87551233020210020701

10.

Boyle

Lamprianou

Boyle

(2005). A longitudinal study of teacher change: What makes professional development effective? Report of the second year of the study. School Effectiveness and School Improvement, 16(1), 1–17. https://doi.org/10.1080/09243450500114819

11.

Bredeson

P. V.

(2002). The architecture of professional development: Materials, messages and meaning. International Journal of Educational Research, 37, 661–675. https://doi.org/10.1016/S0883-0355(03)00064-8

12.

Bush

J. E.

(2007). Importance of various professional development opportunities and workshop topics as determined by in-service music teachers. Journal of Music Teacher Education, 16(2), 10–18. https://doi.org/10.1177/10570837070160020103

13.

Buysse

Winton

P. J.

Rous

(2009). Reaching consensus on a definition of professional development for the early childhood field. Topics in Early Childhood Special Education, 28(4), 235–243. https://doi.org/10.1177/0271121408328173

14.

Conkling

S. W.

Henry

(2002.). The impact of professional development partnerships: Our part of the story. Journal of Music Teacher Education, 11(2), 7–13. https://doi.org/10.1177%2F105708370201100203

15.

Conway

(2007). Setting an agenda for professional development policy, practice, and research in music education. Journal of Music Teacher Education, 17(1), 56–61. https://doi.org/10.1177/10570837070170010109

16.

Conway

(2011). Professional development of experienced music teachers: Special focus issue. Arts Education Policy Review, 112(2), 55–59. https://doi.org/10.1080/10632913.2011.545751

17.

Conway

C. M.

(2008). Experienced music teacher perceptions of professional development throughout their careers. Bulletin of the Council for Research in Music Education, 176, 7–18. https://www.jstor.org/stable/40319429

18.

Darling-Hammond

McLaughlin

M. W.

(2011). Policies that support professional development in an era of reform. Phi Delta Kappan, 92(6), 81–95. https://doi.org/10.1177/003172171109200622

19.

Darling-Hammond

Richardson

(2009). Research review/teacher learning: What matters? Educational Leadership, 66(5), 46–53.

20.

Desimone

L. M.

(2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199. https://doi.org/10.3102/0013189X08331140

21.

Desimone

L. M.

(2011). A primer on effective professional development. Phi Delta Kappan, 92(6), 68–71. https://doi.org/10.1177/003172171109200616

22.

Desimone

L. M.

Porter

A. C.

Birman

B. F.

Garet

M. S.

Yoon

K. S.

(2002). How do district management and implementation strategies relate to the quality of the professional development that districts provide to teachers? Teachers College Record, 104(7), 1265–1312.

23.

Desimone

L. M.

Porter

A. C.

Garet

M. S.

Yoon

K. S.

Birman

B. F.

(2002). Effects of professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81–112. https://doi.org/10.3102/01623737024002081

24.

Ekman

(1992). An argument for basic emotions. Cognition and Emotion, 6(3), 169–200. https://doi.org/10.1080/02699939208411068

25.

Eraut

(1994). Developing professional knowledge and competence. The Falmer Press.

26.

Feiman-Nemser

(2001). From preparation to practice: Designing a continuum to strengthen and sustain teaching. Teachers College Record, 103(6), 1013–1055. http://hdl.handle.net/10192/33196

27.

Feinerer

Hornik

(2018). Text mining package (tm) (R package version 0.7-6).

28.

Ferrara

D. J.

(2009). Professional development for music educators: A proposed model and sample program relative to teacher preferences (Dissertation No. 3414063) [Doctoral dissertation, Rutgers The State University of New Jersey]. ProQuest Dissertations & Theses Global.

29.

Forde

McMahon

McPhee

Patrick

(2006). Professional development, reflection, and enquiry. SAGE.

30.

Gallo

D. J.

(2018). Professional development quality in U.S. music education: An analysis of the 2011-2012 schools and staffing survey. Journal of Research in Music Education, 66(2), 168–189. https://doi.org/10.1177/0022429418764453

31.

Garet

M. S.

Porter

A. C.

Desimone

Birman

B. F.

Yoon

K. S.

(2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915–945. https://www.jstor.org/stable/3202507

32.

Gibbs

G. R.

(2007). Thematic coding and categorizing. In U. Flick (Ed.), Analyzing qualitative data (pp. 33–55). SAGE.

33.

Grieser

D. R.

Hendricks

K. S.

(2018). Review of literature: Pedagogical content knowledge and string teacher preparation. Update, 37(1), 13–19. https://doi.org//10.1177/8755123318760970

34.

Guskey

T. R.

(2000). Evaluating professional development. Corwin.

35.

Guskey

T. R.

(2014). Planning professional learning. Educational Leadership, 71(8), 10–16. https://uknowledge.uky.edu/edp_facpub/15

36.

Guskey

T. R.

Yoon

K. S.

(2009). What works in professional development? Phi Delta Kappan, 90(7), 495–500. https://www.jstor.org/stable/20446159

37.

Helterbran

V. R.

Fennimore

B. S.

(2004). Collaborative early childhood professional development: Building from a base of teacher investigation. Early Childhood Education, 31(4), 267–271. https://doi.org/10.1023/B:ECEJ.0000024118.99085.ff

38.

Hiebert

(1999). Relationships between research and the NCTM standards. Journal for Research in Mathematics Education, 30(1), 3–19. https://www.jstor.org/stable/749627

39.

Jockers

(2017). Extracts sentiment and sentiment-derived plot arcs from text (syuzhet) (R package version 1.0.4).

40.

Johnson

J. F.

Jr. Uline

C. L.

(2005). Preparing educational leaders to close achievement gaps. Theory in Practice, 44(1), 45–52. https://doi.org/10.1207/s15430421tip4401_7

41.

Kennedy

M. M.

(1998). Form and substance in in-service teacher education (Research Monograph No. 13). National Science Foundation.

42.

Kiritchenko

Mohammad

S. M.

(2014, June). The effect of negators, modals, and degree of adverbs on sentiment composition [paper presntation]. NAACL 2016 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), San Diego, CA.

43.

Kiritchenko

Mohammad

S. M.

(2016, June). Capturing reliable fine-grained sentiment associations by crowdsourcing and best-working scale [Conference session]. 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA.

44.

Koner

Eros

(2019). Professional development for the experienced music educator: A review of recent literature. Update, 37(3), 12–19. https://doi.org/10.1177/8755123318812426

45.

Kyriakides

Creemers

B. P. M.

Antoniou

(2009). Teacher behavior and student outcomes: Suggestions for research in teacher training and professional development. Teaching and Teacher Education, 25, 12–23. https://doi.org/10.1016/j.tate.2008.06.001

46.

Leong

W. S.

Tan

(2019). Policy and practice of assessment in music education: The case of Singapore. In Brophy

T. S.

(Ed.), The Oxford handbook of assessment policy and practice in music education (Vol. 1, pp. 357–378). Oxford University Press.

47.

Linacre

J. M.

(1994a). Many-facet Rasch measurement. MESA Press. (Original work published 1989)

48.

Linacre

J. M.

(1994b). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.

49.

Linacre

J. M.

(2004). Test validity and Rasch measurement: Construct, content, etc. Rasch Measurement Transactions, 18(1), 970–971.

50.

Linacre

J. M.

(2014). Facets (Version 3.71.4)[Computer software]. MESA Press.

51.

Linacre

J. M.

Wright

B. D.

(1994). Reasonable mean square fit values. Rasch Measurement Transactions, 8(3), 370.

52.

Little

Hayes

(2003). Continuing professional development (CPD): GPs’ perceptions of post-graduate education-approved (PGEA) meetings and personal professional development plans (PDPs). Family Practice, 20(2), 192–198. https://doi.org/10.1093/fampra/20.2.192

53.

Liu

(2015). Sentiment analysis: Mining sentiments, opinions, and emotions. Cambridge University Press.

54.

Lopez

W. A.

(1998). Rating scales and shared meaning. Popular Measurement, 1(1), 60.

55.

Loucks-Horsley

(2010). Designing professional development for teachers of science and mathematics (3rd ed.). Corwin.

56.

Lydon

King

(2009). Can a single, short continuing professional development workshop cause change in the classroom? Professional Development in Education, 35(1), 63–82. https://doi.org/10.1080/13674580802264746

57.

Mandouit

(2018). Using student feedback to improve teaching. Educational Action Research, 26(5), 755–769. https://doi.org/10.1080/09650792.2018.1426470

58.

Mäntylä

M. V.

Graziotin

Kuutila

(2018). The evolution of sentiment analysis—A review of research topics, venues, and top cited papers. Computer Science Review, 27, 16–32. https://doi.org/10.1016/j.cosrev.2017.10.002

59.

May

B. N.

Willie

Worthen

Pehrson

(2017). An analysis of state music education certification and licensure practices in the United States. Journal of Music Teacher Education, 27(1), 65–88. https://doi.org/10.1177%2F1057083717699650

60.

McCaffrey

Lovins

(2019). The status of arts assessment in the United States. In Brophy

T. S.

(Ed.). The Oxford handbook of assessment policy and practice in music education (Vol. 2). https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780190248130.001.0001/oxfordhb-9780190248130-e-42

61.

Mohammad

(2012). #emotional tweets. In Agirrem

Bos

Diab

Manandhar

Marton

Yuret

(Eds.), SEM 2012: The first joint conference on lexical and computation semantics- Vol 1: Proceedings of the main conference and the shared task and Vol 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) (pp. 278–281). Association of Computation Linguistics.

62.

Mohammad

S. M.

Turney

P. D.

(2010). Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Inkpen

Strapparava

(Eds.), Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text (pp. 26–34). Association for Computational Linguistics.

63.

Peixotto

Fager

(1998). High-quality professional development: An essential component of successful schools. Northwest Regional Educational Laboratory.

64.

Pellegrino

Kastner

J. D.

Reese

Russell

H. A.

(2018). Examining the long-term impact of participating in a professional development community of music teacher educators in the USA: An anchor through turbulent transitions. International Journal of Music Education, 36(2), 145–159. https://doi.org/10.1177/0255761417704214

65.

Pluchik

(1980). Emotion: A psychoevolutionary synthesis. Harper & Row.

66.

Ragland

R. G.

(2017). Impact of discipline-specific professional development: Implementation of core instructional practices. Teacher Development, 21(1), 118–141. https://doi.org/10.1080/13664530.2016.1185030

67.

Rasch

(1980). Probabilistic models for some intelligence and attainment tests. MESA Press. (Original work published 1960)

68.

R Core Team. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. http://www.R-project.org/

69.

Reeves

D. B.

(2010). Transforming professional development into student results. ASCD.

70.

Rinker

(2019). Calculate text polarity sentiment (SentimentR) (R package version 2.7.1).

71.

Shin

Seog

(2018). A collaborative group study of Korean mid-career elementary teachers for professional development in music. International Journal of Music Education, 36(1), 85–95. https://doi.org/10.1177/0255761417704011

72.

Smith

A. B.

Rush

Fallowfield

L. J.

Veilikova

Sharpe

(2008). Rasch fit statistics and sample size considerations for polytomous data. BMC Medical Research Methodology, 8, Article 33. https://doi.org/10.1186/1471-2288-8-33

73.

Stanley

A. M.

Snell

Edgar

(2014). Collaboration as effective music professional development: Success stories from the field. Journal of Music Teacher Education, 24(1), 76–88. https://doi.org/10.1177/1057083713502731

74.

State Education Agency Directors of Arts Education. (2016). Status of arts assessment in the USA. https://www.nationalartsstandards.org/sites/default/files/NCCAS-State-Reports-Since-2014.pdf

75.

Wei

R. C.

Darling-Hammond

Andree

Richardson

Orphanos

(2009). Professional learning in the learning profession: A status report on teacher development in the United States and abroad. National Staff Development Council.

76.

Wells

(2014). Elements of effective and sustainable professional learning. Professional Development in Education, 40(3), 488–504. https://doi.org/10.1080/19415257.2013.838691

77.

Wesolowski

B. C.

(2019). Item response theory in music testing. In Brophy

(Ed.), The Oxford handbook of assessment policy and practice in music education (pp. 437–460). Oxford University Press.

78.

West

J. J.

(2020). Is music teacher professional development becoming more effective? Evaluating practice and policy in the United States, 1993-2012. Journal of Research in Music Education. Advance online publication. https://doi.org/10.1177/0022429420982525

79.

Wright

B. D.

(2000). Rasch analysis for surveys. Popular Measurement, 3(1), 61.

80.

Wright

B. D.

Linacre

J. M.

(1992). Combining and splitting categories. Rasch Measurement Transactions, 6(3), 233.

81.

Wright

B. D.

Linacre

J. M.

(1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370.

82.

Wright

B. D.

Masters

G. N.

(1982). Rating scale analysis. MESA Press.

83.

Wright

B. D.

Tennant

(1996). Sample size again. Rasch Measurement Transactions, 9(4), 468.

84.

Zwick

Thayer

D. T.

Lewis

(1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36(1), 1–28. https://doi.org/10.1111/j.1745-3984.1999