Triangulated Research Methods to Operationalize Career and Technical Education Administrator Knowledge and Skill Core Competencies in Pennsylvania

Abstract

The position of the career and technical education (CTE) administrator is generally well-known but the multitude of tasks (e.g., management/leadership) are not widely understood by those outside CTE. The purpose of this study was to operationalize and establish a set of knowledge and skill core competencies (KSCC) nested in specific focus areas for CTE leadership training in Pennsylvania. The study used triangulated multi-method research procedures to establish consensus. The resulting KSCC frame standards for CTE leadership training for administrator candidates in Pennsylvania and can be used to establish a national skill set. The KSCC compliments the NELP (2018) Standards.

Keywords

leadership knowledge and skills carrer and technical education leadership management secondary education

Career and technical education (CTE) has been an important part of the secondary public school curriculum since the Smith-Hughes Act of 1917 (PL 64-347). CTE has had an impact on occupational training in the United States at both the secondary and postsecondary level (Imperatore & Hyslop, 2017). Although having seen a decline in enrollment from 1990 to 2009 (U.S. Department of Education [National Center for Education Statistics], 2013), CTE has emerged as a viable program of study with the push for STEM/STEAM education, college and career readiness (CCR), and an emphasis on career pathways (American Institutes for Research (AIR), 2013). The recently reauthorized CTE legislation (Perkins V) continues CTE’s mission of teaching rigorous academic and technical skills for high-skill, high-wage, and in-demand occupations (Strengthening Career and Technical Education for the 21st Century Act of 2018, PL 115-224). Data from the U.S. Department of Education [National Center for Education Statistics] (2019) indicate that nearly 77% of all high school students have taken one of more classes in CTE. In addition, CTE courses were identified (91%) as an important strategy to support student engagement and motivation in public education (Education Week Research Center, 2014). Having a workforce that is prepared to support the economy of the 21st century and is the focus of CCR is important (Cushing et al., 2017). Students participating in CTE are more prepared in planning for their careers and have more defined career decision-making compared to non-CTE students (Mobley et al., 2017). With increased interest in CTE, CCR, and the need to address the skills shortage in technical occupations, CTE administrators need to be prepared to lead CTE in its historical role and meet the needs of CTE’s emerging importance in STEM, CCR, and career pathways for all students. Having a specialty set of CTE Leadership standards to guide administrative practice is essential to address these challenges.

The field of career and technical education (CTE) is rapidly changing (Rojewski & Hill, 2014; Zirkle & Martin, 2012). The complex role of the CTE administrator is well documented but not well understood by those outside the profession (Clark & Cole, 2015; Gregson & Allen, 2005). There has been a push to identify broadly defined leadership skills needed in educational administration for all educational leaders, as espoused by the National Policy Board for Educational Administration (2018) National Educational Leadership Preparation (NELP) Program Standards. These standards provide an excellent framework for skills in educational leadership needed in today’s schools but they do not capture the essential knowledge and skills needed by the CTE administrator specific to management and leadership core competencies required to successfully run a CTE school/program (Zirkle & Jeffery, 2017).

This issue is exacerbated by the fact that several states do not have specific administrative licensure or certification requirements to serve as an administrator of CTE. Currently only 16 states have such requirements as noted by Zirkle and Jeffery. This is a significant decline (–47%) from 1979 with 34 states having specific requirements in 1979 (Bowers, 1979), 31 states in 1989 (Kraska, 1989), 26 states in 1998 (Zirkle, 1998), 20 states in 2005 (Zirkle et al., 2005), and the current 16 states in 2017 (Zirkle & Jeffery, 2017). There are fewer CTE programs, including CTE leadership programs, offered nationally in institutions of higher education (IHE) and there is a changing environment for many remaining CTE programs that have been merged into loosely coupled departments in Colleges of Education or other Colleges in IHEs based on financial considerations (Fletcher et al., 2015; Zirkle & Cotton, 2001; Zirkle & Martin, 2012).

There are no specific personnel preparation CTE Administrator Standards nationally (Zirkle & Jeffery, 2017). The Association for career and technical education (ACTE) established the ACTE Quality CTE Program of Study Framework (Imperatore & Hyslop, 2018) and other documents that support CTE leadership, but the field does not have a set of specific CTE administrator standards that guide CTE leadership preparation like other professional organizations (e.g., The Council for Exceptional Children (2015) Advanced Preparation Standards and Advanced Specialty Set: Special Education Administration Specialist). These standards form the framework of professional training for special education administrators and are the basis for teacher preparation programs in IHEs that are accredited using the knowledge and skills set established by specialized professional associations (SPA). Zirkle and Jeffery (2017) indicate the need for CTE standards is essential and that these can guide the accreditation and educator training process encompassed in a SPA for the field.

The Pennsylvania Department of Education (PDE) Framework for Leadership for CTE directors includes specific domains, measures of proficiency, and types of evidence (PDE, 2014). The Pennsylvania Department of Education (PDE) established the Educator Effectiveness System with rules promulgated under PA Code: Title 22 Education Chapter 19. The Framework for Leadership includes effectiveness ratings for the principal and also includes those for the director of vocational education (CTE director) as outlined in Act 82 (2012). The Pennsylvania Department of Education (2014, July) Framework for Leadership: Types of Evidence—CTC Directors includes four specific domains, Domain (1) Strategic/Cultural Leadership; Domain (2) Systems Leadership; Domain (3) Leadership for Learning; and Domain (4) Professional and Community Leadership, which support CTE Leadership training. This framework, while specific to Pennsylvania, articulates leadership domains that alignment with the NPBEA (2018) National Educational Leadership Preparation (NELP) Program Standards—District Level: (1) Mission, Vision, and Improvement; (2) Ethics and Professional Norms; (3) Equity, Inclusiveness, and Cultural Responsiveness; (4) Learning and Instruction; (5) Community and External Leadership; (6) Operations and Management; (7) Policy, Governance, and Advocacy; and (8) Internship. Neither framework specifically state specialized knowledge and skill core competencies needed by the CTE administrator nor do they delineate a specified skill set or pathway for administrator training as would a SPA.

CTE administrator preparation conducted in Pennsylvania is typically administered through the three (3) PDE supported centers for CTE personnel preparation and development. These centers serve the Commonwealth in three regions located at Temple University in the east, Penn State University in the central, and Indiana University of Pennsylvania (IUP) in the west. These centers provide CTE professionals with teacher pre-service and in-service training, professional development opportunities, support research in CTE and occupational competency testing, and serve as a conduit for CTE certificates through the PDE Bureau of Career and Technical Education (BCTE) and Division of Certification Services (DCS). All three PPD centers have CTE Leadership programs but do not share a common curriculum and/or a set of professional preparation standards beyond those established by PDE for the Vocational Director certificate. There is a need for common administrator training specific to the CTE director in PA.

The literature indicates that states have shifted away from specific CTE leadership credentials by reducing, collapsing, or eliminating CTE licensing requirements (Fletcher et al., 2015). This is a significant issue given the importance of CTE in workforce development. This is equally important given the association with administrative leadership tied to positive student achievement (Leithwood et al., 2004; Nettles & Herrington, 2007). Pennsylvania is working to address this issue with support from the PDE Bureau of Career and Technical Education (BCTE), the Pennsylvania Association for Career and Technical Administrators (PACTA), and the three PPDC Centers.

This research advances the field of CTE administration and leadership training by identifying a set of knowledge and skill core competencies (KSCC) needed to be successful as a CTE director in Pennsylvania. The research also contributes to identifying standards for CTE leadership using evidence-base methods (triangulated research methodology) framed in the context of a SPA for CTE administrative training purposes. This study identified, operationalized, and codified specific KSCC within administrative focus areas extending the National Policy Board for Educational Administration [NELP] (2018) standards. In addition, a set of KSCC was identified by utilizing research processes involving a national jury panel of subject matter experts, Pennsylvania CTE administrators, a focus group of CTE experts in Pennsylvania, and Delphi CTE experts in PA. Identification of essential KSCC for success in the role of the CTE Director in Pennsylvania was derived from using multiple-methods research focused on consensus building. The findings address the critical concerns identified by Zirkle and Jeffery (2017) and create a framework that can be used to establish a national common core skill set (KSCC). The objectives of this research were to:

identify the essential knowledge and skill core competencies (KSCC) needed for CTE administrator success in Pennsylvania; and

reach consensus on specific codified knowledge and skill items deemed important by specific focus areas for CTE leadership training in the Commonwealth of Pennsylvania.

Method

Group Processes

Multiple research methods: survey, focus group, and Delphi procedures were used to establish a set of essential KSCC needed for CTE administration in Pennsylvania. Triangulation research methodology provide a richer, comprehensive means of establishing knowledge and skill core competencies drawn from the population of interest (PA CTE directors). This research approach was selected to “provide a foundation for more intensive evaluation” to establish an intended outcome (Egan et al., 1995, p. 459). The study used evidence-based methods across three phases: I survey, II focus group, and III Delphi procedures. Phase III provided a specific focus on consensus-building through Delphi procedures with three-rounds to reach final consensus on the KSCC as established in nine focus areas (Linstone & Turoff, 2002; Rowe et al., 2015).

Survey methods were used in Phase I with a census sample (Dillman et al., 2009) as the initial part of this three-phase design. The survey was reviewed by a national expert jury panel with 12 subject matter experts (nationally recognized experts in the field of CTE, CAEP accreditation, and National SPA Standards) and pilot tested with 30 CTE professionals (members of a Research One University’s Personnel Professional Development Center for workforce and agricultural education) to establish face, content and construct validity and internal consistency. The study was conducted electronically using Qualtrics (a survey software package) with email and an embedded link to the electronic questionnaire (Rea & Parker, 2014). The Phase I survey included all CTE administrative directors in Pennsylvania (n = 93 individuals across the Commonwealth). Means and standard deviations, cross-referenced with high-middle-low percentile rankings, were used to establish the knowledge and skill areas within each of the nine focus areas rated/ranked by item importance by Phase I survey respondents. Data were analyzed using SPSS Version 25, a statistical software package (IBM 2018, SPSS v25 software). The results of Phase I were used as baseline for the focus group review process in Phase II. Twelve subject matter experts (Distinguished retired PA CTE Directors in PA) were initially identified. Six of these experts were included in the focus group which was designed to clarify and validate Phase I survey results. The focus group (Lee & Nelson, 2015) was used to clarify knowledge and skills competencies identified from the survey related to CTE administration and validate the itemization of these as core competencies for the nine focus areas aligned with the PDE Leadership Framework for CTE Directors. This information served as the basis for the Phase III Delphi procedure (Round I) which was conducted with 45 CTE directors (15 from each of the three PDE CTE Professional Development Regions in PA). These CTE directors were specifically selected as outstanding leaders from the CTE directors in their specified region (east-central-west [see Participants—Phase III: Delphi Process below]. A modified Delphi technique (Linstone & Turoff, 2002) with a three-round design was used to reach consensus resulting in the KSCC established. Data from the Delphi group was shared over three consecutive rounds to build census through a structured process of electronic communication where participants could communicate perceptions of content, edit, add and/or delete previous items, and provide on-going suggestions for consideration as part of the process (Hsu & Sandford, 2007; Rowe et al., 2015). The Delphi process has been used in CTE research investigating educator competencies (Clark et al., 2010; Manley & Zinser, 2012). Delphi group consensus was measured using percentages as the indicator of agreement among specified choice options for each Delphi round to establish the KSCC (Rowe et al., 2015).

Participants

The authors used purposeful methods of sample selection for each phase of this study. All participants were current or former CTE administrators in Pennsylvania with first-hand knowledge of the unique skills and competencies needed to be successful as a CTE director in CTE settings across the Commonwealth.

Phase I: Survey—All current CTE Executive Directors, CTE Directors, or CTE Administrators responsible for the operation of career and technical education in Pennsylvania (i.e., CTE offered by Career and Technology Centers [CTC regional programs supported by local educational agency (LEAs)]; Comprehensive CTC [centers offering secondary 9 to 12 academics and CTE], High Schools (LEAs) offering eight or more CTE programs [Program of Study—CIP]) receiving Perkins funding were included in this study. Pennsylvania has 92 CTE vocational directors serving as executive directors (or related roles) in 78 CTCs and 14 Comprehensive High Schools across the Commonwealth. There are 26 CTE sites in the eastern region, 34 CTE sites in the central region, and 32 in the western region. A census sample with a total of 93 CTE directors were included in the Phase I Survey.

Phase II: Focus Group—The researchers identified a total of 12 retired CTE administrators recognized as subject matter experts in Pennsylvania for focus group participation. The researchers reached out to six (6) subject matter experts (representing a balanced group by gender, region, type of CTE setting, years of experience) who were selected for the Phase II focus group. The retired CTE administrators selected were professionally active; three were PACTA Pennsylvania Inspired Leadership (PIL) instructors, one was a consultant in CTE, and one was an administrator in higher education. Their involvement in CTE provided added value because these CTE professionals were well aware of current issues and trends in CTE leadership based on their professional experiences.

Phase III: Delphi Process—The directors of the three PDE supported centers for personnel preparation and development located at Temple University, Penn State University, and IUP identified 20 CTE Directors from each of their regions to be considered for selection in the Phase III Delphi process. The Delphi process included a balanced selection of 15 of the recommended CTE directors from each region with a total of 45 CTE directors invited to participate. These members were part of the Phase I Survey conducted with all current CTE Directors in Pennsylvania.

Summary: The Phase I: Survey included all CTE administrators serving as the COE of the 92 CTE schools in the Commonwealth. The 12 former CTE directors identified in the Phase II: Focus group were retired and not represented in the other study phases. The 45 CTE Directors identified for the Phase III: Delphi process were selected from recommendations provided by the PPDC Directors. These members were a sub-set of the Phase I: Survey (<50%). These specific individuals were noted as having expertise needed in the Delphi consensus building process.

Response Rate

Phase I: Survey—The survey was conducted electronically over an eight-week period during the spring-summer 2018. There were four follow-up reminders sent to non-respondents every two-weeks. There were 93 surveys sent that resulted in 49 usable (fully completed) surveys for a 53% return rate.

Phase II: Focus Group—Six former CTE directors were invited to a 2-day retreat focus group conducted in November 2018. One member was not able to attend and notified the primary research investigator the day of the event. Five members participated (83% participation rate).

Phase III: Delphi Process—45 CTE directors in Pennsylvania were invited to participate in the Delphi process as part of Round 1 conducted in January-February of 2018. There were 28 CTE directors who completed the initial Delphi Round 1 for a 62% return rate. These 28 CTE directors were invited to participate in the second Delphi Round 2 conducted in February-March of 2018. There were 24 CTE directors that completed Round 2 for an 86% return rate. All 24 CTE directors were invited to participate in the final Delphi Round 3 conducted in March-April of 2018. All 24 completed the final Round 3 for a 100% return rate.

Procedures

Data was collected in three phases: (a) Phase I survey methods to establish levels of importance of specific knowledge and skill competencies in each of nine focus areas, (b) Phase II focus group process to seek input from subject matter experts to clarify, define, and validate the findings from the Phase I survey regarding the specific knowledge and skill areas per focus area, and (c) Phase III Delphi procedures to build consensus and establish a final set of KSCC by focus area to ensure effective training efforts for CTE administrators in CTE Leadership programs across Pennsylvania. The process followed an evidence-based approach to build consensus to establish a KSCC for CTE administrative training using a multi-method design (Egan et al., 1995; Hsu & Sandford, 2007; Rowe et al., 2015). The process used in each of the three phases is further described.

Phase I: Survey methods

The Professional Personnel Development Center at Penn State University has used a framework for CTE administrator preparation that includes nine focus areas: (1) Management of Business and Finance; (2) Administration and Supervision of Curriculum; (3) Management of Instruction; (4) Personnel Management; (5) Management of Student Services; (6) Physical Facilities Management; (7) Management of Communication; (8) Professional Development and Leadership; and (9) Administration and Accountability with a total of 96 core competencies. These are the framework skills of the Career and Technical Education Administrative Director Internship (WFED 595B) that was the capstone in the Penn State Workforce Education and Development CTE Leadership program. The Administrative Internship content areas and associated knowledge and skill areas align with the broader NPBEA (2018) NELP Standards, PDE Framework for Leadership for CTE Directors (2014), and PDE Pennsylvania Inspired Leadership (PIL) Program Core and Corollary Standards (2014).

The survey in Phase I used this structure as a framework for KSCC. The initial survey was reviewed by the researchers, codified with validation established by seven CTE professionals, and formatted in Qualtrics. This version was sent to a national jury panel of 12 subject matter experts (i.e., CTE leadership faculty, higher education faculty familiar with standards development, and higher education administration responsible for institutional assessment and SPA/CAEP accreditation) to critique and provide input on the CTE leadership focus areas and associated competencies. Based on the review, adjustments were made concerning minor readability and itemization edits. Suggested items were adopted adding items to the Focus Areas in Leadership of Instruction and Management of Communication; these items reflected current/best practice. The survey was pilot tested with 30 CTE PDDC professionals to ensure face, content, internal consistency and test the analytic plan. No changes were made to the instrument based on pilot test feedback.

The online survey was conducted over an eight-week period during the spring-summer 2018. Participants were asked to rate each focus area KSCC using a 4-point Likert scale (1 = not significant; 2 = significant; 3 = very significant; 4 = essential) and rank the importance (1 = low level; 2 = medium level; 3=high level). Phase I survey results are reported in multiple tables for all 9 Focus Areas as part of the Phase I survey research (e.g., Means, SD, Kruskal-Wallis tests, epsilon-squared effect size for differences by CTE PDE Region) for manuscripts in development. The final results of the broader three phase study are reported here to include the summary results of the Phase I Survey by the nine (9) Focus Areas reported in Table 1. These data are presented by each Focus Area, with the number of knowledge and skill area core competencies represented by (n), the range of lowest to highest means (M) based on a four-point Likert-type scale for KSCC items, and the rankings of importance (low-medium-high) by (n) and percentages by focus area. All survey data were shared with Phase II Focus Group members as the basis for the focus group session.

Table 1.

Summary Measures for Phase I Survey by KSCC Focus Area.

Focus area knowledge and skill areas		n	Rating means	Ranking percentages
Focus area knowledge and skill areas		n	Rating means	Low (%)	Middle (%)	High (%)	Tie (%)
1	Management of business and finances	13	2.69–3.55	0–0	8–62	5–38
2	Administration and supervision of curriculum	10	2.88–3.60	0–0	4–40	6–60
3	Management of instruction	20	2.53–3.47	0–0	10–50	10–50
4	Personnel management	17	2.65–3.57	0–0	5–29	11–65	1–6
5	Management of student services	8	2.29–3.49	1–12.5	4–50	3–37.5
6	Management of physical facilities	8	2.58–3.41	0–0	4–50	4–50
7	Management of communication	9	2.79–3.38	0–0	6–66.6	3–33.3
8	Professional development and leadership	7	2.92–3.65	0–0	5–71.4	2–28.6
9	Administration and accountability	7	2.69–3.45	0–0	3–42.8	4–57.1

Note. n equals the number of knowledge and skill core competencies in each Focus Area. Rating Means represent a 4-point Likert Scale where 1 = not significant, 4 = essential. Ranking Percentages represent frequency and percentage.

Phase II focus group

The focus group sought information from Pennsylvania CTE subject matter experts though guided prompts, facilitated discussion, and collaborative consensus. The focus group’s role was to assess, edit/add, clarify and validate the Phase I survey results. The focus group encompassed a 2-day retreat held in early November 2018. Focus group members were invited to participate by email and signed an initial consent as part of the planning effort. Day one of the retreat focus group members were asked to sign a consent waiver giving permission for audio taping of the session and researchers’ rights for publication. Informational packets were provided to focus group members including all results from the Phase I survey. The focus group was asked to provide insight into the nine focus areas and provide clarification/priority of the KSCC rated/ranked in the survey. Specifically, they were asked to provide insight based on the survey results to clarify and further define the KSCC items within each of these focus areas.

The focus group process involved the three study researchers (one served as the facilitator, the other two provided support and clarifying information as needed) and two graduate assistants (GAs) who assisted with technology and the mechanics during the 2-day session. This group also reviewed the focus group information with summary data and was instrumental in item analysis reached through consensus on focus group input. The researchers used a transcription service (rev.com) as part of the focus group process with input validation through review/consensus from (a) the researchers and (b) the GAs independent analysis. The results of the focus group included three alternatives to the original item (FA1-4-9), 16 updates to the original item (FA 1-2-3-4-5), five delete original item suggested (FA2-3-4-5), four move original item to FA 9 (FA 2-4), and six new items added to the KSCC (FA 6-7-9). Only Focus Area 8. Professional Development and Leadership had no suggested edit, additions and/or revisions.

Phase III Delphi process

The researchers reached out to the selected 45 CTE directors identified as expert CTE leaders in PA prior to conducting Phase III. Forty-two agreed to participate with three CTE directors not responding to the email invite or a follow-up phone message. All 45 CTE directors were invited to participate in the Delphi process. The Delphi rounds were conducted electronically with email and a link to Qualtrics with the data analysis using SPSS.

Delphi Round 1

For Round 1, all 45 CTE experts were invited to participate and those who agreed were provided with the results of the Phase I Survey data and Phase II Focus Group insight (e.g., editing/clarification/additions/deletes). We asked these CTE experts to review the summary findings including: (a) KSCC Mean ratings based on a 4.0 scale where 4 = essential and 3 = very significant. . .; (b) KSCC ranks of Level of Importance measured as High, Medium, Low; (c) the specified ranking (e.g., first, second, third. . .) of the KSCCs within each Focus Area; (d) the specific total ranking for all KSCC items (e.g., number 1-2-3. . .out of 99 for each item in all Focus Areas); and (e) review the original item and the focus group feedback for each specific KSCC item. Experts were asked to indicate if you would: (a) Keep the original item; (b) Edit the original item; (c) Delete the original item; or (d) Accept the focus group edit (if any was noted) for each specific focus area KSCC item. For items that were recommended to be deleted, added, or moved by the Focus Group, experts could (a) Accept the deleted item and/or added item or specified moved item or (b) Reject the deleted item and/or added item or specified moved item. Experts were also asked to provide (a) any rationale for their choice selection and (b) to provide an Edit suggestion for the specific KSCC item where they selected Edit as the choice [as noted in choice selection b above]. These feedback elements were provided in a specified text box question in Qualtrics as part of the Delphi feedback process.

CTE experts were also asked to identify “the five-items that you think are most essential/critical for the CTE Director” for each specific Focus Area KSCC list. They were instructed to (a) list these in 1-5 order of importance with 1 being the most essential/critical and 5 being the least essential/critical of those identified as most important of your top-five choices and (b) to add any CTE Leadership Knowledge and Skill items that you feel should be added to the specific Focus Area KSCC list. This Delphi procedure was similarly used for Rounds 2 and 3.

The Delphi Round 1 results (n = 28 experts) were reported as descriptive data (i.e., frequency and percentages) for each focus area. The KSCC item choice with the highest percentage, along with a synopsis of the edits suggested and the results of the top-five rankings of importance from Delphi Round 1 formed the basis for the Round 2 Delphi process.

Delphi Round 2

For Round 2, the 28 CTE experts who completed Round 1 were invited to participate in Round 2. Those who agreed were provided with the results of the Round 1 Delphi process. The Round 2 Delphi process further clarified and allowed for feedback and edits from the CTE experts working toward a more refined KSCC list for CTE leadership training purposes. Experts reviewed the summary findings of Delphi Round 1 (e.g., frequencies, percentages) for each KSCC item and their ranking (i.e., total point distributions, percentage of the total points, rank order) within each specific Focus Area. This included the steps taken as outlined in Round 1 with the following exceptions. . . Experts were asked to indicate if you would: (b) Accept the Delphi Round 1 edit or added item (if a suggested edit/addition was provided), and similar to Round 1, experts were asked to identify “the five-items that you think are most essential/critical for the CTE Director” using the results of the Delphi Round 1 Focus Area KSCC ranking results as a guide. The Delphi Round 2 results (n = 24 experts) were reported as descriptive data (e.g., frequency, percentages) for each focus area. The KSCC item choice with the highest percentage, along with a synopsis of the edits/addition suggested, and the results of the top-five rank of importance data from Delphi Round 2 formed the basis for the final Round 3 Delphi process.

Delphi Round 3

For Round 3, the 24 CTE experts who completed Round 2 were invited to participate in the final Round 3. We emphasized the critical importance of this final phase of the study in the Delphi Round 3 protocol. Those who agreed were provided with the results of the Round 2 Delphi process. The Round 3 Delphi process sought perspective from experts who completed the Delphi Round 2 process (n = 24) in determining the final KSCC. It was noted that the Delphi Round 3 process would establish a final KSCC listing with rankings of importance for the nine focus areas needed for CTE leadership training in Pennsylvania. Experts were asked to review results from the Delphi Round 2. This included the KSCC items and their suggested Round 2 edits/additions and the top-five ranking results as a guide for Round 3. Specifically, experts were asked if they would: (a) Keep the original, focus group, or Round 1 item (established from Delphi choice selection procedure—see Table 2); (b) Accept the Delphi Round 2 edit or addition (if any listed); or (c) Delete item entirely (eliminate the specific item from the Focus Area KSCC list). Delphi round 3 did not include an Edit item choice. Experts were also asked to: (a) review the summary of findings for rankings with point distributions, percentage of total points, rankings, and the identified Top 5 Ranked items in each specific Focus Area and to (b) identify the “five-items that you think are most essential/critical for the CTE Director” from the results of the Delphi Round 2 Focus Area KSCC as listed. The Delphi Round 3 results (n = 24 experts) were analyzed with KSCC item choice with the highest percentage and the resulting rankings of importance, with the top-five KSCC ranks for each focus area from Delphi Round 3. The final study results are presented in Table 4 as a final defined and operationalized KSCC list.

Table 2.

Example of Choice Options with Consensus Selections across Round 1-2-3 Phase III Delphi Process for Focus Area 1. Management of Business and Finance.

	Delphi RI	Delphi R2	Delphi R3
KSCC	Keep original; edit original; delete original; accept FG edit (if any)	Keep original; edit original; delete original; accept Delphi R1 edit (if any)	Keep item, accept Delphi R2 edit (if any), delete item entirely
CTE1S1	Original (96.4%)	Original or Delphi R1 edit (54.2%)	Delphi R1 item or Delphi R2 edit (87.5%)
CTE1S2	Original (89.3%)	Original or Delphi R1 edit (66.7%)	Delphi R1 item (95.8%)
CTE1K1	Original (71.4%)	Original or Delphi R1 edit (87.5%)	Delphi R1 item or Delphi R2 edit (58.3%)
CTE1K2	Original (85.7%)	Original or Delphi R1 edit (75.0%)	Delphi R1 item or Delphi R2 edit (70.8%)
CTE1S3	Original (92.9%)	Original or Delphi R1 edit (50.0%)	Delphi R1 item or Delphi R2 edit (66.7%)
CTE1K3	Original (92.9%)	Original or Delphi R1 edit (70.8%)	Delphi R1 item (100%)
CTE1K4	Original (82.1%)	Original or Delphi R1 edit (66.7%)	Delphi R1 item or Delphi R2 edit (62.5%)
CTE1K5	Original or FG item (64.3%)	FG item or Delphi RI edit (75.0%)	Delphi R1 item or Delphi R2 edit (75.0%)
CTE1K6	Original (85.7%)	Original or Delphi R1 edit (70.8%)	Delphi R1 item or Delphi R2 edit (66.7%)
CTE1K7	Original (92.9%)	Original or Delphi R1 edit (62.5%)	Delphi R1 item or Delphi R2 edit (83.3%)
CTE1K8	FG update (similar) (78.6%)	FG item or Delphi RI edit (100%)	Delphi R1 item (100%)
CTE1S4	FG update (similar) (85.7%)	FG item or Delphi RI edit (66.7%)	Delphi R1 item or Delphi R2 edit (83.3%)
CTE1K9	FG update (similar) (89.3%)	FG item or Delphi RI edit (66.7%)	Delphi R1 item (87.5%)
CTE1K10		Delphi R1 add item (87.5%)	Delphi R1 item or Delphi R2 edit (66.7%)
CTE1K11		Delphi R1 add item (83.3%)	Delphi R1 item (54.2%) or Delphi R2 edit

Note. FG = focus group (Phase II); R1 = round one; R2 = round two; R3 = round three (Phase III). RI edit = the edited item for consideration in R2 choice selection of the Delphi process, similarly with R2 edit for R3 choice selection. Delphi Round 3 represents the final choice for KSCC consensus building in this study.

Results and Discussion

Final Focus Areas Knowledge and Skill Core Competencies (KSCC)

The study was conducted over a one-year period from May, 2018 to April, 2019. The process allowed for multiple-sources of data (Egan et al., 1995) to be used to build a comprehensive, evidence-based framework where experts reached consensus on the KSCC needed for CTE leadership training in Pennsylvania.

Phase I Survey results were used to define the KSCC (Dillman et al., 2009). Focus Areas with higher ratings/ranks included FA2. Administration and Supervision of Curriculum and FA8. Professional Development and Leadership and those with lower ratings/ranks included FA5. Management of Student Services and FA6. Management of Physical Facilities (see Table 1). Cronbach’s alpha ranged from a low of .864 (FA9. Administration and Accountability) to a high of .956 (FA 4. Personnel Management). Phase II Focus Group insight and feedback further clarified the KSCC identified in this study (Lee & Nelson, 2015). There were a total of 34 items addressed across all nine focus areas identified: (a) sixteen included original survey items updated with additional information, elaboration and/or clarification provided; (b) three items that were alternative items proposed for the original item; (c) five items suggested to be deleted; (d) six additional items included (new KSCC); and (e) four items (i.e., FA2, FA4) suggested to be moved from their original focus area to FA9 which was viewed as a more appropriate domain. Phase III Delphi Procedures (i.e., Round 1-2-3) allowed CTE experts to review prior ratings/rankings/choices from Phase I and Phase II with the opportunity to add insight through editing, suggested additions, and decision-making regarding KSCC item choice and rankings of importance (e.g., top-5 prioritized in order of importance). Each progressive Delphi round 1-2-3 facilitated consensus building (Hsu & Sandford, 2007; Rowe et al., 2015). Table 2 provides an example and overview of the choice and consensus process in Delphi Rounds 1-2-3.

Table 2 represents a sample of the Focus Area 1. choice selection/s, data analysis, and accounting of KSCC item consensus for the Phase III: Delphi process. The table reports: (a) Focus Area Item Code (the actual knowledge (K) or skill (S) item are not included); (b) the Delphi round choice option/s; (c) the Delphi round choice with the highest percentage of agreement; and (d) that choice item carried forward as the Delphi round choice selection for R1, R2, and R3 (parentheses indicate the choice item selected by highest percentages). Delphi R3 was the final choice selection for the KSCC list as denoted in this study.

Table 3 summarizes all stages of KSCC itemization development by focus area highlighting the process where choice options and the opportunity to edit or add KSCC items to be considered for subsequent study phases. This process also represented Phase III Delphi group ranking top priorities (i.e., top-five most importance ranks) allowing for “best fit” in establishing consensus for the standards associated with each KSCC focus area and priority areas for CTE leadership training. The table shows the progress of inputs, insight, and suggestions from each stage/phase (e.g., stakeholder feedback across all iterations of this study) regarding consensus building and highlights the evolution of the KSCC and development steps in this study. Table 3 reports each specific Focus Area, the actual phase of development for this study, and the respondent groups’ feedback (noted in italics) with action steps for the specified Focus Area KSCC items by chronology of this research project. All components were essential in establishing the final 106 KSCC for CTE leadership training purposes in Pennsylvania.

Table 3.

Development of KSCC Itemization Development across all Study Phases by Focus Area.

Focus area	Original WFED 595B	National panel feedback	Phase I survey	Phase II focus group feedback	Phase III Delphi round 1	Delphi R1 feedback	Phase III Delphi round 2	Delphi R2 feedback	Phase III Delphi round 3	Delphi R3 feedback	FINAL Delphi III round 3 KSCC
1. Management of business and finances	13		13		13	+2 [add two items suggested]	13	[Approve—add two items]	15		15
2. Administration and supervision of curriculum	10		10	−1 [move one item to FA 9] −1 [delete one item suggested]	9	[Approve—delete one item] +1 [add one item suggested]	8	[Approve—add one item]	9		9
3. Leadership of instruction	18	+2 [add two items]	20	−2 [delete two items suggested]	20	[Approve—delete two items] +1 [add one item suggested]	18	[Approve—add one item]	19		19
4. Personnel management	17		17	−3 [Move three items to FA 9] −1 [delete one item suggested]	14	[Approve—delete one item] +1 [add one item suggested]	13	[Approve -add one item]	14		14
5. Management of student services	8		8	−1 [delete one item suggested]	8	[Approve—delete one item]	7		7		7
6. Management of physical facilities	8		8	+1 [add one item suggested]	8	[Approve—add one item]	9	+1 [add one item suggested]	9	[Approve add one item]	10
7. Management of communication	8	+1 [add one item]	9	+1 [add one item suggested]	9	[Approve—add one item]	10		10		10
8. Professional development and leadership	7		7		7		7		7		7
9. Administration of accountability	7		7	+4 [add four items moved FAs above] +4 [add four items suggested]	11	[Approve—four moved items to FA9 and add four item]	15	−2 [option to move two items back to original focus areas]	15	[Not approve—option to move two items back to original focus areas]	15
Total KSCC items	96		99		99		100		105		106

Note. Each stage of this study in represented by the number of KSCC items in that specific phase and the respondent groups’ feedback and actions (presented in italics) sequentially for this research project.

Consensus building was guided not only by choice decision-making but by the edits and addition of items suggested. A critical step in the process was the choice to Edit this item for all KSCC items in each Focus Area. This choice was tied directly to having CTE experts in the Phase III Delphi provide an actual edit in the Qualtrics text box provided. In many cases, the choice to Edit was not accompanied with an actual text box Edit recommendation. There were a number of edits recommended for the KSCC Focus Areas, yet fewer actual edits were provided by respondents (e.g., FA4. Personnel Management—Delphi Round 1 Edit this item choice included six of the 14 KSCC items = 43%, while actual text Edits were provided for 2 items = 14%; Delphi Round 2 Edit this item choice included six of the 14 KSCC items = 43%, while actual text Edits were provided for five items = 36%). Typically for those items where respondents chose to Edit the item, only one or two respondents from the total group (3.6-8.2%) selected this choice. There were only a few items where three respondents from the total group (10.7%) selected this choice. Only four (4) items from the entire list of 100+ KSCC items had five or six respondents from the total group (17.9%–21.4%) select the Edit the item choice.

In the final analysis, all Focus Areas had edit suggestions incorporated in the Phase III Delphi consensus building process. The percentages of Focus Area KSCC items with edits included FA1: Management of Business and Finance = 100%, FA2: Administration and Supervision of Curriculum = 67%, FA3: Leadership of Instruction = 79%, FA4: Personnel Management = 50%, FA5: Management of Student Services = 71%, FA6: Physical Facilities Management 20%, FA7: Management of Communication = 20%, FA8: Professional Development and Leadership = 43%, and FA9: Administration and Accountability = 47% of the KSCC items were edited in Round 1 and/or Round 2 and ultimately selected as the KSCC final choice option in Round 3 of the Phase III Delphi process.

There were a total of 106 KSCC represented in the final selection with 27% were Survey Original items; 14% were Focus Group updates; 17% were Delphi Round 1 Edits; and 42% were Delphi Round 2 Edits. FA1: Management of Business and Finance, FA2: Administration and Supervision of Curriculum, FA3: Leadership of Instruction, and FA 5: Management of Student Services had higher numbers of final KSCC items selected from Delphi Round 1 and Round 2 Edits as part of KSCC consensus building process.

Table 4 includes the final KSCC list (N = 106) established from this research. The KSCC are presented by specific codification denoted for each focus area [code: CTE/Focus Area #/ K = Knowledge or S = Skill, #=progressive number sequence of K or S within the Focus Area. As an example, CTE3K7 represents CTE Focus Area 3. Leadership of Instruction, Knowledge item #7 in the final KSCC list for CTE Leadership training in PA. In addition to the final KSCC, the ranking of importance within each focus area (calculated by overall top-5 ratings total points across the number of KSCC items in each specific focus area) is presented.

Table 4.

Operational Definitions of Essential Knowledge and Skill Core Competencies for Career and Technical Education Leadership Training in Pennsylvania.

Item number	KSCC code	Focus area 1: management of business and finance	Rank of importance
1	CTE1S1	Work with the business manager to prepare and administer an annual budget for a career and technical education center understanding that the director is responsible for what is in the budget.	1 of 15
2	CTE1S2	Work with the business manager to prepare a capital improvement budget or review an existing capital improvement budget/plan for improving a career and technical education facility.	5 of 15
3	CTE1K1	Understand how to implement/maintain procedures for purchasing, receiving, dispensing, and accounting for career and technical education supplies, equipment, and services.	4 of 15
4	CTE1K2	Understand how to provide managerial oversight for an approved accounting system that provides control, analysis, and a successful audit for a career and technical center.	9 of 15
5	CTE1S3	Analyze the cost of operating career and technical education programs and identify annual trends based on previous budgets comparisons to the current year budget (outcome-focused).	2 of 15
6	CTE1K3	Understand the different type of school accounts in use at a career and technical center.	13 of 15 (tie)
7	CTE1K4	Collaborate with business manager to understand how the career and technical program subsidy from the Pennsylvania Department of Education is calculated and the source of data used in the formula.	12 of 15
8	CTE1K5	Understand grant writing and how to prepare proposals for federal, state, and local (public and private) grants and successfully seek alternative sources of funding.	3 of 15
9	CTE1K6	Understand how to collaboratively work with business manager to interpret and present fiscal and audit reports.	11 of 15
10	CTE1K7	Understand the major function of the business manager and the focus on fiscal responsibilities as it relates to the CTE program and sending districts.	10 of 15
Item number	KSCC code	Focus area 1: management of business and finance	Rank of importance
11	CTE1K8	Understand the different revenue streams and financial aid resources (i.e., federal, state, local) for career and technical centers for secondary, post-secondary financial aid, and adult education programs (e.g., Title IV).	7 of 15
12	CTE1S4	Prepare and organize for federal, state, local, and programmatic audits and interpret corrective actions (e.g., corrective action plans) when appropriate.	8 of 15
13	CTE1K9	Understand the relationship of government regulations (i.e., Act 1, economic indexes, market value indexes) to the career and technical centers.	13 of 15 (tie)
14	CTE1K10	Understand the impact of salary, benefits, and discretionary spending on the annual budget.	6 of 15
15	CTE1K11	Understand the “request for proposal” (RFP) and “bid” process involved in the management of business and finances in a career and technical education center.	15 of 15
Item number	KSCC code	Focus area 2: administration and supervision of curriculum	Rank of importance
1	CTE2S1	Cooperate with regional business and industry and occupational and local advisory committees to obtain support and approval of the technical and academic programs offered at the career and technical center.	4 of 9
2	CTE2S2	Obtain input from the occupational advisory committees in establishing or advising a career and technical education program to include industry regulations and standards.	3 of 9
3	CTE2K1	Understand how to comply with state and federal laws in career and technical education and special education pertaining to the operation of the career and technical center.	1 of 9
4	CTE2K2	Develop an understanding of the Pennsylvania Department of Education (PDE) program approval process for career and technical education programs.	7 of 9
5	CTE2S3	Understand where to find data (i.e., local, state, national) and interpret labor market analysis to project and determine program needs for career and technical education.	8 of 9
Item number	KSCC code	Focus area 2: administration and supervision of curriculum	Rank of importance
6	CTE2S4	Implement a data-driven systematic evaluation of the career and technical education curricula to facilitate administrative decisions (e.g., to include industry based technology needs).	5 of 9
7	CTE2S5	Demonstrate, interpret, and apply Chapter 339 Vocational Education of the Pennsylvania School Code.	2 of 9
8	CTE2K3	Understand and implement industry credentials and industry standards into a CTE program.	9 of 9
9	CTE2K4	Understand how the teacher evaluation process, including the development of student learning objectives (SLOs) can aid in achieving student success in career and technical education.	6 of 9
Item number	KSCC code	Focus area 3: leadership of instruction	Rank of importance
1	CTE3S1	Provide appropriate career and technical education instructional materials and resources based on student needs and industry standards, within the limitations of the budget.	6 of 19
2	CTE3K1	Understand the process of earning credentials in a career and technical education program that is aligned with industry standards.	14 of 19 (tie)
3	CTE3S2	Facilitate and lead faculty and staff to improve the instructional process.	1 of 19
4	CTE3S3	Incorporate instructional coaching in the process of conferencing with teachers to provide feedback on strategies and practices to enhance student learning.	16 of 19 (tie)
5	CTE3S4	Work with faculty members on understanding the legality of Individualized Education Plans (IEP) and implementing those plans for career and technical education students with disabilities.	2 of 19
6	CTE3S5	Facilitate/direct curriculum planning, development, and teacher reflection which aligns to classroom and instructional best practice.	5 of 19
7	CTE3S6	Facilitate/direct teacher effectiveness and instructional improvement for individual and group instruction.	10 of 19 (tie)
8	CTE3S7	Assist teachers in developing classroom management skills.	10 of 19 (tie)
Item number	KSCC code	Focus area 3: leadership of instruction	Rank of importance
9	CTE3S8	Assist teachers in developing student management skills emphasizing safety and safe practices.	4 of 19
10	CTE3S9	Assist teachers in the integration of related and correlated academics into career and technical education.	8 of 19 (tie)
11	CTE3S10	Assist teachers in acquiring knowledge of local, regional, and national industry employment trends to support life-long learning and college and career preparation and planning.	13 of 19
12	CTE3S11	Assist teachers to develop skills in planning and implementing an effective career and technical center community relations program.	14 of 19 (tie)
13	CTE3S12	Assist staff in understanding the importance of CTSOs in rounding students’ educational opportunities and experiences.	16 of 19 (tie)
14	CTE3S13	Assist in the development and implementation of Student Learning Objectives (SLO).	12 of 19
15	CTE3S14	Facilitate and assist teachers with knowledge about cooperative education programs and other work-based learning programs and provide guidance in helping place students in industry.	16 of 19 (tie)
16	CTE3S15	Evaluate the career and technical education program in terms of test results, instruction, curriculum, and job placement that includes PVASS, NOCTI, industry credentials, attendance and placement and other local assessments.	3 of 19
17	CTE3S16	Understand Chapter 339 and that CTE personnel should serve as an IEP team member for student placement in career and technical education programming.	7 of 19
18	CTE3K2	Develop an understanding of the scheduling requirements for students enrolled in a career and technical education center.	16 of 19 (tie)
19	CTE3K3	Understand the teacher evaluation process in Act 82.	8 of 19 (tie)
Item number	KSCC code	Focus area 4: personnel management	Rank of importance
1	CTE4S1	Comply with federal, state, and local laws, rules, regulations, and negotiated collective bargaining agreements pertaining to the employment of personnel at a career and technical education center.	1 of 14
Item number	KSCC code	Focus area 4: personnel management	Rank of importance
2	CTE4S2	Describe the role of collective bargaining agreements in education and CTE.	8 of 14 (tie)
3	CTE4S3	Prepare job descriptions for relevant personnel at career and technical centers.	8 of 14 (tie)
4	CTE4K1	Understand how to recruit and select qualified personnel for employment in career and technical education centers.	4 of 14
5	CTE4S4	Determine professional development needs of all staff members and implement a program of school improvement that addresses those needs.	3 of 14
6	CTE4K2	Understand how to develop, review, and update an employee handbook that address necessary functional and legal information.	11 of 14
7	CTE4K3	Understand how to maintain a personnel salary and benefit structure in a career and technical center.	8 of 14 (tie)
8	CTE4S5	Supervise and evaluate the performance of administrative, teaching and non-teaching, and support staff, to include improvement plans and termination where appropriate.	2 of 14
9	CTE4K4	Understand how to maintain personnel employment records including required personnel trainings to ensure compliance.	13 of 14
10	CTE4K5	Understand confidentiality of HIPPA, FERPA, I-9 Clearance Documents and State Police Policy with right to privacy requirements of medical and personnel records.	6 of 14 (tie)
11	CTE4K6	Understand workers compensation and explain how workers compensation insurance applies to the operation of career and technical centers.	14 of 14
12	CTE4K7	Understand the role of superintendent of record and member district superintendents in the operation of a career and technical center.	5 of 14
13	CTE4K8	Demonstrate an understanding of ACT 82 Educator Effectiveness related to principal, teacher, and non-teaching professionals effectiveness.	12 of 14
14	CTE4S6	Develop skills related to conflict resolution, including issues in gender equity, diversity, and interpersonal relationships that impact personnel management.	6 of 14 (tie)
Item number	KSCC code	Focus area 5: management of student services	Rank of importance
1	CTE5S1	Coordinate and/or assist appropriate personnel with recruitment and admissions of career and technical education students.	1 of 7
2	CTE5S2	Ensure that an effective guidance and counseling system is provided/maintained to support career and technical education student needs.	4 of 7
3	CTE5S3	Maintain an effective student information system which interfaces effectively with the Pennsylvania Information Management System (PIMS)	5 of 7
4	CTE5S4	Supervise a system of student placement and follow-up in a career and technical center.	6 of 7
5	CTE5S5	Promote student-based activities within the school such as career and technical student organizations (CTSOs) and student councils.	7 of 7
6	CTE5S6	Maintain a system of student behavior expectations and disciplinary regulations to include bullying, harassment, discrimination, and potential bias.	2 of 7
7	CTE5S7	Evaluate the total student services program at the career and technical center.	3 of 7
Item number	KSCC code	Focus area 6: physical facilities management	Rank of importance
1	CTE6S1	Manage the facility, technology, and equipment needs for career and technical education programs.	2 of 10
2	CTE6K1	Understand the basic operations of building infrastructure including electrical, data transfer, voice, climate control, plumbing, and all other operating systems.	4 of 10
3	CTE6S2	Provide for the security of physical equipment and IT cyber security in a career and technical center.	6 of 10 (tie)
4	CTE6S3	Provide for the security of personnel and students (PA School Safety Plan) and crisis management strategies in a career and technical center.	1 of 10
5	CTE6S4	Maintain career and technical center plant and equipment in a safe, clean, and operable condition.	3 of 10
6	CTE6S5	Maintain a system for receiving, storing, and dispensing materials, equipment, and supplies in a career and technical center.	6 of 10 (tie)
Item number	KSCC code	Focus area 6: physical facilities management	Rank of importance
7	CTE6S6	Maintain career and technical center facilities and equipment in compliance with all codes, laws, rules, and regulations.	4 of 10
8	CTE6K2	Understand how to apply policy regarding use of facilities and technology in a career and technical center.	8 of 10 (tie)
9	CTE6K3	Understand how to supervise food service facilities for feeding students in a career and technical center (i.e., state/federal regulations and requirements).	10 of 10
10	CTE6K4	Understand the duties of custodial, maintenance, groundskeeper, and supervisory staff in facilities operation.	8 of 10 (tie)
Item number	KSCC code	Focus area 7: management of communication	Rank of importance
1	CTE7S1	Promote career and technical education within the community.	6 of 10
2	CTE7S2	Speak in support of career and technical education to external individuals and groups.	4 of 10
3	CTE7S3	Present logical, factual, and documented support for recommendations and proposals pertaining to the career and technical education center.	1 of 10
4	CTE7S4	Prepare status reports on career and technical education for external agencies as required.	9 of 10
5	CTE7S5	Articulate career and technical education needs and program explanations with representatives of labor and other community groups.	5 of 10
6	CTE7S6	Implement policy for internal/external communication for career and technical education.	7 of 10
7	CTE7S7	Provide for external communication strategies to include marketing and promotional activities of CTE for career and technical education.	3 of 10
8	CTE7K1	Develop an understanding of social media and a web or internet presence for a career and technical center.	10 of 10
9	CTE7S8	Work with members of the professional advisory committee and local advisory committee on items of importance to the career and technical school.	2 of 10
10	CTE7K2	Understand the role of members and the function of leadership teams as an integral part of organizational success.	8 of 10
Item number	KSCC code	Focus area 8: professional development and leadership	Rank of importance
1	CTE8S1	Maintain ethical standards prescribed by the education profession.	1 of 7
2	CTE8S2	Apply current educational leadership theory and practice in CTE settings.	2 of 7
3	CTE8S3	Participate in professional career and technical organizations including the Pennsylvania Association of Career and Technical Administrators.	3 of 7
4	CTE8K1	Read and become knowledgeable of current research and professional literature in career and technical education.	4 of 7
5	CTE8S4	Develop and implement a professional development plan to improve leadership abilities.	5 of 7
6	CTE8S5	Participate in continuing education activities.	6 of 7 (tie)
7	CTE8S6	Coordinate career and technical center needs with the Professional Personnel Development Centers.	6 of 7 (tie)
Item number	KSCC code	Focus area 9: administration and accountability	Rank of importance
1	CTE9K1	Understand and interpret data from NOCTI exams and End of Program Assessment (EPA) for curricular and instructional improvement.	2 of 15
2	CTE9K2	Understand Statewide Mandatory Assessment and interpret the achievement of career and technical students on these assessments.	12 of 15 (tie)
3	CTE9K3	Develop an understanding of the Individuals with Disabilities Act and explain how special education law applies to students in career and technical education.	3 of 15
4	CTE9S1	Understand the school accountability system for Pennsylvania that is based on the Every Student Succeeds Act (2015) as related to career and technical education.	10 of 15 (tie)
5	CTE9S2	Explain the purpose of the articles of agreement of the career and technical center for various audiences.	5 of 15
6	CTE9S3	Be able to utilize an analysis of student achievement data, including End of Program Assessment (EPA), in academic and career and technical programs to make improvements in curriculum, instruction, and/or professional development.	1 of 15
Item number	KSCC code	Focus area 9: administration and accountability	Rank of importance
7	CTE9S4	Explain how the PA Future Ready Index are utilized for instructional planning and decision making.	9 of 15
8	CTE9S5	Review existing and develop new career and technical center policies and procedures to reflect practices and circumstances to meet federal, state, and local requirements.	10 of 15 (tie)
9	CTE9K4	Understand how to develop, implement, review career and technical education personnel policies established by the joint operating committee.	6 of 15
10	CTE9K5	Develop an understanding of new policy adoption by the joint operating committee of the career and technical center.	14 of 15 (tie)
11	CTE9K6	Understand how to delegate responsibility and empower subordinates to succeed or fail.	7 of 15
12	CTE9K7	Understand the importance of business and industry engagement in the administration of career and technical education.	8 of 15
13	CTE9K8	Understand the importance of adult education and post-secondary education engagement in the administration of career and technical education and the process of developing adult education programming.	12 of 15 (tie)
14	CTE9K9	Understand the importance of grades 5-8 Career Pathway (College and Career Readiness) exploration and engagement in the administration of career and technical education.	14 of 15 (tie)
15	CTE9K10	Understand the importance of Government-Education-Business and Industry engagement in the administration of career and technical education (i.e., WIOA, VR, Work One, CBO, Chamber of Commerce, Community Foundations).	4 of 15

Note. The final Knowledge and Skill Core Competencies (KSCC) were adopted as a CTE Leadership specialty skill set used in CTE leadership training across the Commonwealth of Pennsylvania (copyright © 2019 Harvey, Fleck & Threeton).

Limitations

There are three specific limitations associated with this particular study: (a) a lack of a specific set of CTE administration standards to guide the process; (b) online research methods; and (c) overall response rate. First, one of the driving forces in establishing this research was the lack of CTE administration standards and no national SPA (Zirkle & Jeffery, 2017). There is no differentiation of the unique role and responsibilities of the CTE administrator compared to others in educational leadership positions (e.g., NPBEA, 2018). This study sought to establish a framework to define essential KSCC needed for CTE administrators in Pennsylvania. This framework allowed us to define a process using multi-method research (Egan et al., 1995) to establish a KSCC for CTE administration training purposes in the Commonwealth.

Second, the use of online survey methods was another limitation. While Dillman et al. (2009) promote internet research methods, the authors note limitations with online survey research. Our study was challenged by limitations in email distribution, SPAM, clutter, and/or junk mail delivery, email overload, and potential apathy (Denscombe, 2009). To what extend these factors played a role is not fully known. There were follow-up emails from the principal investigator (PI) seeking input from participants as to the email delivery through Qualtrics for study Phases I and III (all rounds). Participants with email delivery issues were directed to look for the Qualtrics email in their Spam, Junk, or Clutter folder/s. Many were recovered and in three cases new emails were sent through Qualtrics. We worked to address the online survey delivery limitations.

Third, potential bias is noted based on the research design which used a consensus building model (Rowe et al., 2015) as part of the Phase III: Delphi process. The selection of 45 CTE directors in the Commonwealth (15 from each region) was purposeful. These CTE Directors were a sub-set of the Phase I: Survey administered to all CTE administrators with the intend to include subject matter experts from the population that would provide additional insight and input in the Delphi three rounds to establish a comprehensive KSCC list for Pennsylvania.

Finally, response rate is always a concern with electronic surveys (Dillman et al, 2009). Factors mentioned above regarding non-response may have directly applied here. Our response rates included Phase I: Survey = 53%, Phase III: Delphi process Round 1 = 62%, Round 2 = 86%, Round 3 = 100%. These response rates are typically considered good for electronic surveys (Sheehan, 2001). Given the nature of this research and the concerns presented in the literature associated with CTE leadership and administrative credentials (Clark & Cole, 2015; Gregson & Allen, 2005; Zirkle & Cotton, 2001; Zirkle & Jeffery, 2017), participation from the field was important. We purposefully designed this research project to accommodate our participants. CTE administrator typically have limited down time during the academic year. The Phase I Survey was conducted during the summer months when classes are not in session. Our focus group subject matter experts were polled regarding their availability. We scheduled the Phase II focus group during the time participants were available. The Phase III Delphi process was a commitment of time based on the three-rounds. We designed a one-month turn around cycle with completion of the three rounds before the demanding administrative budget cycle in spring for CTE directors. The Delphi process occurred from January through early April, 2019 minimizing impact on our CTE experts. We offered an incentive for those who completed all three rounds of the Phase III Delphi Process (three $75.00 gift cards, one for each of the three PPDC regions). This may have helped with respondents’ staying power.

Online research methods, noting the limitations above, was appropriate given the complex nature of this research study. A richer sharing of ideas and a more comprehensive picture of the issues associated with CTE administration training was a major goal of this research. We had an adequate response rate to validate our findings (Fan & Yan, 2010).

Implications for Practice

CTE has provided added value in workforce development and occupational skill training (Imperatore & Hyslop, 2017). Today’s CTE continues the tradition in CTE STEM education and career pathways for all by addressing the U.S. economy’s technical skills shortage and by facilitating college and career readiness for high school participants (AIR, 2013; Cushing et al., 2017; Mobley et al, 2017). CTE leadership programs need to ensure that the unique knowledge and skills needed to effectively and efficiently manage and lead CTE programs into the 21^st century are focused on the balance of CTE tradition and the challenges of an expanded role in today’s educational system. The KSCC compliments the NELP (2018) Standards.

This study established an essential set of KSCC needed for the CTE director in Pennsylvania. The emphasis was on CTE leadership and management KSCC to establish standards for CTE administrative training. The evidence-based KSCC delineated here is specific to Pennsylvania and provides a framework for CTE Leadership programs. The KSCC, coupled with the NELP (2018) and PDE CTE (Pennsylvania Department of Education) Framework (2014), establish “critical mass” (essential knowledge and skills needed) for CTE administration (e.g., management and leadership essential skills). Given the lack of national CTE administrative standards and the absence of a CTE SPA (Zirkle & Jeffery, 2017), states are left to define and design their approach to CTE leadership training. Pennsylvania has established an evidence-based KSCC that can provide baseline information regarding the uniqueness and importance of the CTE administrator’s role in public education and compliance with the Perkins V (2018) legislation. In addition, the KSCC emphasize the responsibilities and skills needed to successfully undertake, engage, and succeed in the CTE Director’s role.

Pennsylvania can see impact at the state, regional, and local level with the direct implementation of the KSCC. All stakeholders can potentially see a benefit. The focus areas and KSCC establish an operationalized set of core competencies needed for the CTE Director (e.g, PDE CTE Director certificate) in Pennsylvania. The BCTE can provide guidance related to the KSCC associated with PDE CTE Director certification requirements. The regional PPDC centers at Temple, Penn State, and IUP have adopted the KSCC and can design administrative training using the KSCC framework (these efforts are supported by PDE BCTE and PACTA). Course curricula and CTE administrative internships will embed the KSCC and assess proficiency of CTE Leadership candidates as part of the credentialing process. In addition, CTE administrators at the local level can use the KSCC to guide effective (a) professional development and team building, (b) leadership and management practices, and (c) building business and industry, community and school-based relationships to better support CTE in their setting.

Finally, the KSCC can be used to foster communication and collaboration in the CTE community (e.g., state DOE, IHEs, and CTE Administrative Associations) to build a shared pathway for CTE leadership with national implications. The KSCC established in Pennsylvania and the research methods used to establish these core competencies (i.e., triangulated research methodology/evidence-base) is a model that can move this effort forward. Coordinated efforts to establish a KSCC for CTE leadership training should be guided by sound research methods (e.g., Egan et al, 1995; Hsu & Sandford, 2007).

Recommendations for Research

There is “no mention in the ELCC standards of any specific CTE skills or knowledge needed by any administrator” and in light of this fact “credentialing requirements for CTE administrators can be differentiated from the more generalized standards offered by the ELCC” (Zirkle & Jeffery, 2017, p. 23). This research used multi-methods with results that established three specific recommendations for future research. First, there is continued work to be done in Pennsylvania. We need to implement the KSCC in each of the three IHE PPDC to include (a) full program adoption, (b) curriculum revision and alignment, and (c) instructional/internship application with fidelity. Research needs to follow all phases of implementation with stakeholder evaluation and/or program assessment with faculty, students, PPDC, and PDE involvement where appropriate. Areas of challenge need to be addressed and appropriate support provided to ensure successful implementation, including needs assessments and instructional design as part of the process (Lee & Nelson, 2015). Once the KSCC has been fully implemented, each PPDC should do a full evaluation to make appropriate adjustments. Finally, the researchers of this study need to conduct a three-year follow-up to update, revise and/or modify/adjust the KSCC list for CTE administrative training. This will ensure that the KSCC is validated and adequately updated to meet CTE leadership training needs in Pennsylvania.

Secondly, in an effort to scale-up the KSCC to address national CTE administrator standards, this study should be replicated in other states. The benefit is two-fold: (a) other states involved in replication will be able to develop a KSCC based on the Pennsylvania model to address CTE Leadership core competencies needed for CTE directors/administrators as defined by their Department of Education (DOE) and (b) it can provide validation of the PA KSCC in that common indicators associated within and across states are identified. We are currently replicating this study with three states as a first phase national replication effort. In addition, we are seeking/solidifying commitments from additional states to initiate a phase two replication effort. We hope to expand the process to other states as well (seeking to develop a phase three replication effort).

Finally, we propose developing a national KSCC and recommend the development of a SPA to support CTE leadership training offered nationally. Exploration of opportunities to develop a CTE SPA should begin with some discussion with the Association for Career and Technical Education (ACTE), Advance CTE, and other professional organizations representing CTE. This will strengthen the process of CTE Leadership and validate the multi-faceted role, responsibilities, and skills needed to be successful in professional practice unique to the CTE administrator. More research is needed to compile the states’ replication KSCC findings noted above. Collectively, we can identify a common skill set of KSCC needed for the CTE administrator nationally. This KSCC should account for the differing roles the CTE administrator plays in each state as determined by state regulation/s, rules, and/or DOE requirements (e.g., certification/licensure). This common core of unique skills needed in CTE administration can complement the NELP (2018) Standards.

In conclusion, the purpose of this study was to identify essential knowledge and skill core competencies needed for CTE administrator training in Pennsylvania. We operationalized and defined the KSCC using a multi-method research approach (Egan et al, 1995) with consensus building (Hsu & Sandford, 2007). Results reported here represent consensus on a set of KSCC across all nine focus areas for Pennsylvania’s evidence-based CTE Leadership training. In addition, our recommendations support efforts to address and promote research to identify a national set of KSCC standards to compliment the NELP (2018) Standards and efforts to establish a CTE Leadership SPA.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Michael W. Harvey

Author Biographies

Michael W. Harvey, Ph.D. is a Professor of Education in the Workforce Education and Development Program at Pennsylvania State University, University Park. His research interests include school leadership, secondary transition, and post school outcomes. His recent work includes establishing CTE administrator standards for training and research efforts in CTE for individuals with disabilities.

Joey A. Fleck, Ph.D. is an Assistant Professor and the Coordinator of the Competency Based Teacher Education Program within the Workforce Education and Development Program at Pennsylvania State University. He consults and conducts research in the areas of leadership development and Career and Technical Education teacher education.

Mark D. Threeton, Ph.D. is an Associate Professor of Learning and Performance Systems within the Workforce Education and Development Program at Pennsylvania State University. He consults, teaches and conducts research in the areas of Career and Technical Education (CTE) and Experiential Learning practices which promote the transfer on knowledge, skills and dispositions.

References

American Institutes for Research (AIR). (2013). How career and technical education can help students be college and career ready: A primer. American Institutes for Research.

Bowers

(1979). Women in vocational education administration: A nation wide analysis (ERIC Document Reproduction Service No. ED 181 316). National Center for Research in Vocational Education.

Clark

R. W.

Cole

(2015). A look at leadership: An examination of career and technical administrator preparation in the United States. Career and Technical Education Research, 40(1), 63–80. https://doi.org/10.5328/cter40.1.63

Clark

Farmer

Welch

(2010). An examination of leadership issues facing Pennsylvania career and technical administrators. Career and Technical Education Research, 35(1), 47–62. https://doi.org/10.5328/cter35.104

Council for Exceptional Children. (2015). Advanced preparation standards and advanced specialty set: special education administration specialist. CEC. https://www.cec.sped.org/~/media/Files/Standards/CEC%20Initial%20and%20Advanced%20Specialty%20Sets/Advanced%20Specialty%20Set%20%20Special%20Education%20Administration%20Specialist.pdf

Cushing

Therriault

English

(2017, November). Developing a college and career readiness workforce: An analysis of ESSA, Perkins, and WIOA. American Institutes for Research.

Denscombe

(2009). Item non-response rates: A comparison of online and paper questionnaires. International Journal of Social Research Methodology, 12(4), 281–291. https://doi.org/10.1080/13645570802054706

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2009). Internet, mail, and mixed-mode surveys: The tailored design method. Wiley & Sons

Education Week Research Center. (2014). Engaging students for success: Findings from a national survey. Education Week Research Center

10.

Egan

A. F.

Jones

S. B.

Luloff

A. E.

Finley

J. C.

(1995). The value of using multiple methods: An illustration using survey, focus group, and Delphi techniques. Society & Natural Resources, 8(5), 457–465. https://doi.org/10.1080/08941929509380936

11.

Every Student Succeeds Act. (2015). Pub. L 114-95, 20 U.S.C. sec. 6301 et. se.

12.

Fan

Yan

(2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26, 132–139. https://doi.org/10.1016/j.chb.2009.10.015

13.

Fletcher

Edward

Gordon

H. R.

Asunda

Zirkle

(2015). A 2015 status study of career and technical education programs in the United States. Career and Technical Education Research, 40(3), 191–211. https://doi.org/10.5328/cter40.3.191

14.

Gregson

J. A.

Allen

J. M.

(2005). Leadership in career and technical education: Beginning the 21^st century. University Council for Workforce and Human Resource Education.

15.

Hsu

C. C.

Sandford

B. A.

(2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1–8.

16.

Imperatore

Hyslop

(2017). CTE policy, past, present, and future: Driving forces behind the evolution of federal priorities. Peabody Journal of Education, 92, 275–289. https://doi.org/10.1080/0161956X.2017.1302221

17.

Imperatore

Hyslop

(2018) 2018 ACTE quality CTE program of study framework. Association for career and technical education. https://www.acteonline.org/wpcontent/uploads/2019/01/HighQualityCTEFramework2018.pdf

18.

Kraska

(1989). Certification and preparation of local vocational and technical administrators. Journal of Studies in Technical Careers, 11, 269–275.

19.

Lee

H. D.

Nelson

(2015). Instructional analysis and course development. American Technical.

20.

Leithwood

Louis

K. S.

Anderson

Wahlstrom

(2004). How leadership influences student learning. The Wallace Foundation. http://www.wallacefoundation.org/knowledge-center/school-leadership/key-research/Documents/How-Leadership-Influences-Student-Learning.pdf

21.

Linstone

H. A.

Turoff

(2002). The Delphi method: Techniques and applications. https://web.njit.edu/~turoff/pubs/delphibook/

22.

Manley

A. R.

Zinser

(2012). A Delphi study to update CTE teacher competencies. Education+ Training, 54(6), 488–503. https://doi.org/10.1108/00400911211254271

23.

Mobley

Sharp

J. L.

Hammond

Withington

Stipanovic

(2017). The influence of career-focused education on student career planning and development: A comparison of CTE and non-CTE students. Career and Technical Education Research, 42(1), 57–75. https://doi.org/10.5328/cter42.1.57

24.

National Policy Board for Educational Administration. (2018). National educational leadership preparation (NELP) program standards—District level. National Policy Board for Educational Administration. www.npbea.org

25.

Nettles

S. M.

Herrington

(2007). The single best idea to improve K-12 education. Peabody Journal of Education, (82)4, 724–736.

26.

Pennsylvania Department of Education. (2014, July). Framework for leadership: Types of evidence – CTC directors. https://www.education.pa.gov/Documents/TeachersAdministrators/Educator%20Effectiveness/Principals%20and%20CTC%20Directors/General%20%20Framework%20for%20Leadership%20Evidence%20List.pdf

27.

Rea

L. M.

Parker

R. A.

(2014). Designing and conducting survey research (4th ed.). Jossey-Bass A Wiley Brand.

28.

Rojewski

J. W.

Hill

R. B.

(2014). Positioning research and practice in career and technical education: A framework for college and career preparation in the 21st century. Career and Technical Education Research, 39(2), 137–150. https://doi.org/10.5328/cter39.2.137

29.

Rowe

D. A.

Alvesron

C. Y.

Unruh

D. K.

Fowler

C. H.

Kellems

Test

D. W.

(2015). A Delphi study to operationalize evidence-based predictors in secondary transition. Career Development and Transition for Exceptional Individuals, 38(2), 113–126. https://doi.org/10.1177/2165143414526429

30.

Sheehan

(2001). E-mail survey response rates: A review. Journal of Computer-Mediated Communication, 6(2), JCMC621.

31.

Smith-Hughes Act of 1917, Pub. L. 64-347; 39 Stat. 929.

32.

Strengthening Career and Technical Education for the 21st Century Act (Perkins V), Pub. L. 115-224 20 U.S.C. 2301 (2018). https://www.congress.gov/bill/115th-congress/house-bill/2353

33.

U.S. Department of Education. (2019, September 27). U.S. Department of Education releases interactive data story on career and technical education in high school. https://www.ed.gov/news/press-releases/us-department-education-releases-interactive-data-story-career-and-technical-education-high-schoo

34.

U.S. Department of Education. (2013, November). National center on education statistics, high school transcript study, 1990, 2000, 2005, and 2009: NCES data point trends in CTE coursetaking (NCES 2014-901). National Center on Education Statistics.

35.

Zirkle

(1998). Vocational administration requirements, certification and preparation. Journal of Vocational Education Research, 23(3), 239–251.

36.

Zirkle

Cotton

(2001). Where will future leadership come from? Tech Directions, 61(5), 15–18.

37.

Zirkle

C. J.

Jeffery

J. O.

(2017). Career and technical education administration: Requirements, certification/licensure, and preparation. Career and Technical Education Research, 42(1), 21–33. https://doi.org/10.5328/cter42.1.21

38.

Zirkle

Martin

(2012). Challenges and opportunities for technical and vocational education and training (TVET) in the United States. In Pilz

(Ed.), The future of vocational education and training in a changing world (pp. 9–23). Springer VS.

39.

Zirkle

Parker

McCaslin

N. L.

(2005). The changing state of career and technical education leadership development in the United States. In Gregson

J. A.

Allen

J. M.

(Eds.), Leadership in career and technical education: Beginning the 21^st century (pp. 63–94). University Council for Workforce and Human Resource Education.