Abstract
Keywords
Evaluating practitioner knowledge is a vital part of assessing professional social work competence (Drisko, 2014). Competent social workers need to have a significant understanding of both theoretical and practical knowledge (Drisko, 2014). However, there are varied expectations for how competency of knowledge should be evaluated in social work (Apgar, 2021; Bibus & Boutté-Queen, 2011; Donaldson, et al., 2016; Donaldson, et al., 2014). This article describes and reports on the process used to develop a single standardized instrument, the SWEAP Curriculum Instrument, Version 10, which aims to measure competency in knowledge of social work students at the undergraduate program level and generalist practice level of masters programs.
Requirements for the Assessment of Social Work Program Outcomes
The 2015 version of the Council on Social Work Education's (CSWE) Educational Policy and Accreditation Standards (EPAS) requires baccalaureate programs, as well as masters programs at the generalist practice level, to report outcomes from a multi-dimensional assessment of each of nine social work competencies (CSWE, 2015). Masters programs are additionally expected to develop and assess student practice of competencies at the specialized practice level designed by their own programs (CSWE, 2015). This article focuses on assessment of knowledge at the generalist practice level, not the specialized practice level. “Multi-dimensional assessment” under EPAS 2015 requires programs to assess students in at least 2 of the following categories for each of the 9 social work competencies. knowledge, value, skills, and cognitive/affective processes. Programs must elaborate in their assessment plan how they capture at least two of these dimensions for each competency.
Undergraduate and masters level social work programs must report on assessment of student competency at the generalist practice using two measures. One of those measures must involve the assessment of student competency in real, or simulated, practice situations; most often, programs use evaluations from field placement to meet this requirement. Programs have significant latitude to choose the second measure for use in their assessment process. Two common second measures include knowledge tests and competency-linked assignment rubrics.
Programs are required by CSWE to report on student achievement of professional competency, explained through performance related to program-defined benchmarks. Reports on the assessment of student competency are required through the program's self-study for initial accreditation or reaffirmation. This data is also required to be reported, and updated regularly, through public posting on each program's website (i.e., Assessment of Student Learning Outcomes- Form AS4B or Form AS4M) (CSWE, 2015).
Social Work Education Assessment Project
The Social Work Education Assessment Project (SWEAP) is a business that grew out of the effort of a group of undergraduate social work program faculty, field directors, and program directors to design standardized program assessment tools across social work programs. SWEAP is currently a Colorado State corporate entity co-owned by four executive team members (including two of the authors of the present article, Kathryn Krase and Tobi Delong Hamilton). Annual gross income of SWEAP in 2022 was approximately $85,000, with a profit of approximately $60,000.
All SWEAP instruments are copyrighted, and thus, unauthorized use of the instruments is punishable by applicable law. SWEAP instruments are purchased and administered online, for $5 per instrument. Programs use the SWEAP online user portal to distribute and organize responses. The SWEAP online user portal provides CSWE EPAS responsive reporting.
The original organization, known as the Baccalaureate Education Assessment Project (BEAP), was formed in the late 1980s to create instruments for use in internally and externally driven outcomes assessment. The initial team came together organically through the identification of a shared interest in the science and practice of program assessment.
In 2013, BEAP transitioned to SWEAP, reflecting the applicability of SWEAP standardized instruments to graduate, as well as undergraduate, social work programs. The SWEAP consulting and executive team is currently made up of a group of seven social work educators with experience from a diverse sampling of undergraduate and graduate programs across the country. Over the past 20 + years, 18 different social work educators have been part of the SWEAP team. The SWEAP team continues to grow and change. New members are often identified through networking at professional conferences, where mutual interest in improving program assessment and supporting program improvement is often found and cultivated.
All SWEAP team members have extensive experience in social work education, with particular expertise in outcomes assessment. Most SWEAP team members have served as the “assessment coordinator” at their institutions. Multiple SWEAP team members have served as BSW and/or MSW Program Directors, for a combined 20 + years of experience in these roles. Most of the team members have been responsible for the development of successful self-studies in support of initial CSWE accreditation and program reaffirmation at the undergraduate and/or graduate levels.
SWEAP currently supports the use of six different standardized instruments: the Curriculum Instrument, the Field Instrument, the Entrance Survey, the Exit Survey, the Employer Survey, and the Graduate/Alumni/ae Survey. The reliability and validity of SWEAP instruments have been supported through regular analyses (Buchan et al., 2004; Rodenhiser et al., 2007; Christenson et al., 2015; Sullivan et al., 2021; Delong Hamilton, 2009; Delong Hamilton, et al., 2011). The SWEAP team regularly reports these analyses through peer-reviewed conference presentations (SWEAP, n.d.)
SWEAP Curriculum Instrument, Version 10
The SWEAP Curriculum Instrument is a standardized knowledge instrument designed to evaluate students for readiness for social work practice at the generalist level. This instrument was originally called the foundation curriculum assessment instrument (“FCAI”). The FCAI was designed in response to EPAS 2001, but has been regularly updated over the years to reflect changes in social work curricular expectations. The EPAS 2015 responsive instrument, Version 10, has 53 items, in multiple choice format, with four answer options per item. The items are divided by their relation to each of the EPAS 2015 social work competencies. Below, please find a table detailing the number of questions in each curriculum area on the curriculum instrument.
The SWEAP Curriculum Instrument is often used as a second measure, separate from the requirement to assess student competency in real or simulated practice scenarios. Nearly, 300 undergraduate and 100 masters level social work programs have used one of the ten different versions of the SWEAP Curriculum Instrument, since its inception. Version 10 of the SWEAP Curriculum Instrument has been completed by over 30,000 undergraduate and social work students since it was introduced in 2016.
Concerns for the Use of Standardized Tests in Social Work
Standardized exams have long been used as an assessment of social workers’ competency for professional practice through the licensing process (Marks & Knox, 2009; Apgar, 2021), even though many have highlighted concerns (Thyer & Biggerstaff, 1989; Thyer, 1994). Presently, the Association of Social Work Boards (ASWB) is charged with developing and maintaining the social work licensing examinations that are used to evaluate a social worker's competence to practice ethically and safely (ASWB, 2022). Annually, more than 50,000 social workers take an ASWB exam toward the goal of licensure in their state (ASWB, 2022).
Over the years, many social work academicians have raised concerns about the relevance of standardized exams for social work practice, and the lack of connection to the educational objectives of social work programs (Cherry, Rothman, & Skolnik, 1989). Others have specifically criticized the ASWB for the lack of transparency related to the structure of the exams (Albright & Thyer, 2010; Thyer, 2011), and pass rates (DeCarlo, 2022). More recently, publication of statistics reporting that BIPOC social workers are significantly less likely to pass the ASWB exams, especially on the first try, has brought the relevance of the ASWB exams, all together, into question (CSWE, 2022).
The SWEAP team is conscious of the concerns of the use of standardized instruments in social work education and practice. This current article reports on the development of the SWEAP Curriculum Instrument, Version 10, including reliability and validity analyses conducted in support of the responsiveness to EPAS 2015.
Method
Development of the SWEAP Curriculum Instrument, Version 10
In July 2015, the SWEAP team revised the SWEAP Curriculum Instrument, to be responsive to the EPAS 2015 as an assessment of the dimension related to student knowledge. During the development of the SWEAP Curriculum Instrument, Version 10, the SWEAP team consisted of seven doctoral level faculty members and administrators, all with at least one social work degree, employed in five different colleges or universities, including four undergraduate and four masters level social work programs.
Over the course of a 3 day summer meeting, the SWEAP team reviewed the existing questions from Version 9 of the SWEAP Curriculum Instrument for their applicability to the new nine competencies of EPAS 2015. Some questions were deemed to no longer be adequate measures of the new EPAS 2015 competencies; some questions needed updated language; other questions were retained as is. Additional questions were developed by the team over the course of the meeting period to ensure an adequate number (at least 5, as determined by the group) per competency. Ultimately, 53 questions were identified; 5–7 per competency.
Instrument Pilot & Distribution
The first pilot of the SWEAP Curriculum Instrument, Version 10, commenced in August 2015 with a group of undergraduate social work students at a medium-sized, public, midwestern university. In the interests of time, the initial pilot sample included nine students from one undergraduate social work program available during the immediate summer session. The initial pilot sample was used to determine initial reliability and validity of Version 10 of the SWEAP Curriculum Instrument. This initial pilot established initial internal consistency through Chronbach's Alpha and test–retest reliability analyses. This initial pilot also established a baseline for item difficulty analysis. Findings from the initial pilot supported initial reliability and validity of the instrument, and thus the instrument was made available for wide scale application for undergraduate and generalist practice-level graduate level social work programs starting in November 2015. Additional data, aimed to evaluate instrument validity and reliability, was collected from November 2015 to December 2017, and included in the reliability analysis reported below.
Results
Validity Analysis
Construct, content and face validity of the entire instrument, as well as the competency level subscales, were secured through expert involvement in the process of instrument development. The seven doctoral-level social workers on the SWEAP team had, collectively, over a century of social work practice and a century of social work program experience, as classroom educators, field instructors, field directors and program directors at the undergraduate and graduate levels.
To support construct validity, the SWEAP team examined each EPAS 2015 competency as a construct. Questions relevant to the construct definition were identified or developed and assigned to each competency. To support content validity, the SWEAP team identified key areas of content for each EPAS 2015 competency. The team then identified, developed, and assigned relevant questions to address each key content area of each competency. To support face validity, the SWEAP team reviewed the draft tool to make sure each question was relevant to the construct and content identified for each competency. Criterion validity has not been established and is difficult to achieve in the context of the SWEAP Curriculum Instrument because of the limitations of access to comparison data at the student achievement level.
Construct, content, and face validity were all explored at the initial phase of Version 10 development, and continues on an annual basis at the SWEAP annual team meetings.
Reliability Analyses
Test–Retest Analysis
Test–retest reliability analysis using the pilot sample of nine students was conducted in an effort to determine the internal validity of the instrument, the competency level subscales, and the individual items. Correlation coefficients were calculated to test the relationship of correct answers on the initial testing experience to correct answers on the repeated testing experience. Correlation coefficients ranged from 0 (no correlation) to 1.0 (perfect correlation). Correlations of at least. Seven are considered acceptable for reliability (Zanotti et al., 2018).
Item test-retest coefficients ranged from 0.33 to 1.0, with a mean of 0.75. Competency level scale test-retest coefficients ranged from 0.25 to 0.87, with a mean of 0.58. Total scale level test–retest was 0.84. While some of the item test-retest and competency test–retest coefficients were very low, the sample was very small. These findings, therefore, generally support test–retest reliability of the SWEAP Curriculum Instrument, Version 10.
Internal Consistency–Chronbach's Alpha
Chronbach's alpha reliability analysis was conducted on the entire instrument, as well as on the competency level subscales, in order to evaluate the instrument's internal consistency. The closer Cronbach's alpha coefficient is to 1.0 the greater the internal consistency of the items in the scale (Gliem & Gliem, 2003).
Internal consistency analysis of all 53 items completed by 2382 student respondents over 24 months yielded a Chronbach's Alpha of 0.87. This figure supports internal consistency of the entire SWEAP Curriculum Instrument as a reliable measure of student knowledge. The Chronbach's Alpha when each individual item was singularly deleted ranged from: 0.860–0.867 Table 1.
Number of SWEAP Curriculum Instrument Items by EPAS 2015 Social Work Competency.
Reliability analysis of the subscales per competency found a range in Chronbach's Alpha from 0.303 to.602 (See Table 2). The low Chronbach Alphas of the various competency subscales can be explained by a number of reasons. The competency subscales ranged from 5 to 7 questions in composition; a small number of items covering a vast amount of curricular content. It is possible that competencies, as constructs, are too heterogeneous to capture in a small number of items. Additionally, it might be that the questions selected for inclusion in the SWEAP Curriculum Instrument, as a result of covering a heterogeneous construct, are not particularly inter-related.
SWEAP Curriculum Instrument Reliability Analysis Results.
Item & Scale Difficulty Analysis
Item difficulty analysis was conducted to determine the difficulty level of each item, and scale difficulty analysis was conducted to determine the combined difficulty of questions within each competency subscale. The item and scale difficulty numbers report on the percentage of participants who answered the item or scale correct. An item or scale difficulty analysis of.5–.7 for each item was considered ideal. The goal of item and scale difficulty in that range is to increase the likelihood that students selecting the correct answer were doing so because of their actual knowledge related to the item, and not likely due to their ability to guess the appropriate answer based on the available choices.
Individual item difficulty was calculated using data from 2382 participants ranged from .23 to .95. All items with a difficulty index of less than .4 (7 items) or above .8 (12 items), were reevaluated by the team, and modifications to the item language, or answer choices were made in an effort to bring the item difficulty analysis in line with expectations between .5 and .7.
Scale difficulty for the entire instrument, as well as for each of the nine competency subscales was calculated using data from 2382 participants. Findings ranged from 0.49–0.70 (See Table 2).
Discussion
The SWEAP Curriculum Instrument, Version 10, was developed to provide social work programs with a standardized knowledge test responsive to CSWE's EPAS 2015 (Sullivan, et al., 2020). The instrument's reliability and validity, reported above, support the use of the SWEAP Curriculum Instrument, Version 10, to measure student knowledge at the competency level. And thus, it is not surprising that dozens of undergraduate and graduate social work programs have successfully used the instrument towards accreditation and reaffirmation since 2015.
Critical in the development and use of the SWEAP Curriculum Instrument, is the responsiveness of this instrument to the changing role of assessment in social work education, and higher education, in general. There are numerous challenges experienced by small and large social work programs, especially when it comes to assessment. As regional accrediting bodies have placed more pressure on colleges and universities to use data to support their student outcomes, the burden of developing tools and reporting assessment has fallen on, already overburdened, faculty and program administrators.
The role of “assessment” coordinator (and other titles) is often handed to untenured faculty, or at-will staff, with little (or no) experience in program evaluation. Many of these colleagues have risen to the task, and successfully developed assessment instruments for their own programs, and provided excellent reporting. Many others have utilized professional networking to “crowd source” ideas for assessment in the interest of saving time, and not being alone in their process. Whether the program develops their own assessment tools, or not, they are still left with the tasks of designing, running, and interpreting reports based on the data.
A major benefit of the SWEAP Curriculum Instrument is that the time and energy otherwise necessary for developing tools, collecting data, and calculating outcomes is done methodically in a system designed by experts in social work educational assessment. By using standardized instruments through a mechanized and online process, program faculty and staff can, instead, focus on the bigger picture of using assessment findings to improve their programs, and better support their students.
While the focus on assessment has only increased in higher education, even in social work, there continues to be a dearth of published research on the validation of instruments used in these efforts (Christenson et al., 2015; Rowe et al., 2020). Limitations of validating assessment instruments through evaluation of data from a single social work program is a major concern (Rowe et al., 2020). Further research exploring the use of data collected using a singular instrument, but across multiple social work programs is strongly recommended to improve the assessment scholarship related to the validity of assessment instruments.
CSWE gives programs significant latitude in selected measures appropriate to their own program assessment. There is no CSWE requirement that programs provide evidence to support the validity or reliability of their assessment measures. However, the analyses reported here support the use of the SWEAP Curriculum Instrument as a valid and reliable instrument in the assessment of the knowledge dimension related to the EPAS 2015 Social Work Competencies.
Standardized instruments have benefits and limitations. One benefit of using a standardized instrument is that they are designed for a specific purpose and have support for their validity and reliability (Morgan, Gliner, & Harmon, 2006). However, testing bias can lead to over or under estimates in the performance of certain groups of students.
Recent attention in social work education has highlighted concerns for racial bias in the development and interpretation of standardized instrument results, with a focus on the ASWB exams for licensing (CSWE, 2022). The SWEAP team cannot report on differential student achievement on the SWEAP Curriculum Instrument based on demographic factors since student demographic information is not currently captured through the SWEAP Curriculum Instrument. Demographic questions were removed from the instrument during the transition from version 9 to version 10. The SWEAP team justification in 2015 for this decision was to keep the focus on program-level achievement, and not student-level achievement, as is the purpose of the instrument.
The SWEAP Curriculum Instrument, unlike the ASWB exam, is not designed to measure individual achievement; the instrument is designed to inform program assessment. For instance, the instrument does not cover all constructs relevant to each competency; choices for which items are included are made in the interests of instrument utility, as well as completeness. Student achievement on the instrument is best interpreted in the aggregate as a reflection of how the program has prepared them with knowledge necessary for competent practice as a social worker. The level of achievement on the instrument necessary to signal competent knowledge is selected by the program but needs to be justified under accreditation requirements.
With all of this in mind, the potential that the instrument is biased in design, and/or interpretation exists. As the SWEAP team develops and pilots the EPAS 2022 responsive SWEAP Curriculum Instrument, Version 11, we are taking steps to ensure that the expert panel who develops the instrument, as well as student groups who pilot the questions, are demographically diverse. The SWEAP team will also explore a pilot of the instrument that allows for demographic comparisons, in the interests of ensuring that the instrument, and its use, are not furthering bias of any form.
While the reliability and validity of the SWEAP Curriculum Instrument, Version 10, is generally supported through the present study, there are several limitations of the instrument that should be noted. As the next version of the instrument is developed, we will use a larger sample for piloting, and pay special attention to the internal consistency calculations as items are selected for inclusion in the final instrument that makes Version 11.
Even though the present sample used for most of the reliability analysis was large, it cannot be assumed that the nearly 3,000 students who completed the instrument are representative of students from all CSWE-accredited undergraduate and graduate programs. Consequently, the possibility of bias must be recognized.
There are barriers to the use of the SWEAP Curriculum Instrument, Version 10, most notably the expense (Rowe et al., 2020). SWEAP is a business, and the instruments, along with the reporting of the student data, must be purchased in order to be used. However, the value spent in the purchase of SWEAP instruments is, arguably, comparable to the level of service and expertise received from the products. When considering the expense of the SWEAP Curriculum Instrument, Version 10, one also needs to consider, in balance, the amount of time that the faculty or staff members responsible for assessment of a social work program spends on the design of instruments, collection of data, and then calculating statistics to report on that data.
Another barrier to the use of the SWEAP Curriculum Instrument, Version 10, is technology. All SWEAP instruments are now only available online. As a result, all SWEAP instrument users need to have internet access in order to complete instruments. However, SWEAP instruments have been optimized for completion on personal computers, as well as mobile devices.
Since the SWEAP Curriculum Instrument, Version 10, is responsive to the nine EPAS 2015 competencies, the instrument is only appropriate for use in the reporting of student outcomes data for undergraduate social work programs and the generalist practice experience of graduate social work students. Undergraduate programs with additional, program-defined, competencies could still use the SWEAP Curriculum Instrument, Version 10, and work with SWEAP to develop and operationalize a customized instrument to measure the additional competencies. Graduate programs can also work with SWEAP to develop and operationalize a customized knowledge test as an instrument to measure program-defined competencies for the specialized practice level. Customized instruments, however, do have an added fee to reflect the work of the SWEAP team to support the changes
The authors acknowledge that EPAS standards, as defined by CSWE change periodically, necessitating alterations to the instrument, and/or its interpretation. Programs must therefore be careful to appropriately choose and interpret the version of FCAI that is reflective of the current EPAS standards under which they are operating. SWEAP updates all instruments as new iterations of EPAS are released. Work is underway to develop and pilot the SWEAP Curriculum Instrument, Version 11, which is responsive to EPAS 2022. As conducted in the past, the SWEAP team will also provide guidance on how Version 10 questions relate to EPAS 2022 competencies, so that programs can translate their existing data under the new standards.
Conclusions
Assessment is an ongoing process that every social work program should engage in to ensure continuous program improvement. Program evaluation is important for accountability. Therefore, social work programs should have a well-articulated assessment plan that meets CSWE standards. A well-articulated assessment plan that provides detailed outcome data is the first step toward developing a culture of assessment within a social work program (Poulin, et al., 2007).
The SWEAP team acknowledges that SWEAP instruments are not the only valid and reliable instruments available for use. The SWEAP Curriculum Instrument, Version 10, is just one of many valid and reliable instruments available for programs to choose from. The beauty of the current process of accreditation and reaffirmation is that programs get to make their own, informed, decisions about what assessment tools to use.
Future research in the area should focus on guiding undergraduate and graduate social work programs as they strive for effective translation of their program assessment into valuable program improvements. Special attention needs to be paid to the characteristics of students, faculty, and staff utilized in the creation of assessment tools in order to ensure equitable, reliable, and valuable data that can be used for appropriate program evaluation. The SWEAP team is honored to join our colleagues in this process.
Footnotes
Declaration of Conflicting Interests
As owners of SWEAP, Kathryn Krase and Tobi Delong Hamilton have financial interests in the success of the SWEAP Curriculum Instrument reported in this manuscript.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
