Teaching Research and Practice Evaluation Skills to Graduate Social Work Students

Abstract

Objective: The authors examined outcomes of a graduate course on evaluating social work practice that required students to use published research, quantitative measures, and single-system designs in a simulated practice evaluation project. Method: Practice evaluation projects from a typical class were analyzed for the number of research references cited, type of client, goals or problems, measures, interventions, single-system designs, and outcomes. Results: More than half of the students conducted self-improvement projects monitored with self-report measures, and goals or problems selected and interventions applied varied widely. More than 80% of the projects were evaluated with simple AB designs, over 45% of which were associated with statistically significant improvements and an additional 43% showed gains that did not reach statistical significance. Conclusions: Results suggest that students can be taught techniques and skills needed to formulate interventions derived from published research and to evaluate effects of these interventions using single-system designs.

Keywords

teaching practice evaluation teaching program evaluation practice evaluation exercise graduate social work students single-case designs single-subject designs single-system designs

The Council on Social Work Education’s (CSWE) Educational Policy and Accreditation Standards (2008b) enjoins schools of social work to teach students how to evaluate interventions and the outcomes of their practice. The importance of practice evaluation is also reflected in the knowledge and practice behaviors contained in the instructional curriculum of the CSWE’s Advanced Social Work Practice in Clinical Social Work [ca. 2008a]. A well-developed method of program evaluation applicable to clinical practice is single-case or single-system research designs (SSDs). Not requiring large samples of homogenous subjects or random assignment to treatment and control groups, this methodology gauges treatment efficacy with procedures such as introducing and withdrawing, or successively administering treatment across individual problems or clients. Social work textbooks on single-system evaluation first appeared in the 1970s and that content has since infused mainstream social work education (Bloom, Fischer, & Orme, 2009; Grinnell & Unrau, 2011; Noia & Tripodi, 2008; Royse, Thyer, & Padgett, 2010).

One educational exercise that can give students direct experience in applying SSDs is a personal self-change project. Barth (1984) assigned social work students self-change projects aimed at improving their professional skills in the practicum setting. Student beliefs and behaviors affecting interactions with clients, agency staff, and persons in the community were monitored during baseline and subsequent intervention phases. Barth reported substantial gains in varied outcomes for several sample self-change projects during the intervention phases suggesting that these projects were effective educational assignments. Anderson (2000) used more conventional self-change projects focused on personal concerns or desired goals while teaching upper-level undergraduate psychology students. These students selected projects on topics such as improving time and money management, promoting healthy habits, increasing appropriate assertiveness, and decreasing negative self-statements, which they attempted to change with behavioral interventions and evaluated with AB (baseline, intervention) designs. Anderson reported that slightly more than half of these self-change projects reached the student’s goal and one third showed significant progress toward the goal. Similar to the previous study, Morgan (2009) had undergraduate psychology students implement behavior change projects aimed at personal habits such as exercise, caffeine consumption, or study time. Baseline and intervention data from these projects were later graphed and analyzed with statistical process control procedures. Morgan (2009) only described one sample student self-change project that obtained statistically significant improvements, but he reported that its results were “fairly typical” for the class.

An alternative assignment to self-change projects is applying evaluative methods in supervised practice with actual clients. In a study involving graduate-level social work students, Dillenburger, Godina, and Burton (1997) taught behavioral principles to students who then applied these techniques with a client in their field placement and evaluated their effects. These authors presented two sample client behavior change projects evaluated within SSDs that appeared to significantly increase clients’ desired behaviors. In an examination of process variables associated with group therapy, Johnson, Beckerman, and Auerbach (2001) instructed students on how to apply AB designs to evaluate effects of encouraging the development of trusting relationships between members on group attendance, and various interventions aimed at increasing verbal participation in a veterans’ support group. Employing statistical tests and software for SSDs, Johnson et al. (2001) found that changes in group processes associated with students’ interventions in these two examples reached statistical significance.

This article describes a graduate-level social work course using personal self-change and client behavior change projects to teach students techniques to evaluate their own practice. This class was not a clinical course in behavior therapy, behavior analysis, or any particular treatment approach, although it presented numerous illustrations involving behavioral interventions. In addition, this article reports on multiple measures of student performance for all evaluation projects from one semester, rather than reporting on selected and perhaps unrepresentative student projects.

Framework for the Course

Assigned Readings and Class Lectures

The text for this course was Evaluating Practice: Guidelines for the Accountable Professional (Bloom et al., 2009). The course content and weekly schedule closely followed the organization of the book. The first several chapters of the book present principles and strategies of measurement, and specific assessment procedures and instruments. The next set of chapters discusses uncontrolled case study and controlled single-system designs, and their appropriate applications and limitations. The final chapters of the book cover principles and methods of visual and statistical analyses. Class lectures explained key concepts and procedures, illustrated them with published studies relevant to social work practice, and provided opportunities for questions and discussion.

Practice Evaluation Projects (PEPs)

The main assignment for teaching students how to formulate and evaluate evidence-based interventions was the PEP. All students were required to conduct a PEP, which incorporated the following four tasks:

use the research literature to conceptualize a problem or a goal and to find evidence-supported interventions to improve the condition;

conduct an individualized assessment involving quantitative measurement of the above problem or goal;

design and apply one or more evidence-supported interventions; and,

evaluate effectiveness of the intervention/interventions using a single-system design.

PEPs could be based on interventions revolving around a client, a client system, or a self-management project. Client or client system projects aimed at assessing a concern or problem of a person or a social system (e.g., family, small group, and agency), designing and applying an intervention for the problem, and evaluating effects of the intervention. As an alternative to working with an actual client, students were allowed to recruit family members, friends, and acquaintances to participate as voluntary “clients” for these projects. Topics of client or system service plans were significant concerns or goals such as improving parenting skills or staff supervision practices; advancing social, self-care, academic, job-seeking, or recreational skills; or decreasing verbal aggression, marital conflict, child misconduct, or other interpersonal problems.

Another type of PEP was a self-management project that assessed a personal goal or problematic behavior, designed and applied an intervention to achieve the goal or to alleviate the problem, and evaluated effects of the intervention. Previous self-management projects were aimed at improving diet, exercise, sleep hygiene, study habits, and time management as well as reducing annoying habits, tics, compulsions, angry outbursts, negative and self-defeating thoughts, smoking, drinking, excessive spending, and other addictive behaviors.

Students were required to write a detailed description of their evaluation project using a format for research reports derived from the Journal Article Reporting Standards of the Publication Manual of the American Psychological Association (APA, 2010). Midway through the semester, students submitted a draft of their report and received feedback from the instructor that allowed them to improve their project methodology and to prepare a more refined final project report. At the end of the semester, students submitted their final report and gave an 8-min oral presentation that allowed students to learn about the procedures and results of all other evaluation projects in the class.

Evaluation of PEPs From One Sample Class

To evaluate outcomes of the previously described educational activities and assignments, all student PEPs from one semester were reviewed and analyzed. This course had been taught by the first author with the same basic format for the last 8 years. The course selected for evaluation represented an average or middle-of-the distribution class, containing neither an exceptionally strong nor exceptionally weak group of students (indicators of this will be discussed in the Discussion section).

Results

A total of 29 student PEPs were analyzed and the results are summarized in Table 1. The table presents the type of client, goals or problems, number of published references cited, measures used, interventions applied, and single-system designs employed.

Table 1.

Summary of Evaluation Project Reports

Client

Self (16)

Family member, friend, or acquaintance (8)

Clients (5)

Goal/Problem

Repetitive habit (e.g., nail biting, hair pulling, lock checking; 6)

Time management, money management, or personal organization (5)

Stress, anxiety, or panic (3)

Weight loss or high blood pressure (3)

Depression or negative thoughts (2)

Marital communication or conflict (2)

Aggressive behavior (1)

Bedwetting (1)

Fear of lizards (1)

Hyperactive behavior (1)

Recurring nightmares (1)

Sleep difficulties (1)

Social skills deficit (1)

Substance abuse (1)

Published references cited

Mean: 7.2

Range: 3–14

Measures

Self-report log (22)

Behavioral observation (2)

Marital interaction diary (2)

Parental or family log (2)

Weight scale, skinfold fat caliper (2)

Behavior problem checklist (1)

Diastolic and systolic blood pressure (1)

Random urine drug test (1)

Interventions

Engaging in an alternative response to replace the problem behavior (4)

Habit reversal (4)

Progressive muscle relaxation (4)

Graduated exposure (3)

Self-monitoring (3)

Social reinforcement (3)

Tangible reinforcement (3)

Cognitive behavior therapy (2)

Exercise (2)

Improved diet (2)

Meditation and controlled breathing (2)

Awareness training (1)

Brief solution focused therapy (1)

Brief timeout from reinforcement (1)

Cognitive restructuring (1)

Goal setting (1)

Guided self-dialogue (1)

Listening to classical music (1)

Personal budgeting (1)

Problem solving (1)

Response prevention (1)

Scheduling work on smaller tasks (1)

Sleep hygiene techniques (1)

Social skills training (1)

Evaluation designs

AB (27)

AB with reconstructed baseline (1)

ABB₁ (1)

ABAB (1)

ABCB (1)

A-B-BC-BD-BC-BD-BC (1)

B (1)

Type of Client

Students utilized themselves as clients in a slight majority of the projects (n = 16) focusing either on self-improvement or alleviating a personal problem. The remaining projects were divided between assisting a family member, a friend, an acquaintance or assisting actual clients.

Goal or Problem Selected

The most common concern selected for the project was a repetitive habit (e.g., nail biting, hair pulling; n = 6), closely followed by self-management concerns (e.g., time management, money management; n = 5). Stress or anxiety problems (n = 3) and weight loss and other health issues (n = 3) were additional recurring concerns. The remaining goals or problems were highly individualized and varied widely from unconstructive negative thoughts to a fear of lizards.

Published References Cited

The average number of published references (i.e., printed or online articles, book chapters, and books) cited in the PEPs was 7.2 (range = 3–14; SD = 2.73).

Measures Utilized

Over two thirds of the measures utilized in the projects were self-report logs or social interaction diaries (n = 22). Behavioral observation or family logs accounted for a much smaller portion of the measures (n = 6). The rest of the measures relied on mechanical devices (e.g., weight scale, blood pressure meter) or other physical tests (e.g., urine drug assay).

Interventions Applied

Multiple interventions were sometimes applied with a single concern, so the number of interventions exceeded the number of clients. Performing an alternative response to replace a problem behavior, habit reversal, and progressive muscle relaxation were the interventions used at the highest frequency (n = 4 for each). Graduated exposure, self-monitoring, and social or tangible reinforcement were the next most frequency utilized procedures (n = 3 for each). Cognitive behavior therapy, exercise, improved diet, and meditation were each used in two projects. A wide variety of interventions ranging from Brief Solution Focused Therapy to listening to classical music were applied once in the remaining projects.

Evaluation Designs Utilized

An overwhelming majority of the students utilized a simple AB design (A = baseline phase; B = treatment phase) to evaluate their interventions (n = 27). An AB design with a reconstructed baseline was used in one project and an AB₁B₂ design (revised treatment procedure in the third phase) was utilized in the second. One student used an ABAB or reversal design, and another student used a variant of this design, an ABCB. Finally, one student utilized a sophisticated A-B-BC-BD-BC-BD-BC design to analyze the combined and additive effects of multiple treatment procedures.

Outcomes of Student PEPs

At the beginning of this class, students were informed that their PEPs would be graded on their quality of planning and execution, not their outcomes. Nevertheless, data on PEP outcomes might give some indication as to the care and competence with which these assignments were carried out. What is an appropriate statistical test of single-system design data is a complex question without an answer supported by a broad consensus. In an effort to objectively quantify the outcomes of the present evaluation projects, we utilized the Conservative Dual-Criteria (CDC) method, a relatively straightforward statistical test developed by single-case design researchers and presented in the Bloom, Fischer, and Orme (2009) text. The CDC (Fisher, Kelley, & Lomas, 2003) requires the calculation of an adjusted baseline mean line and an adjusted baseline regression line that are projected into the treatment phase. In PEPs where the desired outcome of the intervention was to increase behavior, both the baseline mean line and regression line were adjusted by adding .25 standard deviation of the baseline data to the lines. In PEPs where the desired outcome was to decrease behavior, both the lines were lowered by .25 standard deviation of the baseline data. After this step was completed, the number of data points in the treatment phase above both lines was counted in PEPs where intervention behaviors were expected to increase, or the number of data points below both lines was counted in PEPs where behaviors were expected to decrease. If the number of data points in the treatment phase falling above (or below) these two lines equaled or exceeded the criterion number of points for the chosen significance level (based on the binomial test), the null hypothesis was rejected. The CDC test was run on each of the students’ evaluation projects graphs. Thus, if the student had more than one independent variable, she or he would have results from multiple CDC tests.

Results of CDC tests of evaluation project data showed that 15 of the 32 graphs contained statistically significant improvements at the .05 level in the transition from baseline to treatment phase. An additional 13 graphs displayed gains ranging from 23% to 600% where differences did not reach statistical significance or the data did not conform with requirements for the CDC test. The remaining graphs revealed no appreciable improvement from baseline to treatment phase, but in no instance was data in the treatment phase worse than in baseline.

Discussion and Applications to Social Work Education

The data obtained from the PEPs showed that as required by the course assignment students read and reviewed multiple research articles while formulating their simulated client’s goals or concerns. Using this research, students found or designed measures of those goals or concerns, and subsequently selected, adapted, applied, and evaluated evidence-based interventions to address those goals or concerns. Although PEPs were not graded on the basis of outcome, a large proportion of student projects were associated with statistically significant improvements or positive changes that did not reach statistical significance. These improvements suggested that student PEPs were well constructed and often efficacious in alleviating the targeted concern. PEPs encompassed a wide range of goals and problems, objective measures, and interventions, demonstrating during class presentations that social work practice aimed at a broad spectrum of personal and social concerns could be evaluated on an individualized and ongoing basis. In sum, PEPs appear to be a valuable classroom exercise for teaching social work students to evaluate their practice, as mandated by several CSWE standards.

One question about this study is whether the class chosen for analysis was representative of other classes, given the same instruction and assignments. In other words, could this particular class be giving an overly favorable impression about student performance and outcomes? It should be mentioned here that a deliberate effort was made to choose an average and not a superior or outstanding class for examination. Final grades and the number of projects employing innovative research designs (e.g., Wong, 2010) are a couple of indicators of class performance, and this class did not earn an especially large number of “As” or employ numerous sophisticated evaluation designs as compared to other classes of this type (the project incorporating an A-B-BC-BD-BC-BD-BC design being the one exception).

Another possible criticism of this study and the described course is that issues selected for assessment and intervention in the student PEPs did not represent the sorts of clinical problems encountered by social workers in the field. Indeed, serious problems such as domestic violence and child abuse were not subjects of the PEPs, and students were discouraged from selecting urgent or dangerous issues for this classroom exercise. Difficulties addressed by self-improvement projects, which constituted more than half of the PEPs, were unlikely to have been as severe as problems encountered on the job by professional social workers. However, instructional exercises and illustrations used for training are often less complex and demanding than actual problems dealt with by experienced professionals. Moreover, this course was not primarily clinical in nature and the main focus was teaching the application of assessment and program evaluation techniques.

Future work in teaching practice evaluation to social work students should strive to better integrate this instruction with students’ field placements and agency operation. Classroom-based courses separated from students’ field placements, such as the present one, do not prepare students for the difficulties and obstacles of practice evaluation in actual service agencies (e.g., short-term treatments, lack of baseline phases, limited access to standardized measures). In addition, the weak connections between classroom curriculum and field placement settings do not promote the adaption of academic evaluation techniques to the demands and constraints of service settings. Finally, separating instruction in evaluative methods from students’ placement settings gives up the opportunity to demonstrate to agency staff the value of these procedures for gauging treatment outcomes and developing more effective interventions. Although PEPs appear to be a useful exercise for blending instruction in research and practice, it is only a preliminary step in teaching social work students to use evidence-based research and research methodology to formulate and evaluate their practice.

Footnotes

Author’s Note

This article was invited and accepted at the discretion of the editor.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

References

American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). [Journal Article Reporting Standards (JARS), p. 247–249]. Washington, DC: Author.

Anderson

D. A.

(2000). Exposing undergraduates to behavior therapy: Self-change projects. Behavior Therapist, 23, 122–128.

Barth

R. P.

(1984). Professional self-change projects: Bridging the clinical-research and classroom-agency gaps. Journal of Education for Social Work, 20, 13–19.

Bloom

Fischer

Orme

J. G.

(2009). Evaluating practice: Guidelines for the accountable professional (6th ed.). Boston, MA: Pearson/Allyn & Bacon.

Council on Social Work Education. [ca. 2008a]. Advanced social work practice in clinical social work (p. 5). Retrieved February 18, 2012, from the Council on Social Work Education’s Website: http://www.cswe.org/File.aspx?id=26685

Council on Social Work Education. (2008b). Educational policy and accreditation standards (pp. 5, 7). Retrieved February 18, 2012, from the Council on Social Work Education’s Website: http://www.cswe.org/File.aspx?id=13780

Dillenburger

Godina

Burton

(1997). Training in behavioral social work: A pilot study. Research on Social Work Practice, 7, 70–78.

Fisher

W. W.

Kelley

M. E.

Lomas

J. E.

(2003). Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs. Journal of Applied Behavior Analysis, 36, 387–406.

Grinnell

R. M.

Unrau

Y. A.

(2011). Social work research and evaluation: Foundations of evidence-based practice. New York, NY: Oxford University Press.

10.

Johnson

Beckerman

Auerbach

(2001). Researching our own practice: Single system design for groupwork. Groupwork, 13, 57–77.

11.

Morgan

D. L.

(2009). Using single-case design and personalized behavior change projects to teach research methods. Teaching of Psychology, 36, 267–269.

12.

Noia

J. D.

Tripodi

(2008). A primer on single-case design for clinical social workers (2nd ed.). Washington, DC: NASW Press.

13.

Royse

Thyer

B. A.

Padgett

D. K.

(2010). Program evaluation: An introduction. (5th ed.). Belmont, CA: Wadsworth, Cengage Learning.

14.

Wong

S. E.

(2010). Single-case evaluation designs for practitioners. Journal of Social Service Research, 36, 248–259.