Building Research Capacity in a Mobile Psychiatric Rehabilitation Program: Service Learning in a University-Agency Collaboration

Abstract

Social service agencies face increasing demands for accountability, emphasizing the necessity of preparing professionals to effectively evaluate practice. University-agency collaborations incorporating service learning can build community research capacity while providing opportunities for application of student research skills. We describe a partnership model between a Master of Social Work program evaluation course and a mobile psychiatric rehabilitation program to design a formative evaluation. Course structure and tasks central to the management of the partnership are described. Agency staff benefited from access to peer-reviewed literature, introduction to standardized assessment instruments, and stimulation of staff discussion and critical thinking around service provision. Students valued the applied research experience including exposure to the agency context and a real-world impact of their work. Lessons learned and implications for university-agency collaborations are discussed.

Keywords

community-based research program evaluation engaged scholarship service learning social service education

Background/Significance

During the past 20 years, social service agencies and nonprofit organizations have faced increased pressure to demonstrate their effectiveness and document program outcomes, as the political and funding environment emphasizes the importance of accountability and measuring performance in making funding decisions (Mott 2006; Newcomer, Hatry, and Wholey 2004). To respond to the performance-based focus of funders, social service agencies need to demonstrate program effectiveness. Practitioners and other direct care professionals are now charged with the expectation to not only provide care and services to various client systems, but also evaluate their own practice, accentuating the significance of the ability to apply knowledge and skills of social science research. In addition, administrators must be aware of relevant empirical findings, collect and analyze data on multiple aspects of service provision, and incorporate these data to inform programmatic decision making (Wulczyn et al. 2014). The push for evidence-based practice further adds to the emphasis on documenting practice outcomes to ensure provision of the most effective interventions.

Yet, despite the fact that the field of social services increasingly demands these evaluation skills, Master of Social Work classroom instruction and agency internships often fall short of providing opportunities for students to develop them (Lery, Wiegmann, and Berrick 2015). Studies have shown that social work students experience a disconnection between research concepts and direct practice content areas during their educational training (Anderson 2002), and feel intimidated when faced with the prospect of conducting research (Morgenshtern et al. 2011). This averse attitude toward research concepts and application to practice may be the reason practitioners report either being unsure of the value of research (Rubin and Parrish 2007) or lacking confidence in their skills to conduct it (Howard, McMillen, and Pollio 2003; Kelly and Bronstein 2003; Montcalm 1999; Rubin and Parrish 2007). Lack of research expertise (Fook 2003) and demands from expanding caseloads (Gitterman 2014; Shaw 2005) are also frequently cited as reasons why practitioners avoid engaging in research activities or program evaluation. In addition, Barratt (2003) reports that many social service agencies fail to provide tangible support for staff to access and use research in their day-to-day work, which further segregates the knowledge advances made in social sciences from actual application in the field. These personal and organizational barriers work against the development of an individual’s capacity to conduct research or engage in evidence-based practice in social service agencies despite the contextual expectations for performance-based practice.

Building Research Capacity through University-Agency Collaboration

There are many potential advantages to collaborations between universities and agencies, including enhanced student learning, improved service delivery (Berg-Weger et al. 2004), and enhanced rigor of agency evaluations (Braga 2010). Agency staff may have limited research knowledge and time to conduct program evaluation. For example, in implementing and evaluating an evidence-based HIV prevention initiative, Kegeles, Rebchook, and Tebbetts (2005) reported that many community-based agencies did not have the skills to design an evaluation and were confused about different approaches to evaluation. Agency staff may also experience limited access to current empirical literature, while faculty may have difficulty staying informed of innovations in practice and the current challenges community agencies are facing (Berg-Weger et al. 2004). Connecting scholarly research with practice is imperative to providing better services to the community (Bolton and Stolcis 2003; Davidson and Bowen 2011), and university-agency partnerships are increasingly seen as effective mechanisms for facilitating communication across these institutions (Mathias et al. 2015). Community-based research addresses both of these needs and can offer a strategy to build social service research capacity (Shannon, Kim, and Robinson 2012). For example, in a collaborative evidence-based practice project, Bellamy and colleagues (2008) reported that community-based agency staff applied practice knowledge to shape the evaluation, an experience that resulted in more positive attitudes overall toward evidence-based practice. At the same time, university researchers gained access to current practice information and received feedback on the utility of potential research topics for practitioners. Similarly, Fitch and Grogan-Kaylor (2012) partnered with a residential treatment center for adolescents to assess shifts in the client population and evaluate the impact of specialized programming on adolescent functioning. Through the application of advanced statistical models, the university researchers analyzed existing agency administrative and clinical data in ways that agency staff reported to be extremely helpful for organizational decision making and evidence-based programming. Lery et al. (2015) also reported a successful university-agency partnership where Master of Social Work students developed their data analysis skills using data from a public child welfare agency to address agency-defined research priorities. As part of the collaboration, agency supervisory staff also received training in evidence-based decision making to assist in guiding case-level practice for line workers. This suggests that university-agency collaborations can serve as a successful mechanism to not only increase the evidence-based practice capacity of community organizations, but also effectively train future social workers in evaluation of practice (Bledsoe-Mansori et al. 2013).

However, a number of challenges are associated with agency collaboration as a model for teaching applied research methods, such as program evaluation, in graduate-level programs (Harder 2010). These challenges reflect the larger obstacles often inherent in university-agency partnerships, including need for the establishment of trust, power imbalances, failure to satisfactorily establish priorities, differing values, and conflicting university-agency demands (Dulmas and Cristalli 2011). A review of factors impacting the success of the university-agency partnerships suggests that the collaboration must serve the interests of both partners, communication regarding objectives and roles needs to occur early in the process, necessary resources must be available, and the timeframe should be realistic for the project proposed (Berg-Weger et al. 2004).

Connecting Service Learning to University-Agency Collaboration

Literature on collaboration between universities and community-based organizations often presents case examples where the university researcher offers expert knowledge and skills to assist an agency engaging in research. There has been limited discussion about student participation in this literature. The university-agency collaboration model can be further enhanced if we create an experiential learning opportunity for students, which offers both the tacit and explicit knowledge necessary for the effective integration of research skills (Pierce, McGuire, and Howes 2015). It can provide socialization for students as evidence-based practitioners as few field practicum provide venues for students to apply the concepts and skills discussed in research courses. It may also address the lack of opportunity to practice what students learn in research courses, which has been identified by students as a barrier to appreciating their research abilities and engaging in research activities (Freymond et al. 2014).

In addition to various teaching approaches such as case studies, simulations, involvement in faculty research projects, and integration of research in field experiences as methods of helping students to acquire knowledge and skills in conducting research (Shannon et al. 2012), teaching research via service learning has demonstrated promising outcomes. Numerous studies report improved student confidence in conducting research and appreciation of the ways research contributes to the professional knowledge base (Anderson 2002; Harder 2010; Knee 2002; Shannon et al. 2012). These experiential models of teaching and learning have had a consistent presence in the discourse of effectively teaching research methods.

Collaboration with a community agency to conduct program evaluation creates a natural environment where students can actively participate in the formulation of evaluation questions, assess resources needed (such as updated literature or statistical software), conduct literature reviews, assist in data analysis, and report research findings while anchoring their research activities in the context where services are delivered.

This paper focuses on a university-agency collaboration approach which incorporates a service learning element to develop student’s ability to conduct applied research and increase agency capacity to conduct program evaluation. The partnership is not defined by contract or an institutionalized arrangement, but rather, the initiation of the faculty member to facilitate evidence-based practice. This paper shares the lessons learned from this experience with the hope that it will increase the willingness of both academics and practitioners to participate in similar projects of capacity building for the current and future workforce in the social work and human service professions. The paper begins by describing the context of this collaboration, followed by a description of the implementation process of the university-agency collaboration model for an applied social work research (program evaluation) course. The outcomes of this collaboration from the perspective of both students and practitioners are presented. We conclude this paper with suggestions and recommendations for future collaborative endeavors.

Context

The semester-length applied course in program evaluation followed a required foundation course in research methodology. Five sections of the course were offered simultaneously. In three sections of the course, students selected a field practicum agency from among their group members on which to focus their evaluation plan. In two of the course sections, the university-agency collaboration model was piloted where the instructor formed a partnership to match each course section to a local social service program in need of an evaluation plan.

The instructor initially approached a large nonprofit social service organization headquartered near the university. Established in 1970, the organization operates a broad range of programs in 14 states. A mobile psychiatric rehabilitation (MPR) program was selected for collaboration based on the recommendation of division leaders. A recent recovery-focused, citywide initiative had disbanded all congregate care programs for individuals with chronic mental illness. A primary goal of this initiative was to promote community integration through the restructuring of social services to provide support to these individuals while they lived independently in the community. The collaborating program represented one of two MPR programs in the city which had been created in response to this initiative and had been in operation for two years. The program serves between 40 and 50 individuals at a time, with each participant receiving services primarily from one of four teams comprised of an MPR worker, peer specialist, and tenant services coordinator. The program funder did not require the program to produce outcome data for the first two years of operation. Out of a desire to be proactive, the agency staff identified a need for formative evaluation to assess the program’s success in meeting three goals of the initiative: reduction of rates of psychiatric hospitalization, increased community integration, and achievement of housing stability. Increased measurability of person-centered treatment planning goals was also identified as a priority area for a process evaluation.

Forging Relationship/Assessing Agency Readiness

The initial meetings served to underscore the scope of potential projects to ensure realistic expectations given the semester timeframe, to clarify roles and responsibilities, and to assess the readiness of the program to engage in program evaluation. Agency readiness for evaluation is typically conceptualized as having two main components: the capacity and the commitment to engage in utilization-focused research (Ramirez and Broadhead 2013). To assist in gaining program “buy-in” and help program staff to identify potential areas of focus for the evaluation, the instructor posed a series of questions to guide the discussion, including the following:

What questions do you have about your program (services, clients, staff)?

What do you feel that your program does well?

What do you wish your program did better?

The program’s capacity for engaging in evaluation was assessed through the following questions:

What outcomes are you currently accountable for?

What data do you currently collect?

How is this process working for you?

Are you happy with the way these outcomes are defined and measured?

Handouts and interactive activities for engaging with stakeholders, along with a general checklist to assess organizational readiness were adapted from the University of Wisconsin–Extension (University of Wisconsin–Extension: Cooperative Extension 2008). These initial activities aimed to assess staff attitudes toward, and perceptions of, the benefits of evaluation while identifying organizational resources and potential barriers. Finally, the need for involvement of additional stakeholders was probed by asking, “Who else should be at the table?” Additional resources for engaging stakeholders in the development of evaluation questions may be obtained through the Robert Wood Johnson Foundation (Preskill and Jones 2009). Approximately 20 hours of instructor time during the two months prior to the start of the semester was invested in staff engagement, assessment of agency readiness, identification of evaluation focus, collection of agency materials, and planning of semester logistics. For newer programs or those without prior evaluation experience, it is anticipated that the process of identifying the focus of the evaluation may be lengthier than established programs with a clear program theory and experience collecting outcome data.

Preparing Student Competency

The 14-week course provided instruction on program theory and logic modeling, ethics, data analysis, and utilization of findings. Two weeks of instruction each were devoted to content on formative, process, and outcome evaluation models. Consistent with a team-based learning approach, students developed familiarity with the content of the course, prior to applying it within the context of a group project (Michaelsen 1994). Four assignments were required for the course. First, each student completed an online training in research ethics through the National Institutes of Health. Next, students individually completed a logic model on an identified outcome of the collaborating program. Once this individual work was completed, students were divided into one of four groups. This small group format sought to capitalize on the enhancement of higher order cognitive skills produced by team-based learning (Michaelsen, Bauman, and Fink 2004), while mimicking the collaborative manner in which real-life program evaluation is conducted. Each student group was assigned an area of evaluation from a list generated by the agency at the start of the semester. In groups of four to five students, an evaluation plan was developed that includes a literature review, logic model, and a plan for data collection and analysis. The small group was also responsible for oral presenting the evaluation plan to the class and collaborating program. The presentation included a brief analysis of the agency context, problem statement, research question, a description of the method, and a description of the strengths and weaknesses of the proposed evaluation approach. This project was exempt from Institutional Review Board review.

Technical Support

In addition to classroom instruction, the instructor provided technical assistance to the student groups through management of the group process, provision of technical consultation on the evaluation plans, and facilitation of agency contact and feedback (see Figure 1).

Figure 1.

Facilitating the collaboration: Instructor roles and tasks.

Managing the Group Process

To provide structure to the group project, in-class meeting time was provided for the groups at the end of the weekly class session. Project management tools were distributed to facilitate delegation of tasks and creation of a timeline for completion of the proposal. These tools included a calendar on which significant completion dates were marked and a task list that itemized project tasks and included space for assignment of the task and projected completion date. These task lists facilitated timely completion of the projects, as well as assisting in ensuring that work was distributed equitably among group members. The instructor also created an online room and discussion board for each student group to facilitate ease of communication between group members outside of class session. In addition, students were asked to complete a group member assessment tool at the conclusion of the semester by which they anonymously evaluated the contribution of each group member. The assessment tool was made available to students at the beginning of the semester to ensure understanding of the criteria on which they were to be evaluated.

Technical Consultation

The in-class group meeting time provided an ongoing opportunity for the instructor to troubleshoot methodological issues with student groups, monitor progress on the overall project, and answer questions. Central to this role was balancing quality control of the research products and stimulating student-initiated solutions to research issues (Anderson 2002). The ninth week of the semester was designated exclusively to half-hour individual consultations with each student group. Students were requested to submit a draft of their work one week prior to the meeting to allow for instructor review. Meetings included feedback on the draft provided and the opportunity for students to ask questions and seek additional information and consultation.

Facilitating Agency-Student Collaboration

Prior to the start of the semester, the instructor requested that the program provide written materials describing their program and its services, along with relevant existing data collection tools. The instructor posted these materials on the course Web site, along with resources on measuring outcomes in psychiatric rehabilitation services from the peer-reviewed literature and the national Psychiatric Rehabilitation Association. Immediately prior to the agency presentation, students were provided class time with their groups to formulate questions. During the presentation, agency staff offered information on the program context, population served, and evaluation priorities. This in-class presentation allows program staff to answer questions that only they can answer, while emphasizing to students that their evaluation project is not just a classroom exercise (Anderson 2002).

The agency was asked to identify the preferred way for students to contact them with questions. To assist in managing the student-agency interactions, an electronic discussion board was created for students to post their questions regarding the agency and to share information gained. This helped to reduce duplicate questions to the collaborating program by allowing the instructor or other students to respond first.

A visit to the program site was planned for Week 10 of the semester. To avoid overwhelming the program with 20 students, and burdening students with transportation issues, each student group was asked to select a member to attend the program visit. The purpose of the site visit was to offer students an opportunity to observe agency operations and pose the more detailed questions that emerged in the process of completing the evaluation plan. Students presented their evaluation plans to the class during the final week of the semester. Program staff was invited to attend.

Outcomes

Building Research Capacity in Service Agencies

Six weeks following the student presentations, the instructor offered to meet with the collaborating program. The purpose of the meeting was to seek perceptions of the program staff regarding the collaboration and facilitate implementation of the evaluation plans that the program found most helpful. This key informant interview with the program director lasted approximately 50 minutes and took place at the agency site. Prior to the meeting, the program director and peer support coordinator collected verbal feedback from direct service staff during the monthly full staff meeting on the process of the collaboration and the usefulness of its products. These observations, along with the program director’s own perspective, were probed during the interview and recorded in hand-written notes.

The key informant interview with the program director of the MPR program revealed substantial incorporation of student work. While review of students’ logic models provoked thought and discussion on the underlying therapeutic assumptions of the program, program staff were most interested in the adoption of measurement tools to assess outcomes. Prior to the collaboration, the agency did not collect program-level outcome data or use standardized data collection tools. Through this collaboration, program staff began to consider systematic strategies to evaluate client progress. The new treatment review forms created by students to increase the measurability of individual client goals and facilitate longer term aggregation of data on goal achievement were adopted for use, along with the adaptation to track rates of psychiatric re-hospitalization. The new forms made use of Goal Attainment Scales, and included areas for both staff and participant input. The standardized Community Integration Measure (CIM; McColl, Davies, Carlson, Johnston, and Minnes 2001) recommended for assessment of community integration was positively received by the program. Although intended for use with individuals recovering from traumatic brain injury, staff reported that they liked the simplicity and strengths-based tone, and felt the instrument addressed the dimensions of community integration that they aimed to achieve for their clients. Following student suggestions, the program was discussing incorporation of the instrument into the intake assessments and treatment review paperwork to create a pre- and posttest evaluation. This new approach would generate program-level knowledge and facilitate adoption of evidence-based practice. In the absence of published standardized measures to assess housing stability, students compiled a list of issues that were reported by staff to contribute frequently to loss of housing (e.g., nonpayment of rent, property destruction, prohibited activities in the apartment, sanitary issues, etc.). A weighted scoring spreadsheet was created that rated some items (such as nonpayment of rent) as more serious than others (disagreements with other tenants) and produced a participant housing stability score out of 100 points. Program staff were interested in the idea of being able to quantify housing stability based on their collective clinical observations, but stated that the scoring sheet might be too complicated for daily use. The form was, however, used as the basis for a discussion in staff meeting on how they would like to measure this outcome. During the follow-up period, the program funder indicated the need for greater specification of the role of the tenant services coordinator and definition of related outcomes. The collaboration facilitated a clearer operational definition for both the tasks of the tenant services coordinator and the intended housing outcomes.

Program staff indicated that another impact of the collaboration was the stimulation of discussion and critical thinking around the provision of services. Rather than focusing exclusively on provision of direct services, line staff reported more attention individually and collectively to the process of service delivery. “It really got us talking more about how and why we do things a certain way.” The program director reported that access to relevant peer-reviewed literature and standardized instruments was also beneficial. For example, staff used literature presented by the students on risk factors for psychiatric re-hospitalization to help identify program participants at higher risk in advance for targeted support.

Building Research Capacity for Future Practitioners

Overall, student learning objectives for the course were met as measured by grading rubrics that were implemented across all five sections of the course, including three sections that did not implement a collaboration model. These objectives included the following: (1) develop skills in accessing research literature as a foundation for practice and program evaluation; (2) develop skills in conducting social science inquiries (in this case, program evaluation) in a well-organized manner, with clear communication to major stakeholders throughout the process; and (3) participate in the generation of applied social science knowledge through developing and implementing intervention evaluations. The grading rubrics used assessed these research-related competencies through standardized evaluation of course assignments.

Student impressions of this service learning experience were systematically assessed via open-ended survey questions that were administered at the end of the course concurrent with an instructor rating survey. The instructor rating survey is a standardized tool developed by the university, which consists of 18 Likert-type scale questions that assess student perceptions of the preparation and delivery of course content. The open-ended questions anonymously solicited qualitative feedback on aspects of the course that students liked, disliked, and felt could be improved. Both surveys were administered without the presence of the course instructor to ensure objectivity.

Open-ended survey questions were analyzed using open coding, followed by axial coding. To enhance validity of results, the responses and coding were assessed independently by two investigators/instructors, one of whom did not use the university-agency collaboration model.

The most frequent positive comments on the course material and structure spoke to the benefits of having “real world examples of program evaluation” and that “[t]he real life example [of the agency] was helpful.” Providing this context appeared to reinforce for students the utility and applicability of program evaluation for social work practice. One student wrote, “Doing a program evaluation rather than a scholarly research paper has excellent utility for the real world setting.” Another reported, “This course is directly related to what’s happening in the field.” A third student observed, “Program evaluation is a very important concept . . . [t]he more I become aware of different social service agencies the more I see the need.”

The benefits of team-based learning were also highlighted, along with the challenges of group work. The discussion and feedback among peers appeared to assist in comprehension and application of the course concepts. For example, “It was much more helpful working in a group rather than doing the project by myself,” and “The [group] discussion helped me better apply the material that was being taught in class through my classmates’ feedback.”

However, group work presented some practical challenges that were frequently noted such as, “It is extremely difficult to orchestrate schedules to accommodate the requirement for a group project” and “Doing a group paper is difficult to coordinate schedules and people’s writing styles vary so much . . .” One student proposed a potential solution to “[g]ive students more time to meet with their groups in class.”

Discussion/Lessons Learned

A university-agency collaboration model presents one method for building capacity within existing community organizations facing increased demands for accountability and infrastructure to support implementation of evidence-supported practices (Barratt 2003; Mott 2006). By facilitating critical agency discussion of programmatic strengths and challenges, more clearly articulating agency structures and functions, and building a structure for practice-guided data collection, we may enhance the capacity of community-based agencies to engage in quality assurance processes and evidence-based decision making (Lery et al. 2015). University-agency collaboration also offers a potential solution to the pedagogical dilemmas facing social work educators in engaging Master of Social Work students with course content on program evaluation. It requires the application of research course content to a practice setting, addressing concerns of increasing student confidence in skills (Harder 2010; Shannon et al. 2012), and demonstrating the value and applicability of research concepts (Rubin and Parrish 2007). During this project, students participated in the refinement of research questions, review and application of the empirical literature, and the design of a program evaluation that successfully addressed both the research priorities of the agency and the resource limitations of the setting.

Success and Challenges of the Pedagogical Model

From a pedagogical perspective, students enjoyed exposure to the collaborating agency and knowing that their work would have a real-world impact. The exposure to the agency setting, the practical challenges to be addressed through research, and the feedback from the agency staff seemed to reinforce the value of research, its applicability to practice, and its place within the skill set of professional social work. However, the constraint of the available projects within the priorities of the agency frustrated students who resented the limitations on their choice, and this may have dampened the engagement that the approach sought to engender. One solution posited in the prior literature is offering students a range of projects to choose from (within the same agency) or using students’ individual field agencies as the collaborating programs (Anderson 2002). The first approach may strike a balance between offering students choices and the instructor time required to assess readiness and identify clear evaluation goals. This requires selection of a fairly complex agency with numerous evaluation needs. The second approach may increase student engagement and comprehension due the application of course theories to a practice situation with which they are already familiar. In addition, it addresses some of the logistical barriers of time and transportation to the collaborating program as students are already present at the collaborating agency for their field practicum. However, there is less ability for the instructor to assess the readiness and capacity of the program to engage in the evaluation process or to monitor the applicability and feasibility of the students’ proposal.

The team-based learning approach adopted reflects the implementation of program evaluation in community settings involving multiple stakeholders. As a pedagogical model, collaborative learning may also improve students’ assessment of their research self-efficacy (Macke and Tapp 2012). Students did report the benefits of social support and learning to combine complementary skills as part of the group process. While previous literature has reported good results with groups of five to seven students (Anderson 2002), our students reported that groups of four to five were too large. This resulted in fragmentation of learning from division of tasks. Students who did not actively engage with an aspect of the course content through hands-on work in the proposal reported not feeling confident in their understanding of the concepts or future ability to apply them. The size of the groups also presented challenges in coordinating schedules outside of class given that our program is comprised primarily of part-time students with multiple responsibilities.

Oversight of group dynamics and assurance of individual accountability within the group project can present additional challenges in the team-based, service learning model. While the majority of the student teams worked together successfully, interpersonal conflicts and unfair distribution of project tasks must be monitored and addressed by the instructor, with the objective of skill acquisitions for all students and production of an end product meeting the needs of the community-based agency. Utilization of the task list assisted considerably in ensuring fair distribution of the project workload from the beginning. Use of a group log to record progress on tasks has also been suggested (Anderson 2002). In the future, engaging in team-building exercises and the cooperative setting of group guidelines at the stage of group formation (Barkley 2010) may build capacity among student teams for managing their own group dynamics and enforcing group norms. This experiential aspect of working in a team to conduct systemic investigation of service outcomes is key to preparing students to effectively negotiate collaboration with multiple stakeholders and fair distribution of team tasks in their roles as helping professionals.

Refining the University-Agency Collaboration Model

The importance of careful selection of the collaborating agency should not be underestimated. This partnership represented several characteristics that have been identified as necessary to the success of university-agency collaborations, including the recognition of mutual benefit, a commitment to collaborative process, clear roles and communication structures, and an established process to monitor and assess progress on the planned tasks and activities (Mathias et al. 2015). In addition to the factors identified previously as critical to success of the partnership (Berg-Weger et al. 2004; Mathias et al. 2015), organizational culture can impact the agency’s capacity to engage in and use findings from program evaluation (Weiner, Amick, and Lee 2008). Motivation is only one aspect of organizational readiness; institutional resources (adequate staffing, space, and other resources), staff attributes (an orientation to professional growth, learning, and adaptability), and organizational climate (cohesion, inclusive leadership style, communication flow) (Bledsoe-Mansori et al. 2013; Simpson 2009) are also critical elements in fully and successfully engaging in program evaluation efforts. Some of these organizational elements can be difficult for an instructor to assess without substantial familiarity with the agency. In the authors’ experience, a lack of resources and top-down leadership style are the potential barriers that are most likely to be visible during the initial engagement process. Administration of a standardized instrument to assess organizational readiness to change or organizational functioning during the engagement process could be one way to address this, given there is still a lack of consensus in the conceptualization and measurement of organizational readiness for change in the existing literature (Weiner et al. 2008). The Organizational Readiness for Change instrument (ORC-D4; Texas Christian University 2009) may be especially well suited to this task as it addresses the motivation to change, staff attributes, adequacy of resources, and organizational climate. Weiner et al. (2008) provide a list of available measures of organizational culture and the dimensions assessed in their systematic review (Weiner et al. 2008).

An additional practical concern is that the agency location, hours, and accessibility of staff should be matched to the needs of the students. For example, an agency with evening hours should be sought for course sections of part-time, evening students. The availability of agency information electronically is also an important issue, and the agency should not only have a strong Web presence through which students can access background information on the agency, but electronic copies of information such as policies, data collection forms, and staff organization charts that can be readily shared.

Conclusion

Strengthening partnerships between academic and practice communities is a crucial element for social work and human service professions. As a method of fostering research skills in graduate students, the university-agency collaboration model can be time-intensive, requiring a commitment from the instructor both pre- and postsemester, making it unfeasible for replication more than annually. Given the nature of the project, it necessitates ongoing attention to balancing the instruction needs of students and the capacity of the partnering agency. The instructor’s role is more than that of an educator; it also includes actively managing and supervising the program evaluation design process. Parallel to the multiple roles of a program evaluator discussed by Hirsch and Quartaroli (2009), the process of designing an evaluation that incorporates the time, staffing, budgetary, and technological constraints of the agency context can be frustrating for students and challenge the problem-solving skills of the instructor. However, the complicated nature of this applied research project may be what prepares students to implement these skills in their future practice settings, and as such, represents one model of building evaluation capacity for current social service agencies and future practitioners.

Related Resources

American Evaluation Association Coffee Break Series. A series of 20-minute webinars that introduce useful tools for evaluators. See http://comm.eval.org/coffee_break_webinars/CoffeeBreak/.

Coalition for Evidence-Based Policy. A non-profit organization that disseminates evidence on program effectiveness and resources for the conduct of program evaluation. See http://evidencebasedprograms.org.

Community Tool Box. A free, online resource developed and managed by the KU Work Group for Community Health and Development to promote community development, social change, and community-based assessment and evaluation. See http://ctb.ku.edu/en/toolkits.

FRIENDS: National Resource Center for Community-Based Child Abuse Prevention-Evaluation Toolkit. A free resource for developing an individualized outcome evaluation plan. See http://friendsnrc.org/evaluation-toolkit/.

The Innovation Center for Community and Youth Development. Offers consultation, training, and evaluation tools for organizations serving youth. See http://theinnovationcenter.org.

U.S. Department of Health and Human Services, Agency for Healthcare Research and Quality. An agency within the U.S. Department of Health and Human Services that disseminates tools for improving and assessing quality of health care and patient safety. See https://www.ahrq.gov/tools/index.html.

University of Wisconsin Cooperative Extension: Program Development and Evaluation. Online and in-person training resources for developing and evaluating social service programs. See http://www.uwex.edu/ces/pdande/.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Amber Holbrook is an assistant professor in the Graduate Department of Social Work at West Chester University. She received her PhD from Bryn Mawr College. Her research has focused on maternal addiction, and the pedagogy of teaching research and program evaluation. She is currently working on a program evaluation of an effort to disseminate dialectical behavior therapy in adult residential programs.

Wan-Yi Chen, PhD, is a professor in the Graduate Social Work Department at West Chester University. Her research interests include the relationship between victimization, both at home and in the community, and the behavioral outcomes of youth and the caregivers. Her recent work also explores various pedagogies to train students for social work practice.

References

Anderson

Steven G.

2002. “Engaging Students in Community Based Research: A Model for Teaching Social Work Research.” Journal of Community Practice 10(2):71–87. doi:10.1300/J125v10n02_05.

Barkley

Elizabeth F.

2010. Student Engagement Techniques: A Handbook for College Faculty. San Francisco, CA: Jossey-Bass.

Barratt

Moira

. 2003. “Organizational Support for Evidence-based Practice within Child and Family Social Work: A Collaborative Study.” Child & Family Social Work 8(2):143–50.

Bellamy

Jennifer L.

Bledsoe

Sarah E.

Mullen

Edward J.

Fang

Lin

Manuel

Jennifer I.

2008. “Agency-university Partnership for Evidence-based Practice in Social Work.” Journal of Social Work Education 44(3):55–75.

Berg-Weger

Marla

Tebb

Susan S.

Cook

Cynthia A. Loveland

Gallagher

Mary B.

Flory

Barbara

Cruce

Ashley

. 2004. “The Collaborative Research Education Partnership: Community, Faculty, and Student Partnerships in Practice Evaluation.” Journal of Community Practice 12(3–4):141–62.

Bledsoe-Mansori

Sarah E.

Bellamy

Jennifer L.

Wike

Traci

Grady

Melisa

Dinata

Erna

Killian-Farrell

Candace

Rosenberg

Karla

. 2013. “University-agency Partnerships for Evidence-based Practice: A National Survey of Schools of Social Work.” Social Work Research 37(3):179–93.

Bolton

Michael J.

Stolcis

Gregory B.

2003. “Ties That Do Not Bind: Musings on the Specious Relevance of Academic Research.” Public Administration Review 63:626–30.

Braga

Anthony A.

2010. “Setting a Higher Standard for the Evaluation of Problem-oriented Policing Initiatives.” Criminology & Public Policy 9:173–82.

Davidson

M. Meghan

Bowen

Nancy

. 2011. “Academia Meets Community Agency: How to Foster Positive Collaboration in Domestic Violence and Sexual Assault Work.” Journal of Family Violence 26:309–18.

10.

Dulmas

Catherine N.

Christalli

Maria E.

2011. “A University-community Partnership to Advance Research in Practice Settings: The Hub Model.” Research on Social Work Practice 22(2):195–202.

11.

Fitch

Dale

Grogan-Kaylor

Andrew

. 2012. “Using Agency Data from Evidence-based Programming: A University-agency Collaboration.” Evaluation and Program Planning 35(1):105–12.

12.

Fook

Jan

. 2003. “Social Work Research in Australia.” Social Work Education 22(1):45–57.

13.

Freymond

Nancy

Morgenshtern

Marina

Duffie

Mark

Hong

Liu

Bugeja-Freitas

Shirley

Eulenberg

Jesse

. 2014. “Mapping MSW Research Training.” Journal of Teaching in Social Work 34(3):248–68.

14.

Gitterman

Alex

. 2014. “Engaging Social Work Practitioners in Research: Challenges and Opportunities.” Social Work in Health Care 53(10):921–31.

15.

Harder

Jeanette

. 2010. “Overcoming MSW Students’ Reluctance to Engage in Research.” Journal of Teaching in Social Work 30(2):195–209.

16.

Hirsch

Michael L.

Quartaroli

Tina A.

2009. “Many Hats: The Methods and Roles of the Program Evaluator.” Journal of Applied Social Science 3(2):73–80.

17.

Howard

Matthew O.

McMillen

Curtis J.

Pollio

David E.

2003. “Teaching Evidence-based Practice: Toward a New Paradigm for Social Work Education.” Research on Social Work Practice 13(2):234–59.

18.

Kegeles

Susan M.

Rebchook

Gregory M.

Tebbetts

Scott

. 2005. “Challenges and Facilitators to Building Program Evaluation Capacity among Community-based Organizations.” AIDS Education and Prevention 17(4):284–99.

19.

Kelly

Timothy B.

Bronstein

Laura R.

2003. “The Folder Feedback System: Making Research Content More Understandable, Enjoyable, and Usable.” Social Work Education 22:261–70.

20.

Knee

Ryan T.

2002. “Can Service Learning Enhance Student Understanding of Social Work Research?” Journal of Teaching in Social Work 22:213–25.

21.

Lery

Bridgette

Wiegmann

Wendy

Berrick

Jill D.

2015. “Building an Evidence-driven Child Welfare Workforce: A University-agency Partnership.” Journal of Social Work Education 51(Suppl. 2):S283–98.

22.

Macke

Caroline

Tapp

Karen

. 2012. “Teaching Research to MSW Students: Effectiveness of the Team-based Learning Pedagogy.” Journal of Teaching in Social Work 32:148–60.

23.

Mathias

Christine

Gilman

Elizabeth

Shin

Carolyn

Evans

William

. 2015. “California’s Title IV-E Partnership: A Statewide University-agency Collaboration-characteristics and Implications for Replication.” Journal of Social Work Education 51(Suppl. 2):S252–70.

24.

McColl

Mary Ann

Davies

Diane

Carlson

Peter

Johnston

Jane

Minnes

Patricia

. 2001. “The Community Integration Measure: Development and Preliminary Validation.” Archives of Physical Medicine and Rehabilitation 82(4):429–34.

25.

Michaelsen

Larry K.

1994. “Team-based Learning: Making a Case for the Small Group Option.” Pp. 139–53 in Handbook of College Teaching, edited by Pritchard

K. W.

Sawyer

R. M.

Westport, CT: Greenwood Press.

26.

Michaelsen

Larry K.

Bauman

Arletta K.

Fink

L. Dee

. 2004. Team-based Learning: A Transformative Use of Small Groups in College Teaching. Sterling, VA: Stylus Publishing.

27.

Montcalm

Denise M.

1999. “Applying Bandura’s Theory of Self-efficacy to the Teaching of Research.” Journal of Teaching in Social Work 19(1–2):93–107.

28.

Morgenshtern

Marina

Freymond

Nancy

Agyapong

Samuel

Greeson

Clare

. 2011. “Graduate Social Work Students’ Attitudes toward Research: Problems and Prospects.” Journal of Teaching in Social Work 31(5):552–68.

29.

Mott

Andrew

. 2006. “Hand in Hand: Evaluation and Organizational Development.” Nonprofit Quarterly 13(1):42–47.

30.

Newcomer

Kathryn E.

Hatry

Harry P.

Wholey

Joseph S.

2004. “Meeting the Need for Practical Evaluation Approaches.” Pp. 1–10 in Handbook of Practical Program Evaluation, edited by Wholey

J. S.

Hatry

H. P.

Newcomer

K. E.

, 2nd ed. San Francisco, CA: Jossey-Bass.

31.

Pierce

Barbara

McGuire

Lisa

Howes

Patricia

. 2015. “Ready, Set, Go . . . Again: Renewing an Academy-agency Child Welfare Partnership.” Journal of Social Work Education 51:S239–51.

32.

Preskill

Hallie

Jones

Nathalie

. 2009. A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions. Princeton, NJ: The Robert Wood Johnson Foundation.

33.

Ramirez

Ricardo

Broadhead

Dal

. 2013. Utilization Focused Evaluation: A Primer for Evaluators. Penang, Malaysia: Southbound.

34.

Rubin

Allen

Parrish

Danielle

. 2007. “Promoting and Sustaining Evidence-based Practice: Challenges to the Future of Evidence-based Practice in Social Work Education.” Journal of Social Work Education 43(3):405–27.

35.

Shannon

Patrick

Kim

Woosoo

Robinson

Adjoa

. 2012. “Implementing a Service Learning Model for Teaching Research Methods and Program Evaluation.” Journal of Teaching in Social Work 32(3):229–42.

36.

Shaw

Ian

. 2005. “Practitioner Research: Evidence or Critique?” British Journal of Social Work 35(8):1231–48.

37.

Simpson

D. Dwayne

. 2009. “Organizational Readiness for Stage-based Dynamics of Innovation Implementation.” Research on Social Work Practice 19:541–55.

38.

Texas Christian University. 2009. “Readiness for Organizational Change.” Institute of Behavioral Research. Retrieved June 16, 2017 (https://ibr.tcu.edu/forms/organizational-staff-assessments/).

39.

University of Wisconsin–Extension: Cooperative Extension. 2008. Building Capacity in Evaluating Outcomes: A Teaching and Facilitating Resource for Community-based Programs & Organizations. Madison, WI: UW Extension, Program Development & Evaluation.

40.

Weiner

Bryan

Amick

Halle

Lee

Shoou-Yih D.

2008. “Conceptualization and Measurement of Organizational Readiness for Change.” Medical Care Research and Review 65:379–436.

41.

Wulczyn

Fred

Alpert

Lily

Orlebeke

Britany

Haight

Jennifer

. 2014. “Principles, Language, and Shared Meaning: Toward a Common Understanding of CQI in Child Welfare.” Retrieved July 20, 2016 (http://www.chapinhall.org/research/report/principles-language-and-share-meaning-toward-common-understanding-cqi-child-welfare).