Promoting Evidence-Informed Practice in Child Welfare in Ontario

Abstract

Practice and Research Together (PART) is an Ontario-based research utilization initiative, the core function of which is to distil and disseminate practice-relevant research findings to child welfare practitioners. This article addresses (a) the mission and goals of the PART program; (b) the key components of the program design; (c) the conceptual foundations of evidence-informed practice (EIP) as it relates to the program; (d) the successes and challenges of implementation to date; (e) the results of a comprehensive evaluation; and (f) areas for future research and development. Key findings of the formative evaluation include Link PARTners (LPs—organizational representatives) feeling isolated in their role in promoting organizational change; front-line practitioners reporting that they have little time or resources to use the program materials; supervisors stating that they support the concept of EIP but lack the skills and abilities to move these ideas forward; and executive directors are requesting more evidence to promote organizational and systemic change.

Keywords

child welfare evidence-informed practice implementation

Practice and Research Together (PART) Program Mission and Goals

The overall mission of PART is to “promote the understanding and use of evidence-informed practice (EIP) at all levels of the child welfare system in Ontario to ensure the best possible services and outcomes for vulnerable children and families” (PART’s Vision Statement, February 2011). The program achieves this mission through its goals, which are to:

Identify and package evidence in user-friendly formats to disseminate to child welfare practitioners and agencies;

Promote the use of EIP using a range of program components such as PARTicles, webinars, learning events, and electronic journal access;

Create organizational and systemic learning environments in child welfare organizations to promote critical thinking and ongoing use of evidence in practice (PART’s Mission Statement, February 2011).

The evolution and implementation of the PART program and its unique components has been an interesting journey that began as the result of a site visit to an organization known as RiP (Research in Practice) located in the United Kingdom.

Implementing PART: Historical Overview

PART was first developed in 2006, when a group of child welfare leaders went to the United Kingdom to visit RiP, a world-renowned knowledge translation organization located at Dartington Hall, Totnes. RiP assists with the implementation of EIP in social service organizations that work with vulnerable children and families. The site visit explored whether the RiP model could be replicated within the Ontario child welfare system. The visiting child welfare leaders concurred that the U.K. model had significant potential to create change within the Ontario child welfare system, and this marked the genesis for the development of the PART model. PART opened for business on September 24, 2007, beginning with a modest membership of 18 out of a total of 53 child welfare agencies in Ontario. By January 2009, the membership had grown to 37 agencies—70% of those in the Province of Ontario. This membership roster has remained intact despite significant provincial cutbacks. The child welfare system in Ontario is currently undergoing significant restructuring with several agencies merging. PART intends to recruit all child welfare organizations in the province to be members by April 1, 2011.

At the beginning of the program’s inception, each member agency paid a total of $15,300 (CAN) per annum membership fee to cover the costs of running the PART program. The membership fee includes all costs related to PART activities including attendance at the large-scale conferences. Child welfare agencies in the north requested a change to this “one size fits all” membership fee structure. In light of this, the membership fee scale has been updated to include a banding model (small, medium, and large) that structures the fees to reflect the organization’s revenue base. This change has been a catalyst for smaller size organizations to join the program. Organizations sign a membership agreement and make a commitment to remain with the program for a period of 2 years. The PART infrastructure is small and includes three full-time staff members and various contract employees. By minimizing its infrastructure costs, PART is able to channel the majority of its revenues into creative program design components for its member agencies.

The implementation process, in large part, has been a process of trial and error. The program unfolded in a way that took into account the degree of organizational readiness of member agencies, practitioner feedback, and cost-containment challenges. The goal of PART is to provide high-impact, high-quality materials, and experiences that resonate with practitioners and organizations The following section describes the approach that PART has employed in promoting EIP and outlines the core program components including the implementation challenges and successes related to each.

PART’S Evolving Conceptualization of EIP

Evidence-based practice (EBP) refers to a decision-making model in which practitioners competently use interventions that have been proven to be effective by rigorous research studies. This shift toward the use of research evidence in practice has marked an important turning point in a field in which practitioners have traditionally been separated from academic research and have implemented interventions based on traditional practices as opposed to what has been demonstrated to be effective within the research literature (Chaffin&Friedrich, 2004). An emphasis on research utilization by child welfare practitioners and agencies allows for the creation of informed research consumers who are able to critically appraise research outcomes and identify the actions that are required to promote better outcomes for children and families (Aarons&Palinkas, 2007). Child welfare practitioners have traditionally relied upon best practice literature and practice wisdom for selecting effective intervention models for clients and/or families (Kessler, Gira,&Poetner, 2005). The limitation of best practice models is that they often lack the systematic integration of research evidence. Traditional approaches to EBP, however, have also been critiqued because they exclude practitioner knowledge and client perspectives and minimize the contextualization of the research evidence (Hall, 2008). Recent research has put more emphasis on these issues and investigated various factors critical in the implementation of EBP (Manuel et al., 2009; Proctor, 2004).

PART adopted the more inclusive EIP approach that had been developed by RiP in their work on knowledge dissemination in the United Kingdom (Barratt&Hodson, 2006). The concept of EIP, that includes a broader range of research evidence and professional and client knowledge, has expanded the typical approach to EBP and recognized the importance of organizational and cultural contexts (Chaffin&Friedrich, 2004; Dill&Shera, 2009; Petch, 2009). EIP has been put forward as a way of promoting future research initiatives, improving links to policy, and creating an organizational learning culture that supports critical thinking and practice that is firmly rooted in sound evidence but also grounded in the realities of practice (Trocmé, Belanger,&Roy, 2008). The goal of the PART program, and of other knowledge utilization initiatives, is to move the fields of social work and child welfare to embrace a strong emphasis on EIP that involves critical analysis and contextualization to complex case situations (Ferguson, 2001; Kessler et al., 2005).

Petr’s (2009) Multidimensional Evidence-Based Practice Model is congruent with the concept of EIP and includes knowledge from research and from the perspectives of different stakeholder groups, such as consumers/clients and professionals. This model, which is much more congruent with the values and practice of social work, includes systematic criteria for the assessment of knowledge from all three perspectives; research, professional practice, and the views of service users. PART’s member agencies have grown in their respect and understanding of EIP and, in so doing, have become more critically reflective of existing knowledge. They increasingly seek to complement this knowledge by incorporating the wisdom of professionals and the voices of vulnerable children and families.

Hall (2008) echoes Petr’s (2009) conceptualization by exploring how EBP is constructed within the real world of practice. The limitations of EBP as outlined by Hall include a linear construction of problems that in the practice world are not always so “black and white.” He challenges researchers to take into account the complexities of the context in which practice-based issues are located. They concur on the need to contextualize research and to include the practice and consumer perspectives that are so often excluded in traditional evidence-based approaches.

PART’s current conceptualization of multidimensional EIP embraces a systematic appreciation of the professional and consumer practice literatures and values professional and client perspectives. Embracing published, peer-reviewed literature is essential but not sufficient for improving practice in child welfare. Professional and client knowledge must be included in our emerging and deepening understanding of EIP. PART has embraced this notion of multidimensional EIP by promoting the belief that different “ways of knowing” contribute alternative sources of valuable information (Dill&Shera, 2009) for improving practice. PART intends to remain on the leading edge of knowledge translation by designing program components that have the potential to have a high impact on child welfare professionals. The PART program is composed of several key components that include large-scale conferences, webinars, literature reviews, an electronic library, agency-based representatives, and a website portal that allows access to PART materials.

Conferences

The cornerstone of the PART program is the use of conferences as a way of disseminating critical research findings. These large-scale conferences include presentations by distinguished academics and policy makers on child welfare topics. These events are professionally produced learning opportunities that are provided to all member agencies. The events are simultaneously webcast so that member agencies located at a distance from the event can attend virtually. Discussion questions can be posed to the presenters via the Internet, allowing remote participants to interact with the presenters in real time. On-site and virtual participation exceeds over 400 participants per event. The split between on-site attendance and virtual participation runs about 50% on-site and 50% via webcast with increasingly more organizations moving toward the use of this innovative technology.

These conferences are events that have brought academics and practitioners together to dialogue about challenging practice-related issues. Topics included kinship care, outcomes for children in care, research related to siblings, substance misuse, linking evidence to child welfare supervision, and improving educational outcomes for children and youth in care. All of the conferences are focused on substantive practice-related issues and the topics are chosen by child welfare practitioners. In November 2009, the economic downturn had an impact, and many agencies failed to send staff members to the event because of the accommodation and travel-related costs. The PART Director heard from many agency representatives that although the events represent a significant turning point for the integration of research into practice, there remained a need to find a way to disseminate research findings without increasing the financial burden of membership by adding additional training costs. This implementation challenge led to the development of a webinar series.

Webinars

During a webinar, PowerPoint slides are shown on each participant’s computer screen, while the audio portion of the presentation is broadcast via telephone speaker phones or the internet (Voice Over IP). This innovative use of technology allows PART to engage world-renowned academics to present on substantive topics right from the comfort of their offices. Webinars allow hundreds of practitioners to engage with the presentation from their agency, without leaving their desks. These webinars are held on a monthly basis. Agencies have reported a good, enthusiastic level of participation with these events because the sessions are only one hour and do not require long periods of time away from practice. This limited time frame and easy accessibility provides staff members the opportunity to engage in a valuable learning opportunity that can be easily incorporated into a busy child welfare professional’s schedule. The uptake for these sessions has been significant. Since November 2009 over 5,000 staff members have participated in these sessions and the participation rate continues to grow. Many organizations invite partner agencies to observe and watch the sessions with child welfare staff members. At one agency, they invited children’s mental health partners to dialogue about the issues emerging from the webinars:

We have identified the webinar series as an opportunity for collaboration with key partners such as children’s mental health, youth justice and our foster parents. Response from our community partners has been positive. As an outcome of this opportunity to share this resource with our partners we anticipate increased local networking and further collaboration in our day-to-day work as well as at our community planning tables (Director of Service, personal communication).

One of the implementation challenges related to this program area is the need to use low-cost technology that is user-friendly. We have developed a system that ensures that academics presenting are comfortable with the technology and participants receive high-quality audio. Academics who are comfortable using a webcam are encouraged to do so as a further method of enhancing their overall engagement with their virtual audience. As PART continues to have success with this method of disseminating information, future webinar sessions will focus on different substantive areas of practice that will include teamwork, supervision, community outreach, and other specialized areas of practice such as domestic violence, substance misuse, and children’s mental health.

Literature Reviews

Another important service that PART provides to practitioners is literature reviews, called “PARTicles.” These literature reviews provide practitioners with access to up-to-date research on substantive topic areas. PARTicles are then used by child welfare practitioners during case conferences, individual supervision, and team meetings. One pioneering organization posts the PARTicles in high traffic areas that are visible to staff members on a daily basis. As PART publishes new editions of PARTicles, these publications are then rotated to provide new and emerging information to staff members. Most agencies have embraced the dissemination of research materials in innovative ways that include web-links to PARTicles. A team of graduate students from the local university are hired on a part-time basis to write these reviews. These reviews are expected to capture the consumer, practitioner, and academic perspectives. This multidimensional perspective provides front-line practitioners with a richer understanding of the complex issues being reviewed in the literature.

Guidebooks

Drawing on research in the area of clinical supervision in child welfare practice (Bogo&Dill, 2008; Dill&Bogo, 2009), the PART program has worked with a team of agency partners to build a Guidebook of resources for child welfare supervisors. This Guidebook, titled Broadening Horizons: Linking Evidence-Informed Practice to Child Welfare Supervision (2011) includes video clips and other resources for integrating research into daily practice. This material will be interactive and accessible on the PART website as an e-learning module. The publication has been enhanced with several videos that explore the use of evidence in supervision from the perspective of front-line practitioners, supervisors, senior managers, and researchers. A professional videographer was hired to film and capture essential messages that are further reinforced in the publication.

The process of developing this resource has created a parallel process of layered learning for those participating in the evolution of this first large-scale PART publication. The creation of the Guidebook appeared to resonate with the needs of the field, but when we ventured into the development of the product, we quickly realized that the foundational learning related to social work supervision was missing. Hence, the publication had to include a comprehensive framework for supervision, then move “up the ladder” so to speak, and increase middle managers' capacity to promote a more evidence-informed approach to practice. The development of this Guidebook is similar to the Change Project framework that has been pioneered by the RiP program (Bowyer&Moore, in press).

Website

A substantial component of the PART model is the website portal. This colourful, user-friendly website provides practitioners with ready access to all of the archived learning event materials and PARTicles. All the archived webinar and webcast material can be accessed through the website portal for review at any time. Supervisors have been encouraged to use this material in team meetings and retreat days. The beauty of this technology is that all the materials can be accessed on-demand by busy child welfare professionals and caregiver populations such as foster parents. The website is currently being redesigned based on significant feedback from practitioners that the site has become increasingly more difficult to navigate. The website must be easily accessible or practitioners will cease using these important materials. The website is linked to our sister organization RiP, and materials are shared between the two organizations, providing a richer array of resources between Canada and the United Kingdom.

Link PARTners (LPs)

Perhaps the most important ingredient for success with the PART model has been the creation of LPs—representatives from the member agencies who work in partnership with the PART program. LPs work with various levels of their member organization to create and promote practice change related to EIP. Each organization that joins PART must assign an individual Link PARTner to work with the PART program. All of these individuals take on the Link PARTner role in addition to their normal duties and assignments. The challenge for PART is to keep these agency representatives engaged and interested in the ongoing development of EIP and best practice initiatives. PART works with the LPs to create program materials and provide feedback on how to engage practitioners in the ongoing development of the program design. The role of the LPs is invaluable in the overall success of PART. They are, in fact, the ambassadors for creating organizational change within child welfare organizations. These individuals bring new research information back to the organizations through mechanisms such as distribution via staff list serves; discussions at staff and team meetings; and updates on agencies' intranet sites.

The organization and development of the Link PARTner system has created both successes and some of the greatest challenges in the implementation of the PART program. It has become clear that the Link PARTner is the essential “link-pin” in the promotion of a culture of EIP. This group of individuals is comprised of quality assurance staff, directors of service, and some mid-level supervisors. Interestingly, the individual person’s qualifications are not as essential as his or her commitment to creating and promoting a learning culture, and marketing PART through various venues such as committees, senior management meetings, and presentations at staff meetings. As previously mentioned LPs are asked to undertake their duties related to PART on top of their existing responsibilities. It is essential, therefore, that PART provide “perks” that promote continued engagement with this role within the organization. The program achieves this goal through various initiatives that includes (a) Link PARTner Retreats—once a year, PART provides an opportunity for all LPs to come together and generate ideas for promoting organizational and systemic change related to EIP; (b) Zone Based Meetings—LPs, in the most recent program evaluation, expressed a desire to move towards more face-to-face zoned-based meetings. These meetings provide LPs with the opportunity to network, share best practice initiatives, and focus on key elements of organizational change related to EIP.

E-Library

A more recent addition to the PART program design has been the implementation of an e-library system. By purchasing access to the SocIndex database, child welfare practitioners have access to over 800 journals, conference papers, and abstracts. Giving child welfare practitioners access to electronic publications empowers these professionals to consume evidence in a way that was not possible in previous years. PART has acquired rights to the Children and Youth Services Review journal. This electronic holding significantly increases the accessibility of current evidence-based materials to all member agencies. The plan is to move toward downloading key social work/child welfare texts and providing a wider array of electronic publications for use by all member organizations.

EIP Teams

At one mid-size child welfare organization, they chose to move toward the implementation of an EIP team. This team includes representatives from all levels of the organization including senior managers, supervisors, and foster parents. The purpose of this team is to facilitate the organization’s engagement with PART and other related EIP materials. The team has already committed to facilitating discussion sessions following every webinar session. The ultimate goal of the team is to bolster the organization’s integration of EIP into the daily complexity of child welfare decision making. They created an organizational “buzz” when they launched a PART awareness campaign that included a scavenger hunt on the website, a “web-fest” of archived webinar materials, and the dissemination of PARTicles on a daily basis. These types of events ensure “buy-in” at all levels of the organization. A number of other child welfare organizations have since replicated this model.

Critical Factors in Implementation

Various factors have created the synergy for PART’s initial organizational and systemic successes in Ontario. From PART’s inception, the executive directors of several child welfare organizations have facilitated and led the development and creation of this innovative program. Through their leadership, the model came to fruition and was embraced by its members. Without their support in the initial phase of the program design and implementation, PART would not have become a reality. These executive directors continue to support the development and sustainability of the program through their ongoing participation in the governance structure.

Recently PART hosted an international conference on the implementation of EIP during which papers from many different countries were presented. These papers have been compiled in an edited book (Dill&Shera, 2009). The learning that was gained from this event, including the creation of a number on international collaborative projects, will influence PART’s strategic directions for years to come. These international projects provide PART with access to and collaboration with international experts in the area of EIP.

From the beginning, PART has been challenged to make do with limited resources. This lack of staff and financial resources did not impede the program’s ability to disseminate high-quality materials for minimal costs. PART has a revenue base of $530,000.00 (CAN) per annum. These funds pay for all aspects of program delivery, staffing resources, and other overhead costs. Another continuing challenge is to develop a staffing model that is robust and adaptive to the requirements of the field, but not so large that we lose our capacity to be nimble, and responsive to the immediate needs of member agencies. The economic downturn has resulted in yearly debates about member agency retention and results in challenges related to short-term and long-term planning as it relates to staffing costs. Our ability to deliver these creative and important research messages to PART members in a timely and cost-effective manner has been a critical factor in its success to date. Other notable components of the program include its unique and attractive logo and branding and the use of newsletters to keep members and stakeholders informed of new approaches to the implementation of EIP.

The Need for Research on Implementation, Outcomes, and Impact

As described in the previous sections, PART has had both its successes and it challenges in the process of implementation. The literature on implementation is growing and has been very instructive in responding to these challenges. Austin and Claassen (2008) conducted a comprehensive review of literature addressing the organizational factors needed to introduce EBP into human service agencies and concluded that “EBP requires special attention to the processes of organizational change, the understanding of organizational culture, and the specialized expertise to promote the successful dissemination and utilization of research” (p. 274). These findings are also echoed in the analysis done by RiP and led to the development of two excellent resources: Firm Foundations: A Practical Guide to Organizational Support for the Use of Research Evidence (Barratt&Hodson, 2006) and Leading Evidence-Informed Practice: A Handbook (Hodson&Cooke, 2007). Austin and Claassen (2008) argue that the central components of organizational culture that promotes EBP are leadership, the involvement of stakeholders at all levels, the nurturing of cohesive teams, organizational resources, and the readiness to become a learning organization. A qualitative investigation of the implementation of supported employment and integrated dual diagnosis treatment in Kansas found that instituting expectations; upper level championing of EBP; making supportive structural and policy changes; creating intra-agency synergy (through leadership teams, work teams, and supervision); systematic use of information to monitor and evaluate fidelity and outcomes of interventions; and the provision of training and consultation were critical to the successful implementation of EBPs (Rapp et al., 2008).

Two critical research questions in this important area of implementing EIP: (a) how do we know if practitioners and/or programs have achieved a competent level of evidence utilization and (b) if they have, does this level of competence achieve the desired outcomes for children and their families? Law and McDermid (2008) argue that practitioners that are competent in EBP have skills for appraising and applying evidence; extensive knowledge of available evidence; consistently use evidence in their practice; and possess positive attitudes toward an evidence-based approach. Aarons (2004) developed the Evidence-Based Practice Attitude Scale (EBPAS) and concludes that attitudes toward adoption of EBP can be reliably measured and vary in relation to individual differences and service context. More recent research by Nelson and Steele (2007) found that practitioner training, the perceived openness of the clinical setting toward EBP’s, and the practitioner’s attitudes toward treatment research were significant predictors of self-reported EBP use. Other conditions that have been found critical to developing practitioner competence are personal and organizational readiness, personal and organizational expectations, defined timelines, and the availability of support (Law&McDermid, 2008). Horwitz, Chamberlain, Landsverk, and Mullican (2010), with reference to the field of child welfare, identify the continued inability we experience in attempting to implement EPB in routine service settings. They also contend that in comparison to the fields of health and mental health, child welfare systems have only recently had access to EPB information (Austin, 2010), and are only beginning to understand and become comfortable with the EBP implementation process. Chaskin (2008) highlights the importance and utility of program evaluation, particularly formative evaluation and the engagement of multiple stakeholders, in the implementation of EPB.

Program Evaluation of PART

In spring 2010, PART launched a formative program evaluation strategy to measure the progress and impact of PART’s activities on child welfare practice in Ontario. This evaluation strategy focused on three key areas: (a) the responsiveness of PART to practitioners and organizations, (b) satisfaction with the PART model and its components, and (c) engagement in EIP. As an organization dedicated to promoting EIP, PART is committed to modeling how evidence can be integrated to guide program development.

Method

Design of the Evaluation

The PART Council mandated and monitored the design and implementation of the evaluation.

The evaluation included the use of three major methods of data collection: analysis of secondary data that had been systematically collected after learning events and webinars; on-line surveys of all constituent populations (Practitioners, Supervisors, LPs, and Executive Directors) using Survey Monkey; and a series of focus groups with agency partners (LPs).

Ethical Considerations

Participation of respondents in the evaluation was voluntary and anonymous. The data collection process ensured that neither respondents nor agencies were identifiable. Although not part of a human subject ethics review panel at a university level, the ethics and research design were vetted and approved by PART’s governing Council comprised of child welfare leaders from across the province. PART staff members did not facilitate the focus group sessions. External consultants were recruited to facilitate these sessions and act as note-keepers. PART staff members were removed from the facilitation process in order to minimize bias. NVivo8 was used to analyze the qualitative data from the questionnaires and the focus groups.

Recruitment Strategies and Timelines

Recruiting busy child welfare staff members to participate in an evaluation can be challenging. Survey links for the supervisors and practitioners were distributed and followed up by the LPs and the Link PARTner and Executive Director surveys were sent out from the PART office. The surveys required several follow-up e-mails and phone calls to LPs to encourage them to increase the response rate within their organizations. The consultant-facilitated focus groups took place during the annual Link PARTner retreat. The entire data collection process was completed over a 4-month period of time.

Evaluation Results

LPs

This group of organizational participants is comprised of agency representatives from each of the 37 member agencies. This particular group provided the highest response rate at 97%. The majority of respondents were female (71%), 58% had been the assigned Link PARTner for over 2 years, and 48% worked in a mid-size organization of 100–299 staff. The individual’s role within the organization varied including: Director of Service (26%), Quality Assurance (26%), and middle manager (22%). This group was provided with two methods for providing feedback to influence the ongoing design and strategic plan for PART, focus groups, and an on-line survey.

At the annual retreat for LPs, four focus group sessions were facilitated. A strong recommendation that emerged from the focus group sessions and the survey was the need for more support for their efforts to influence organizational change and culture as it relates to EIP in their respective organizations. This group also articulated the need to build a stronger bridge between the research messages evolving from PART events and their utilization in day-to-day practice.

Other themes that emerged from their observations included (a) staff members awareness of EIP has grown; (b) there was an increase in the practical application of research messages into practice through staff discussion, case planning, and written assessments; (c) the use of technology (such as webinars) has become beneficial in disseminating key messages; and (d) PART has had an impact in promoting the critical necessity of integrating evidence into practice.

Front-Line Practitioners

A total of 560 front-line practitioners responded to the survey (13% response rate), 88% were female and 66% had worked in their organization for less than 10 years. The survey results pinpointed that many of the respondents had little or no idea what the PART program was or how to access the website—the portal into all components of the program.

The feedback from those that were able to access the website included the following key points: (a) PART information sources are easily accessible; (b) the use of technology has made it easier to access research messages; (c) many practitioners reported that lack of time was the most significant barrier toward integrating research into practice. The results from this group reinforced our concern that awareness and use of PART and the implementation of EIP was not as significant as we would like it to be for front-line practitioners. It is important to note however that there is a significant level of interest in, and support of, EIP. Front-line practitioners reported that they were supportive (43.6%) or very supportive (41.2%) of EIP.

The qualitative responses from practitioners also highlight their commitment to the crucial need to implement interventions that are known to work with children and families. Many practitioners commented on the need to move from a subjective to a more objective method of practice that includes the integration of evidence into practice. These findings bode well for the future development of EIP in the child welfare system in Ontario.

Managers/Supervisors

A total of 192 middle managers responded to their survey (a 51% response rate). The majority of respondents were female (77%) and had worked in their respective organizations for over 10 years (57%). Half of the respondents (50.5%) worked in mid-size organizations of 100–299 staff members. A wide range of themes emerged from the responses of this group of respondents but perhaps what was most striking was the group’s overall understanding of, and engagement with, EIP. This group highlighted a number of barriers with regards to implementation of EIP that included (a) lack of time by front-line practitioners to access materials and link them to practice; (b) the standards-driven nature of the work does not allow for critical thinking and reflective practice; and (c) some staff members are resistant to change and that can include dismissing the merits of using evidence in practice.

They reported “some increase” (50.5%) or “significant increase” (36.4%) in their awareness and understanding of EIP. They observed, however, that only 26.2% of front-line workers “frequently utilize” evidence in their practice. This group clearly articulated their commitment to the use of EIP but the challenge they found most difficult was to engage front-line practitioners in the systematic use of evidence in daily practice.

Executive Director Survey

A 70% response rate was obtained from the executive director group. Of this group, 70% were male, 32% had been an Executive Director for more than 10 years, and 48% worked in mid-size organizations of 100–299 staff members. The themes that emerged from this group included (a) the vast majority of Executive Directors believed that there was an increased organizational awareness of EIP; (b) PART had provided front-line practitioners with an opportunity to access practical materials that support their complex work; (c) PART and EIP encourage critical thinking and moving beyond the “status quo” of daily work and practice; and (d) the use of web conferences promotes accessibility for all member agencies across the province.

When asked how PART could support the work of child welfare leaders in the province, this group responded with the desire for PART to provide evidence regarding organizational strategies to promote organizational change to further the implementation of EIP. This group suggested that webinars be offered that focus on effective leadership within the context of organizational and systemic change (Benton&Austin, 2010).

Cross-Group Comparative Results

One of the challenges of conducting a large-scale program evaluation is the need to tailor the survey to the needs of each stakeholder group. The front-line workers had a different understanding of, and involvement with, PART as compared to the LPs, hence each survey was designed to be relevant to the respective groups. The use of cross-tabulations and between-group analyses was limited, given the unique nature of each survey.

Responsiveness

LPs rated PART staff as very responsive (64.5%), which could be attributed to their more frequent interaction with these staff. Executive Directors rated staff as responsive (66.7%) and very responsive (29.2%). Managers/supervisors rated staff as primarily responsive (32.6%) and very responsive (25.8%). The PART staff group consists of three full-time staff members. The results indicate that with minimal staffing resources, the program was able to respond to the majority of the needs from the various stakeholder groups. Front-line practitioners were not asked about PART’s level of responsiveness because it was seen as unlikely that this group would have much individual contact with PART staff members.

Satisfaction

LPs, front-line practitioners, and supervisors rated webinars, learning events, and PARTicles (literature reviews) as the three program components that they were most satisfied with. This information indicates that across all three populations, these program components were perceived as being most useful and should receive utmost attention in terms of future improvements in the quality and dissemination of materials. Executive Directors are more removed from the specific program components and did not rate the top three “picks” but instead focused their interests on the ability of PART and themselves to promote EIP through organizational change.

Implementation of EIP

Three of the stakeholder groups: LPs, front-line practitioners, and supervisors were asked to comment on how frequently practitioners refer to evidence in their day-to-day practice. Half (50.4%) of the practitioners claimed that they frequently referred to evidence in their practice. Supervisors and LPs, however, indicated that only about a quarter of practitioners referred to evidence on a frequent basis. More in-depth interviewing would be required to determine the accuracy of the practitioners' self-assessments regarding the frequency of use of evidence in practice (Nelson&Steele, 2007). In the qualitative comments provided by practitioners in the survey they indicated support and enthusiasm for using EIP but identified the lack of time to access evidence and to reflect on their work in a more critical manner as the main barrier. Organizations that infused evidence in supervision and teamwork were seen as being more successful in the integration of EIP.

Impact of PART on Practitioner Awareness and Understanding of EIP

The same three groups were asked to provide their perceptions regarding the degree to which PART had made an impact in terms of increasing child welfare practitioners' awareness and understanding of EIP. All three groups agreed that PART had made “some impact” on practitioners' awareness and understanding of EIP: LPs (75%), supervisors (63.6%), and practitioners (43%). These differences can in part be explained by the degree of involvement that each of the groups have had in promoting EIP within their organizations. All three groups reported that there had been a “significant impact” for 15% of their staff.

The qualitative comments yielded interesting results. LPs stated that PART has helped organizations begin the conversation with respect to EIP. The LP’s reported that they had seen how practitioners and supervisors were integrating the key research messages within the context of daily supervision, case conferences and written assessments. PART was viewed by LPs as helping organizations understand the critical importance of integrating evidence into practice. Supervisors echoed this sentiment in their belief that PART has provided easily accessible information in the form of up-to-date research, from a variety of sources. Front-line practitioners stated that PART promoted the importance and application of evidence informed case decision making. The Executive Directors indicated that PART had significantly increased organizational awareness of EIP. They also observed that PART had effectively promoted critical thinking and analysis that would move practitioners forward in their analysis of complex cases.

Implications and Future Directions

Creating change within the context of a large system of child welfare agencies is challenging. Large-scale change requires the commitment and dedication of senior leaders and engagement at the supervisory and front-line levels. Creating a lasting momentum for change requires a concerted effort on the part of all child welfare leaders and practitioners. PART has been in place for 3 years and to date has only had a modest impact in terms of actually changing practice behaviors. It is encouraging to note, however, that the level of interest and support of EIP by child welfare practitioners is significantly high.

In reality, this type of large-scale change requires both extensive participation and a significant amount of time. Based on the results of this recent evaluation, a new 3-year strategic plan for PART has been developed. In addition to continuing its core services, this plan puts a major emphasis on agency-based capacity building through the endorsement of agency Executive Directors, the use of PART Committees, agency-based PART strategic plans, and a guidebook for supervisors. Other system-wide initiatives include (a) the development of zone meetings with LPs to encourage networking and sharing of best practices in implementation; (b) the redesigning of webinars to put more focus on application in practice; (c) participation in the development of a multiagency agenda for priorities in child welfare research; (d) and the development of a broad-based advisory committee. A more extensive system of ongoing evaluation and a set of performance indicators have been developed for implementation over the next 3 years. We plan to move toward more systematic measurement of EIP competency through the use of more standardized tools and collaborative/comparative work with child welfare systems in other jurisdictions.

PART also recognizes the need for ongoing research and evaluation of the implementation and integration of EIP at both an organizational and a systemic level. PART in collaboration with the children’s mental health sector, sponsored an international conference that addressed the emerging conceptualization of EIP. At the 2010 conference, a unique initiative entitled, “Special Projects” was developed. The goal of the special projects was to provide participants with the opportunity to collaborate and work together on these international EIP projects. The Special Projects will culminate in presentations at the next biennial international event scheduled to be held in the spring of 2012 in Ireland. These projects are sponsored and facilitated by PART. They provide our program team with access to leading-edge knowledge and application of evidence-informed research that extends beyond our own geographical boundary of Ontario, Canada. There are a total of six special international research projects that include:

The Global Implementation Initiative: This conference is scheduled for 2011 in Washington, District of Columbia, and is sponsored by the National Research Implementation Network. The group has created an international consortium of members who are interested in developing and sharing ideas related to furthering the science of implementation. More specialized discussion groups will address the challenges of implementation at a systemic and organizational level.

Engagement with Children, Youth and Families: This project addresses the consumer perspective on EIP. The work of the group includes conducting a literature review that explores the ambiguous concept of engagement and related terms from various disciplines and identifies effective strategies for engaging consumers in improving practice and participating in research. The group also plans to explore and document “real-world” mechanisms for engaging consumers in evidence-informed initiatives.

Evidence-Informed Decision Making: This is a project that seeks to embed the use of research in key child welfare related decision-making processes. Decision making in mandated child protection processes in many countries occurs at similar points (e.g., at intake, assessment, investigation, child removal, and placement). This project seeks to identify and map the evidence that can be used to inform these decision-making points.

Framework for System-Level Decision Making to Achieve an Outcome-Driven System: This is a project that is an extension of models presented by Goldman and her colleagues (Goldman, Hodges, Kanary,&Wotring, in press). The workgroup’s goal is to develop a decision-making framework for achieving positive outcomes for youth and families that is relevant to child welfare leaders in a number of countries. This is a project that focuses on the macro and system-wide application of EIP.

Frameworks and Tools for Measuring Knowledge Uptake and Impact: This project is focusing on increasing the global understanding and measurement of the impact of knowledge utilization activities and their ultimate outcomes for children and families across both child welfare and children’s mental health settings.

Engaging Front-Line Practitioners in EIP: The intent of this project is to complete a literature review on what we currently know about factors that promote the engagement of front-line practitioners in EIP. Out of this review, an agenda for future research will be developed by the various stakeholders participating in this group.

A project manager has been hired, and this person coordinates monthly conference calls to ensure that the work of these international special projects is moving forward. The research and development that is emerging from these groups is well underway and should result in significant new findings and innovative collaborative research initiatives that are sustained and driven by experts in the field of evidence-informed practice. These continuing international linkages are invaluable in further developing PART’s expertise in evidence-informed practice.

Footnotes

Notes

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Aarons

G. A.

(2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health Services Research, 6, 61–74.

Aarons

Palinkas

(2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health, 34, 411–419.

Austin

M. J.

(Ed.). (2010). Evidence for child welfare practice. New York, NY: Routledge.

Austin

M. J.

Claassen

(2008). Implementing evidence based practices in human services organizations: Preliminary lessons from the front-lines. Journal of Evidence Based Social Work, 5, 271–293.

Barratt

Hodson

(2006). Firm foundations: A practical guide to organisational support for the use of research evidence. Totnes, UK: Research in Practice.

Benton

A. D.

Austin

M. J.

(2010). Managing non-profit mergers: The challenges facing human service organizations. Administration in Social Work, 34, 458–479.

Bogo

Dill

(2008). Walking the tightrope: Using power and authority in child welfare supervision. Child Welfare, 87, 141–157.

Bowyer

Moore

(in press). Interactive strategies in evidence-informed practice: Working in collaboration with a multi-professional children’s workforce. In Dill

Shera

(Eds.), Implementing evidence-informed practice: International perspectives. Toronto: Canadian Scholars Press.

Chaffin

Friedrich

(2004) Evidence-based treatments in child abuse and neglect. Children and Youth Services Review, 26, 1097–1113.

10.

Chaskin

R. J.

(2008). Dissemination and impact: Issues, lessons, and future directions. In Chaskin

R. J.

Rosenfeld

J. M.

(Eds.), Research for action: Cross-national perspectives on connecting knowledge, policy, and practice for children (pp. 131–157). New York, NY: Oxford University Press.

11.

Dill

Bogo

(2009). Moving beyond the administrative: Supervisors' perspectives on clinical supervision in child welfare. Journal of Public Child Welfare, 3, 87–105.

12.

Dill

Shera

(2009). Designing for success: The development of a child welfare research utilization initiative. Evidence and Policy: A Journal of Research and Debate, 5, 155–166.

13.

Ferguson

(2001). Promoting child protection, welfare and healing: The case for developing best practice. Child and Family Social Work, 6, 1–12.

14.

Goldman

Hodges

Kanary

Wotring

(in press). Achieving an outcomes-driven system: Critical decision points for leaders. In Dill

Shera

(Eds.), Implementing evidence-informed practice: International perspectives. Toronto: Canadian Scholars Press.

15.

Hall

J. C.

(2008). A practitioner’s application and deconstruction of evidence-based practice. Families in Society, 89, 385–393.

16.

Hodson

Cooke

(2007). Leading evidence-informed practice: A handbook. Dartington, England: Research in Practice.

17.

Horwitz

S. M.

Chamberlain

Landsverk

Mullican

(2010). Improving the mental health of children in child welfare through the implementation of evidence-based parenting interventions. Administration & Policy in Mental Health, 37, 27–39.

18.

Kessler

Gira

Poetner

(2005). Moving best practice to evidence-based practice in child welfare. Families in Society: The Journal of Contemporary Human Services, 86, 244–250.

19.

Law

McDermid

(2008). Evidence-based rehabilitation: A guide to practice (2nd ed.). Thorofare, NJ: Slack Incorporated.

20.

Manuel

J. I.

Mullen

E. J.

Fang

Bellanmy

J. L.

Bledsoe

S. E.

(2009). Preparing social work practitioners to use evidence-based practice: A comparison of experiences from an implementation project. Research on Social Work Practice, 19, 613–627.

21.

Nelson

T. D.

Steele

R. G.

(2007). Predictors of practitioner self-reported use of evidence-based practices: Practitioner training, clinical setting, and attitudes toward research. Administration & Policy in Mental Health & Mental Health Services Research, 34, 319–330.

22.

Petch

(2009). Guest editorial. Evidence & Policy, 5, 117–126.

23.

Petr

C. G.

(2009). Multidimensional evidence based practice: Synthesizing knowledge, research, and values. New York, NY: Routledge.

24.

Proctor

E. K.

(2004). Leverage points for the implementation of evidence-based practice. Brief Treatment and Crisis Intervention, 4, 227–242.

25.

Rapp

C. A.

Etzel-Wisel

Marty

Coffman

Carlson

Asher

Callaghan

Whitley

(2008). Evidence-based practice implementation strategies: Results of a qualitative study. Community Mental Health Journal, 44, 213–224.

26.

Trocmé

Belanger

Roy

. (2008). Knowledge mobilization in child welfare. Retrieved November 23, 2010, from http://www.mcgill.ca/files/crcf/Knowledge_Mobilization_Chapter.pdf