Integrating Assessment and Evaluation Into Partnership Initiatives

Collaboration

In the last two decades, an increasing number of collaborative initiatives in the form of coalitions, partnerships, and alliances have been introduced to address a variety of community health and social conditions. These initiatives have led to a growing body of literature that describes the role of collaborative initiatives within communities (Bayne-Smith, Mizrahi, & Garcia, 2008; Ivery & Akstein-Kahan, 2010; Kegler & Swan, 2011; Krueter et al., 2000; Lasker, Weiss, & Miller, 2001; Wildridge et al., 2004; Zakocs & Edwards, 2006). Despite the abundance of publications on the development and implementation of partnership initiatives to address a variety of health and social issues, there is consensus that more rigorous empirical studies of collaboration are needed to identify best practices. Indeed, there is a great need for longitudinal measures of partnership functioning and an even greater need for validating measures of partnership functions (El Ansari & Weiss, 2006; Granner & Sharpe, 2004; Weiss, Anderson, & Lasker, 2002).

Although more robust studies and more precise measurement is needed to build knowledge in the field of collaboration research, it has been argued that multiple approaches are necessary to address this need (Scharlach & Sanchez, 2010). The complexity and nuanced nature of work with collaborations, however, makes empirical study in this arena challenging at best (Granner & Sharpe, 2004; Kubisch et al., 2010). Foster-Fishman and colleagues describe an integrative framework that identifies the following four elements of capacity to be maximized for partnerships to be effective, sustainable, and demonstrate the ability to evaluate their work: individual, relational, organizational, and programmatic capacity (Foster-Fishman, Berkowitz, Lounsbury, Jacobson, & Allen, 2001). This integrative framework, described in more detail in our discussion section, may serve as a useful conceptual model for evaluating the efforts and impact of partnerships while illustrating their complexity.

Despite the methodological challenges of evaluating effectiveness of community coalitions, there is strong evidence that incorporating both formative and summative evaluation techniques early in partnership development, and throughout implementation of its work, facilitates goal attainment (Krueter et al., 2000; Preskill, 2010; Zakocs & Edwards, 2006). Indeed, evaluation serves many purposes (Francisco, Schultz, Fawcett, 2004). Formative evaluation provides a feedback mechanism for strategic decision making about structure, process, and capacity building within the partnership, whereas summative evaluation documents results of partnership efforts by measuring short and long-term outcomes of the initiatives. Both types of evaluation are needed to inform internal as well as external stakeholders (Foster-Fishman et al., 2001).

In designing and implementing partnership evaluations, both internal and external stakeholders should be considered potential audiences for dissemination, thus informing which methodological techniques to adopt. El Ansari and Weiss (2006) argue that current research paradigms may not be a good fit for capturing the complexity of partnership approaches to community and systems-level outcomes. This supports the value of incorporating case examples within the field’s emerging theoretical frameworks, conceptual models, and typologies (e.g., see Gajda, 2004; Ivery, 2007; Kegler & Swan, 2011; Lasker, Weiss, & Miller, 2001; Mitchell & Shortell, 2000). In-depth case studies contribute substantially to the growing knowledge on collaboration as intervention because, as Jones, Crook, and Webb (2008) point out, “case analyses continue to add nuances and lessons that can be tested in the next case example and refined for those that follow” (p. 66). Participatory action research, where members of a community most affected by a particular issue are key players in the design and implementation of research and evaluation (Greenwood, 2012), also may be suitable for partnership evaluation because this approach incorporates the multiple perspectives of diverse partners inherent in collaborative interventions. These various approaches could build knowledge about interventions that is more seamlessly translated into practice because it has been vetted through the stakeholders who are essential in carrying out such interventions.

CPFOA

CPFOA, a national program of The RWJF from 2002 to 2010, awarded 15 CPs each US$900,000 to develop and implement a local strategic plan to improve lives of older adults through long-term supportive system improvements. Incorporating the principle that there is no one-size-fits-all model to meet the complex needs of older adults across communities, each CP was encouraged to develop a locally tailored approach to planning, implementing, and evaluating local systems change. Consequently, a variety of strategies for improving long-term supportive and service needs within each community were adopted (Bolda, Saucier, Maddox, Wetle, & Lowe, 2006). Characteristics of the CPs were diverse in terms of size, urban/rural/suburban location, region of the country, and organizational auspice of the fiscal agent. The membership of all CPs included older adults as well as health and social service providers from both the private and public sectors. Most CPs included atypical stakeholders in their membership such as utility company representatives, consultants, city planners, architecture firms, and public library employees, among others. Table 1 lists the name and location of each CP. Detailed information about each CP may be obtained at www.cpfoa.org. The overarching vision of the initiative, according to the National Program Office (NPO), was “To improve the long term care system by engaging community-based organizations, older adults, policy makers, and other stakeholders in determining how to best meet the needs of older adults, ensuring their independence and dignity.” ¹ Each community defined for itself how it would address the national program’s broad vision and how the community would evaluate its work.

OPEN IN VIEWER

Table 1.

Community Partnerships for Older Adults Grantees. ^a

City or region	Partnership name
Milwaukee, WI	Connecting Caring Communities
Ann Arbor, MI	Blueprint for Aging
Broome County, NY	Aging Futures
El Paso, TX	SALSA (Seniors Accessing Long-Term Care through Strategic Planning and Advocacy)
Boston, MA	Boston Partnership for Older Adults
La Grange, IL	Aging Well
Manchester, NH	Seniors Count
Atlanta, GA	Aging Atlanta
Rappahannock Rapidan Region, VA	Aging Together
San Francisco, CA	San Francisco Partnership for Community-Based Care & Support
Maui, HI	Aging with Aloha
Fremont, CA	Pathways to Positive Aging
Houston, TX	Care for Elders
Waynesville, NC	Haywood Community Partnership
Jacksonville, FL	Bridging Elder Care Networks of the Life: Act 2 Partnership

Note: ^aThe partnerships listed are those which received funding for both development and implementation of strategic initiatives.

The NPO at the Muskie School of Public Service, University of Southern Maine administered and directed the national initiative. Each partnership met the following minimum criteria to be chosen as a grantee (Bolda et al., 2006): partnership members represented a broad base of service providers that included traditional partners such as older adults, advocates, and other stakeholders from both the public and private sectors as well as less traditional partners such as members of the business community or other arenas; some evidence of prior success working as a partnership; an existing infrastructure for community-based services; and a designated group of core leaders. With these criteria met, each CP was required to implement its 4-year plan in a way that fit the characteristics of the community.

A Technical Assistance Consortium, initially convened by the Duke University Long-Term Care Program, provided technical assistance to the CPs in the following areas during implementation of their plans: communications, assessment and evaluation, leadership development, and fundraising. In addition to technical assistance, the CPs became what George Maddox referred to as “teaching and learning communities” whereby they were encouraged, as their work progressed, to share with each other their strategies and tactics undertaken as well as the lessons learned in the implementation process (see Bolda, Lowe, Maddox, & Patnaik, 2005).

Mathematica Policy Research (MPR) was commissioned by the RWJF to evaluate the national program as a whole, but not evaluate individual partnership initiatives. Therefore, each CP planned and implemented its own evaluation within the community, independently from the MPR evaluation strategy. Although each CP was required to submit a logic model and evaluation plan to the NPO to describe their strategies and projected process and outcomes, the design and methods of each CP varied. This created a challenge for the Mathematica evaluators of the national program which would lead to the national evaluation effort to focus less on narrowly articulated outcome measures and more on broader process and structural lessons (Menehan, 2011).This study complements the work of Mathematica in that it drills down to the evaluation strategies of each individual partnership.

The purpose of this article is to share lessons learned and build on the growing body of knowledge around community change evaluation practices involving multisite partnership initiatives. To that end, we examined how 15 CPFOA Partnerships approached evaluation of their individual efforts toward broader community change. As Claudia Coulton states in her response essay in the Aspen Institute report, Voices from the Field III, “multisite community change efforts have some of the greatest potential to be comparative” (Kubisch et al., 2010, p. 118).

Styles and Methods of Planning and Implementing Evaluation

Formative Evaluation (Partnership Self-Assessment)

Although there was no “top-down” requirement from funders for the CPs to engage in formative evaluation, the data revealed that partnerships used this approach as a way to examine internal partnership functions and determine whether the intended member benefits were being actualized. Each partnership approached formative evaluation differently according to its own community resources and needs. CPs generally did not use the technical term “formative evaluation” to describe the methods they used to examine intrapartnership characteristics and processes, but instead referred to terms such as “partnership assessment” or “partnership self-assessment” to describe these evaluation strategies. Though terminology may have differed, it was clear across the CPs that formative evaluation methods served the purpose of determining if (a) the organizational characteristics of the partnership met the needs of its membership, and (b) the process through which the CP functioned was the most effective means for facilitating the partnership’s work.

Often, formative evaluation was seen as a way for members to determine the value and assess the process of working together. It allowed CPs to identify strengths and potential areas for improvement in the organizational structure, governance, leadership, and management of the partnerships. It also served as a means to build relationships through partnership work. For example, one CP member shared,

This Partnership’s deeply inclusive process of data gathering and strategic planning used tools, venues, and languages to reach across many ethnic and underserved populations. This approach built new & stronger bridges, leadership and relationships able to leverage resources and solutions for their region’s frail elders.

Some CPs approached formative evaluation using more traditional, formal methods whereas others took informal approaches. The evaluation approach was generally driven by the organizational culture of the CP, whereas those with less formal structures used less formal approaches. More formal methods included surveys of individual partnership members (written, electronic, telephone, or in person), focused conversations with key informants, focus groups, or community dialogues with membership representatives. Less formal approaches included simple postactivity polls (written or electronic) or debriefing sessions which solicited reactions or feedback around an action or activity sponsored by a CP. Regardless of the formal versus informal methods and strategies used, the common purpose of formative evaluation across CPs was to assess the structure and process characteristics of the partnership. Examples of indicators (in question form) used by CPs to assess common structure and process characteristics are listed in Table 2.

OPEN IN VIEWER

Table 2.

Formative Evaluation Indicators Adopted by CPFOA Partnerships.

Indicator	Operational examples
Composition of the partnership (size, diversity, representation of membership)*	Always asking “who is missing” from the partnership and always asking whether representation from direct service providers and recipients are sufficiently represented.
Resources* (human, political)	If your partnership is planning to develop workgroups to implement several initiatives, are there enough members participating to ensure the same people are not responsible for the work of all of the groups?
Accountability and responsibility of staff, leaders, and individual partners	Are there clear expectations regarding the responsibilities of partnership staff, core leaders, and other members? Is there a consequence when someone does not meet expectations?
Mechanisms for internal and external communication*	How are partners informed about events, accomplishments, and other “goings on” of the partnership? Do they feel informed and in touch? How are nonmembers informed?
Mechanisms for soliciting participation and leadership	Is there a clear and satisfactory process for appointing formal leaders?
Perceived benefits of participation	Do partners feel the time and resources they are contributing result in benefits to their agencies and the community’s older adults?
Sustainability	Are the types and sources of funding for partnership sponsored activities growing? Do projects initially sponsored by the partnership continue after start-up without partnership support? Are funders supporting the core functions of the partnership (i.e., staffing) and not just the initiatives?
Cooperation among members*	Are organizations increasing their levels of collaboration with each other through shared resources, Memorandum of Understandings, or joint funding applications?
Centrality (awareness of the partnership)*	Is the Partnership the “go-to” entity for information, problem solving, leadership, or other resources? Are community members aware of the Partnership?
Consistency between planning and implementation*	Are activities and initiatives being carried out as expected? Why or why not? What aspects of the partnership structure are fostering or preventing the successful implementation of planned activities and initiatives?

Note: *Dimension that aligns with the Mitchell and Shortell (2000) typology of community health partnerships.

Data revealed that formative evaluation was an ongoing, iterative process to help members (and staff) facilitate the work required to successfully carry out initiatives and activities and make strategic changes as necessary. Because systems change and community-level impact cannot happen overnight, formative evaluation was used by CPs to measure their capacity to face current and future challenges, remain sustainable, and work collectively toward systems improvements. This approach sometimes required changes in courses of action or organizational structure. As one CP member stated,

Our structure changed based on what we learned [from the assessment] . . . now the board defines work direction. The work groups are organized to enact. They are more focused now and have more direction . . . people see results. This will strengthen the Partnership.

While utilizing formative evaluation for strategic decision making, it was clear from the data that evaluation planning was essential, which in itself is also strategic. A representative from one CP stressed the importance of using an evaluation plan to guide this work, stating, “I am still adamant about a strong evaluation plan. It serves as work plan . . . It’s our accountability plan and is our performance plan.”

Finally, respondents noted a critical linkage between formative and summative evaluation strategies in partnership work. According to the data, there was a common perception that if process characteristics such as governance and management of the partnerships were aligned with members’ needs, CPs would be viewed as effective, demonstrated by satisfaction among members, successful program outcomes, and sustainability. Some CPs used existing partnership assessment tools for this purpose. Reflecting on how formative evaluation was used to better align partnership process with member needs, a CP staff member described that their partnership used a “collaborative factors inventory to document satisfaction with the Partnership and make improvements where needed.” This CP learned that members needed more diverse communications methods for sharing information, so improvements were made to this partnership’s website to increase members’ online access of materials and information.

The NPO introduced to all CPs the Mitchell and Shortell (2000) typology of community health partnerships, which identifies seven indicators for examining partnerships. This typology, explicitly used by three CPs as a conceptual framework for evaluation, facilitated communication between all partnerships by offering a shared platform for teaching and learning from each other. For example, despite the variety of approaches to measure community awareness of their work, CPs were able to use the typology to collectively describe this concept as centrality and use this common language to share lessons. As shown in Table 2, among the formative evaluation indicators adopted in CP evaluations, six of them align with Mitchell and Shortell’s typology. Moreover, their typology links formative and summative evaluation in that the indicators integrate governance and management characteristics of partnerships to its overall effectiveness.

Challenges of Evaluation and Strategies for Overcoming Them

Evaluation work is challenging, particularly when it requires interorganizational collaboration. The challenges experienced by CPs and strategies for overcoming them will be discussed along the three emergent themes: limited resources, prioritizing objectives and activities, and gaining buy-in from stakeholders to support evaluation work.

Application of the Integrative Framework of Collaborative Capacity

The four elements of Foster-Fishman and colleagues’ integrative framework may be a useful lens to identify characteristics that may facilitate successful evaluation work. Table 3 illustrates how each element of collaborative capacity (member, relational, organizational, and programmatic) can be used to conceptually interpret the results of this study.

OPEN IN VIEWER

Table 3.

Collaborative Capacity Characteristics that Promote Successful Evaluation.*

Elements of collaborative capacity	Partnership characteristics for successful evaluations
Member capacity	CP members are taught evaluation skills. Evaluation methods undertaken by CPs are feasible and accessible so all members can participate in the work. CP members understand purpose and utility of evaluation.
Relational capacity	External resources in the community are tapped by the CP to assist with evaluation efforts, including “non-traditional” partners. CP identifies key audiences and stakeholders with whom to communicate evaluation results, then prioritizes essential activities accordingly.
Organizational capacity	A flexible organizational structure is adopted to integrate formative evaluation results into strategic, collective decision making. A teaching and learning model is supported both internally (within the CP) and externally (with other partnerships). Internal assets and resources are identified and capitalized (e.g., nontraditional data sources).
Programmatic capacity	Measurement indicators are prioritized based on feasibility (i.e., realizing the CP “cannot measure everything”). Less traditional or alternative data sources are used (e.g., the story bank). The CP believes that successful program outcomes are achieved when members have positive perceptions of the governance and management of the partnership.

Note: *This table applies Foster-Fishman and colleagues’ integrative framework of collaborative capacity to the findings of this study (see Foster-Fishman et al., 2001).

The first element of member capacity posits that members demonstrate skills and attitudes that facilitate collaborative evaluation work. The partnership also needs access to these competencies. For example, educating members around evaluation design and techniques will build current knowledge and skills around evaluation, thus building member capacity that the partnership will then have access to.

The second element of relational capacity values relationships both between members and with external networks. This may involve facilitation of trust-building between organizational representatives who traditionally have competed for limited resources in their community or may require educating members around communications skills for reaching out to new types of partners such as the media or other sectors. In this study, CPs that reached out to nontraditional partners for evaluation expertise exemplified relational capacity.

Third, the partnership demonstrates organizational capacity through effective leadership, policies and procedures, communication, resource acquisition, and the ability to self-correct as the CP as an organization evolves. Applying organizational capacity to evaluation work may involve creating an organizational structure that allows for strategic changes to partnership work based on formative evaluation findings.

The final element in the collaborative capacity framework posits that the coalition must demonstrate programmatic capacity to attain desired outcomes. In this study, an example of programmatic capacity is the successful prioritization of evaluation indicators and strategies, including utilization of less traditional data sources for evaluation work.

Overarching Lessons and Implications for Practice and Research

First, findings of this study suggest that both formative and summative approaches are essential in partnership evaluation. Mechanisms to understand partnership processes and functioning must be integrated into evaluation planning and implementation. Although this supports existing literature stating that effective evaluation planning and implementation are ongoing and iterative activities (Israel et al., 1998; Krueter et al., 2000), partnership agencies are faced with limited resources for research and evaluation and funders or regulatory bodies often cast evaluation in terms of outcomes (i.e., summative) research. In practice, however, the emphasis on outcomes overshadows formative evaluation strategies in an effort to fulfill funding requirements (Kubisch, et al., 2010; Mitchell & Shortell, 2000).

Program evaluation work becomes increasingly complex when the “program” being evaluated is not one organizational entity, but is a network of diverse players and stakeholders representing different organizational entities with various perspectives and perhaps different interests. To add to the complexity, CPs are comprised of individual community members (i.e., unpaid advocates or service recipients) as well as organizational representatives (i.e., paid professionals) from both the private and public sectors. In formative evaluation, this diverse membership creates challenges for consensus building around how to grow all elements of collaborative capacity (member, relational, organizational, and programmatic). In summative evaluation, the two predominant challenges that arise are generalizability or replicability, and determining causality. Implementation of an initiative that may have been successful in one community may not be successful in another unless it is tailored to meet its unique needs. Determining causality is an unattainable goal whereby it is not possible to isolate the work of a partnership from possible confounding variables to explain or predict changes in community indicators such as knowledge, attitudes, behaviors, or service access.

Second, promotion and development of knowledge and skills related to research and evaluation within the partnership may help support successful planning and implementation of evaluation work. The teaching and learning emphasis of the CPFOA program provided opportunities to engage various stakeholders in research knowledge-building, although partnerships differed in the degree to which they involved the staff and volunteers of partner organizations. The partnerships that trained their members in research principles and evaluation practices built member capacity and promoted shared responsibility (Foster-Fishman et al., 2001). These findings support existing literature suggesting that ownership on behalf of stakeholders is essential (Dluhy & Swartz, 2006; Mizrahi & Rosenthal, 2001). Involving stakeholders increases buy-in for evaluation and helps them maintain a sense of “knowing where this is going” and creates a culture of collective responsibility and one, according to Kubisch et al.(2010), “that values candid dialogue and analysis and embraces the idea of learning while doing” (p. xi).

Frontline workers like home care attendants or case managers have the capacity to bring direct client experience to inform evaluation approaches, whereas supervisors or managers may bring facilitation expertise to carry out planned evaluation strategies, and agency or community leaders may garner resources to support evaluation work. An essential player in partnership practice is one that brings these diverse stakeholders together. As Dluhy and Swartz (2006) point out, a neutral convener is generally required in partnership work. Those trained in community practice may bring specific competencies to this role of neutral convener and bring the ability to facilitate strategies for assessing process as well as measuring long-term community impact (Thompson, Minkler, Bell, Rose, & Butler, 2003). Interdisciplinary practitioners could be particularly effective at communicating evaluation findings both within partnerships and with outside stakeholders. The CPs had access to other communities in the CPFOA network for “teaching and learning” opportunities. Seeking information from other communities implementing partnership initiatives, however, was a challenge. The growth of social media may be an opportunity to overcome this challenge in the future.

Third, the results of this study point to the importance of flexibility and creativity in designing and implementing evaluation. This was demonstrated by the CPs use of evaluation measurement strategies not traditionally considered in program evaluation, and by flexible organizational structures that changed as necessary. Although traditional summative evaluation strategies were encouraged by the requirement of a logic model, the NPO created space for alternative research ideas by not mandating a standardized evaluation approach. Creativity was further encouraged by the participation of diverse stakeholders. This finding suggests that perhaps more integration of participatory research principles used in community-based participatory research would fit well in the partnership model (Stahl & Hahn, 2007). Although creative evaluation strategies may be useful in partnership evaluation, this creates a challenge for community practitioners and researchers who are often required by funding and regulatory bodies to complete traditional outcome-based evaluation. This challenge illustrates the need for funders to support evaluation approaches from a variety of research paradigms.

Partnership research is still a nascent field of study with a limited evidence base in the form of rigorous and robust systematic inquiry (Zakocs & Edwards, 2006; Zakocs & Guckenburg, 2007), particularly in the field of aging (Ivery & Akstein-Kahan, 2010; Warburton, Everingham, Cuthill, & Bartlett, 2008). Traditional research methods will not suffice in building our knowledge base given the complex contextual factors involved and the individuality of each local community. Clearly, better use of conceptual frameworks is needed, and the very nature of partnerships—their participation—may be best suited for more innovative research methods such as community-based participatory research, an approach not yet optimized by researchers (Stahl & Hahn, 2007). As the results of this study demonstrate, CPs creatively identified new sources of data and methods to evaluate their work.

To balance creativity and methodological rigor, guiding principles could be developed to assist communities with implementing and evaluating new initiatives through partnerships in a way that allows for individuality while maintaining the fidelity of a new initiative. These principles could be shared using innovative communication media such as blogs or wiki tools updated collectively by stakeholders to document a partnership’s accomplishments. Social media has also emerged as a venue for incorporating new evaluation methods into partnership work, while increasing numbers of older adults are using social media. These innovative dissemination examples require a new approach to evaluation which requires different technical skills to be embedded into the work of CPs and promotes shared responsibility among partnership members.

The fourth lesson suggests that leveraging resources from both within the partnership and outside of the partnership to build capacity for evaluation work is crucial. Evaluation work is not easy and requires additional resources not always available within the partnership. The CPs in this study tapped traditional and nontraditional resources such as private consultants, university-affiliated researchers, and a variety of stakeholders from the private sector in many fields. This may not occur without challenges as nontraditional partners often have different jargon, perceptions of the work to be accomplished and timeline expectations. When these challenges were overcome, stronger working relationships and increased relational capacity of the partnership was built. Indeed, multiple perspectives contributed by different stakeholders may foster easier translation between research, policy, and practice (Feldman, Nadash, & Gursen, 2001; Warburton, Everingham, Cuthill, & Bartlett, 2008).

Effective evaluation of partnership work may help practitioners and researchers tap collective community resources and improve long-term supportive systems. Moreover, partnership work begins to break down disciplinary and funding silos in a way that may lead to more integrated programming and improved access to services for older adults. Social workers, community psychologists, and public health educators may play a key role in the work of CPs, as they are trained to integrate both practice and research into their work and can serve as facilitators of collaborative work. Equally essential, however, are older adults, individuals working directly with older adults, and agency directors and other community leaders with decision-making capacity around home and community-based service delivery.

Finally, partnership evaluation may contribute to the currently inconclusive evidence of the cost-effectiveness of delivering home and community-based services for older adults and people with disabilities (Ivery & Akstein-Kahan, 2010; Warburton, Everingham, Cuthill, & Bartlett, 2008). With health care costs increasing uncontrollably, long-term supportive services for people at home and in community settings rather than in institutional settings may be more cost-effective, but the specific predictors of savings are not yet clear (Grabowski, 2006; Kaye, LaPlante, & Harrington, 2009). For older adults to maintain health and avoid unnecessary hospitalizations, they need better knowledge of and access to home and community-based services. Collaborative interventions such as CPFOA, designed to address the lack of information and other challenges faced by older adults in community settings may help identify specific methods for targeting cost savings.

The study’s methodological limitations do, however, limit the extent the findings may be generalized. A specific limitation is the retrospective approach, in which the data analyzed were not originally collected systematically with the current research questions in mind. A further limitation is that the experience of CPFOA partnerships may be unique, thus making the findings less generalizable.

Future research and more cohesive strategies for incorporating research into practice are clearly needed to identify the most effective means of evaluating partnership work. Better integration of conceptual frameworks, innovative research and dissemination strategies will contribute to our understanding of pathways between partnership structure, process, outcomes, and long-term impact on the lives of older adults.