Integrating Assessment and Evaluation Into Partnership Initiatives

Abstract

Communities nationwide have formed cross-sector partnerships to face the needs of an aging population, particularly for the purpose of improving long-term supportive service systems. Research is limited on how evaluation strategies are incorporated into partnership work, especially in the field of aging. This retrospective qualitative study reviewed administrative and key informant interview data to examine how 15 community partnerships (CPs) within the Community Partnerships for Older Adults (CPFOA) national program incorporated evaluation into their work. The four overarching lessons drawn from our inquiry suggest that effective CPs: (a) incorporate both formative and summative methods into evaluation, (b) use and develop the knowledge and skills of its members, (c) support flexible and creative evaluation methods and strategies, and (d) use internal and external resources to support evaluation efforts, particularly with nontraditional partners. There is a need for continued research to capture the methodological complexity of partnership evaluation.

Keywords

community partnership collaboration program evaluation home community-based services

Although communities have been forecasting the demographic imperative of our aging population for decades, it is now at their doorsteps. Some communities have developed partnerships, coalitions, or other collaborative initiatives designed to address the growing needs of older adults at both the local and state levels (Alley, Liebig, Pynoos, Banerjee, & Choi, 2007; Greenfield, 2012; Scharlach, 2009). Through its Community Partnerships for Older Adults (CPFOA) national program, the Robert Wood Johnson Foundation (RWJF) supported 15 community partnerships (CPs) in the development and implementation of strategic plans informed and designed by each individual community to improve long-term support systems that address the local needs of elders. Each community defined its own unique needs and created an individualized community plan under the auspice of a broad common vision of the national program, which was to improve the lives of older adults in communities by improving long-term care systems.

The growing body of knowledge around collaboration and partnership work toward systems change has concurred that ongoing evaluation of both process and outcomes is crucial (Israel, Schulz, Parker, & Becker, 1998; Krueter, Lezin, & Young, 2000). However, there is little empirical knowledge or systematic inquiry into how evaluation strategies may best be incorporated into the ongoing work of CPs, especially those partnerships engaging in broad systems-change efforts (Kubisch, Auspos, Brown, & Dewar, 2010; Wildridge, Childs, Cawthra, & Madge, 2004). At the same time, there is very limited literature on collaborative community change initiatives focused on aging and long-term supportive service systems (Blair & Minkler, 2009).

This study explores qualitatively, through program documentation and secondary key informant interview data, how 15 CPs supported by the CPFOA initiative incorporated evaluation strategies into their work. The aim of the study was to draw on the collective knowledge gained from the CPs to present lessons learned about integrating formative and summative evaluation strategies into the work of coalitions and partnerships specifically created to improve the lives of older adults and people with disabilities. Two research questions guided this retrospective qualitative inquiry. First, how did partnerships approach evaluation; what strategies, methods, or organizational styles were used by partnerships to plan and implement evaluation? Second, what challenges did the CPs face around evaluation and what strategies and resources were used to overcome them?

Background

Collaboration

In the last two decades, an increasing number of collaborative initiatives in the form of coalitions, partnerships, and alliances have been introduced to address a variety of community health and social conditions. These initiatives have led to a growing body of literature that describes the role of collaborative initiatives within communities (Bayne-Smith, Mizrahi, & Garcia, 2008; Ivery & Akstein-Kahan, 2010; Kegler & Swan, 2011; Krueter et al., 2000; Lasker, Weiss, & Miller, 2001; Wildridge et al., 2004; Zakocs & Edwards, 2006). Despite the abundance of publications on the development and implementation of partnership initiatives to address a variety of health and social issues, there is consensus that more rigorous empirical studies of collaboration are needed to identify best practices. Indeed, there is a great need for longitudinal measures of partnership functioning and an even greater need for validating measures of partnership functions (El Ansari & Weiss, 2006; Granner & Sharpe, 2004; Weiss, Anderson, & Lasker, 2002).

Although more robust studies and more precise measurement is needed to build knowledge in the field of collaboration research, it has been argued that multiple approaches are necessary to address this need (Scharlach & Sanchez, 2010). The complexity and nuanced nature of work with collaborations, however, makes empirical study in this arena challenging at best (Granner & Sharpe, 2004; Kubisch et al., 2010). Foster-Fishman and colleagues describe an integrative framework that identifies the following four elements of capacity to be maximized for partnerships to be effective, sustainable, and demonstrate the ability to evaluate their work: individual, relational, organizational, and programmatic capacity (Foster-Fishman, Berkowitz, Lounsbury, Jacobson, & Allen, 2001). This integrative framework, described in more detail in our discussion section, may serve as a useful conceptual model for evaluating the efforts and impact of partnerships while illustrating their complexity.

Despite the methodological challenges of evaluating effectiveness of community coalitions, there is strong evidence that incorporating both formative and summative evaluation techniques early in partnership development, and throughout implementation of its work, facilitates goal attainment (Krueter et al., 2000; Preskill, 2010; Zakocs & Edwards, 2006). Indeed, evaluation serves many purposes (Francisco, Schultz, Fawcett, 2004). Formative evaluation provides a feedback mechanism for strategic decision making about structure, process, and capacity building within the partnership, whereas summative evaluation documents results of partnership efforts by measuring short and long-term outcomes of the initiatives. Both types of evaluation are needed to inform internal as well as external stakeholders (Foster-Fishman et al., 2001).

In designing and implementing partnership evaluations, both internal and external stakeholders should be considered potential audiences for dissemination, thus informing which methodological techniques to adopt. El Ansari and Weiss (2006) argue that current research paradigms may not be a good fit for capturing the complexity of partnership approaches to community and systems-level outcomes. This supports the value of incorporating case examples within the field’s emerging theoretical frameworks, conceptual models, and typologies (e.g., see Gajda, 2004; Ivery, 2007; Kegler & Swan, 2011; Lasker, Weiss, & Miller, 2001; Mitchell & Shortell, 2000). In-depth case studies contribute substantially to the growing knowledge on collaboration as intervention because, as Jones, Crook, and Webb (2008) point out, “case analyses continue to add nuances and lessons that can be tested in the next case example and refined for those that follow” (p. 66). Participatory action research, where members of a community most affected by a particular issue are key players in the design and implementation of research and evaluation (Greenwood, 2012), also may be suitable for partnership evaluation because this approach incorporates the multiple perspectives of diverse partners inherent in collaborative interventions. These various approaches could build knowledge about interventions that is more seamlessly translated into practice because it has been vetted through the stakeholders who are essential in carrying out such interventions.

CPFOA

CPFOA, a national program of The RWJF from 2002 to 2010, awarded 15 CPs each US$900,000 to develop and implement a local strategic plan to improve lives of older adults through long-term supportive system improvements. Incorporating the principle that there is no one-size-fits-all model to meet the complex needs of older adults across communities, each CP was encouraged to develop a locally tailored approach to planning, implementing, and evaluating local systems change. Consequently, a variety of strategies for improving long-term supportive and service needs within each community were adopted (Bolda, Saucier, Maddox, Wetle, & Lowe, 2006). Characteristics of the CPs were diverse in terms of size, urban/rural/suburban location, region of the country, and organizational auspice of the fiscal agent. The membership of all CPs included older adults as well as health and social service providers from both the private and public sectors. Most CPs included atypical stakeholders in their membership such as utility company representatives, consultants, city planners, architecture firms, and public library employees, among others. Table 1 lists the name and location of each CP. Detailed information about each CP may be obtained at www.cpfoa.org. The overarching vision of the initiative, according to the National Program Office (NPO), was “To improve the long term care system by engaging community-based organizations, older adults, policy makers, and other stakeholders in determining how to best meet the needs of older adults, ensuring their independence and dignity.”¹ Each community defined for itself how it would address the national program’s broad vision and how the community would evaluate its work.

Table 1.

Community Partnerships for Older Adults Grantees.^a

City or region	Partnership name
Milwaukee, WI	Connecting Caring Communities
Ann Arbor, MI	Blueprint for Aging
Broome County, NY	Aging Futures
El Paso, TX	SALSA (Seniors Accessing Long-Term Care through Strategic Planning and Advocacy)
Boston, MA	Boston Partnership for Older Adults
La Grange, IL	Aging Well
Manchester, NH	Seniors Count
Atlanta, GA	Aging Atlanta
Rappahannock Rapidan Region, VA	Aging Together
San Francisco, CA	San Francisco Partnership for Community-Based Care & Support
Maui, HI	Aging with Aloha
Fremont, CA	Pathways to Positive Aging
Houston, TX	Care for Elders
Waynesville, NC	Haywood Community Partnership
Jacksonville, FL	Bridging Elder Care Networks of the Life: Act 2 Partnership

Note: ^aThe partnerships listed are those which received funding for both development and implementation of strategic initiatives.

The NPO at the Muskie School of Public Service, University of Southern Maine administered and directed the national initiative. Each partnership met the following minimum criteria to be chosen as a grantee (Bolda et al., 2006): partnership members represented a broad base of service providers that included traditional partners such as older adults, advocates, and other stakeholders from both the public and private sectors as well as less traditional partners such as members of the business community or other arenas; some evidence of prior success working as a partnership; an existing infrastructure for community-based services; and a designated group of core leaders. With these criteria met, each CP was required to implement its 4-year plan in a way that fit the characteristics of the community.

A Technical Assistance Consortium, initially convened by the Duke University Long-Term Care Program, provided technical assistance to the CPs in the following areas during implementation of their plans: communications, assessment and evaluation, leadership development, and fundraising. In addition to technical assistance, the CPs became what George Maddox referred to as “teaching and learning communities” whereby they were encouraged, as their work progressed, to share with each other their strategies and tactics undertaken as well as the lessons learned in the implementation process (see Bolda, Lowe, Maddox, & Patnaik, 2005).

Mathematica Policy Research (MPR) was commissioned by the RWJF to evaluate the national program as a whole, but not evaluate individual partnership initiatives. Therefore, each CP planned and implemented its own evaluation within the community, independently from the MPR evaluation strategy. Although each CP was required to submit a logic model and evaluation plan to the NPO to describe their strategies and projected process and outcomes, the design and methods of each CP varied. This created a challenge for the Mathematica evaluators of the national program which would lead to the national evaluation effort to focus less on narrowly articulated outcome measures and more on broader process and structural lessons (Menehan, 2011).This study complements the work of Mathematica in that it drills down to the evaluation strategies of each individual partnership.

The purpose of this article is to share lessons learned and build on the growing body of knowledge around community change evaluation practices involving multisite partnership initiatives. To that end, we examined how 15 CPFOA Partnerships approached evaluation of their individual efforts toward broader community change. As Claudia Coulton states in her response essay in the Aspen Institute report, Voices from the Field III, “multisite community change efforts have some of the greatest potential to be comparative” (Kubisch et al., 2010, p. 118).

Method

A retrospective qualitative design consisting of document analysis and analysis of secondary survey data (notes from previously conducted semistructured interviews) was employed. The CPFOA NPO was the source of data and documentation reviewed for this study. At the time of the original interview data collection (2008), eight of the partnerships were in their final year of a 6-year planning and implementation period whereas seven partnerships were in the 4th year of their 6-year process.²

In the first year of its 4-year implementation phase, each CP developed an individualized evaluation plan to assess the partnership in terms of its structure, process, and outcomes. Each CP was encouraged, but not required to use frame their evaluation strategy with the logic model they had previously submitted to the foundation as part of its proposal for funding. The evaluation plan was to serve as a tool to not only measure results of the specific initiatives but also to identify any need for change in how the work was being carried out by the partnership. All CPs received technical assistance on the development of evaluation plans. Documents collected for this study included logic models and evaluation plans created by each CP, other planning documents developed by each CP, documented communications between CPs and the NPO concerning evaluation (i.e., email polls and telephone interviews), and detailed descriptive profiles of each CP developed by NPO staff.

National program staff and technical assistance consultants conducted semistructured interviews via telephone. The telephone interviews were conducted in 2006 with up to three representatives (paid staff or other leadership) from each of the 15 CPs. Summaries of these phone calls were created by the technical assistance consultant (the lead author) and submitted to the NPO. Interview participants were asked the following: What [evaluation] strategies were successful or unsuccessful? How was evaluation used to guide or change implementation of partnership initiatives? What were the lessons learned around designing and implementing evaluation in partnership work? What would you do differently given your newly gained knowledge and experience around evaluation?

Content analysis was used across data sources to identify themes and lessons related to designing and implementing evaluation strategies. First, evaluation plans and logic models were reviewed for trends or distinctive characteristics, including similarities or differences across CP initiatives. Next, notes and summaries of the telephone interviews were analyzed for themes and to further explore the initial trends/distinctions identified in the evaluation plans and logic models. The coding and identification of themes was conducted only by the author, then compared retrospectively with the perceptions of national program staff in an effort to conduct member checking. Finally, CP profiles were reviewed to corroborate and help interpret findings of the content analysis.

Results

Guided by our research questions, we will first describe how CPs approached formative and summative evaluation in terms of their organizational styles and methods of planning and implementation. Next we present common challenges CPs faced in their evaluation work and the strategies or resources used to overcome them.

Styles and Methods of Planning and Implementing Evaluation

Formative Evaluation (Partnership Self-Assessment)

Although there was no “top-down” requirement from funders for the CPs to engage in formative evaluation, the data revealed that partnerships used this approach as a way to examine internal partnership functions and determine whether the intended member benefits were being actualized. Each partnership approached formative evaluation differently according to its own community resources and needs. CPs generally did not use the technical term “formative evaluation” to describe the methods they used to examine intrapartnership characteristics and processes, but instead referred to terms such as “partnership assessment” or “partnership self-assessment” to describe these evaluation strategies. Though terminology may have differed, it was clear across the CPs that formative evaluation methods served the purpose of determining if (a) the organizational characteristics of the partnership met the needs of its membership, and (b) the process through which the CP functioned was the most effective means for facilitating the partnership’s work.

Often, formative evaluation was seen as a way for members to determine the value and assess the process of working together. It allowed CPs to identify strengths and potential areas for improvement in the organizational structure, governance, leadership, and management of the partnerships. It also served as a means to build relationships through partnership work. For example, one CP member shared,

This Partnership’s deeply inclusive process of data gathering and strategic planning used tools, venues, and languages to reach across many ethnic and underserved populations. This approach built new & stronger bridges, leadership and relationships able to leverage resources and solutions for their region’s frail elders.

Some CPs approached formative evaluation using more traditional, formal methods whereas others took informal approaches. The evaluation approach was generally driven by the organizational culture of the CP, whereas those with less formal structures used less formal approaches. More formal methods included surveys of individual partnership members (written, electronic, telephone, or in person), focused conversations with key informants, focus groups, or community dialogues with membership representatives. Less formal approaches included simple postactivity polls (written or electronic) or debriefing sessions which solicited reactions or feedback around an action or activity sponsored by a CP. Regardless of the formal versus informal methods and strategies used, the common purpose of formative evaluation across CPs was to assess the structure and process characteristics of the partnership. Examples of indicators (in question form) used by CPs to assess common structure and process characteristics are listed in Table 2.

Table 2.

Formative Evaluation Indicators Adopted by CPFOA Partnerships.

Indicator	Operational examples
Composition of the partnership (size, diversity, representation of membership)*	Always asking “who is missing” from the partnership and always asking whether representation from direct service providers and recipients are sufficiently represented.
Resources* (human, political)	If your partnership is planning to develop workgroups to implement several initiatives, are there enough members participating to ensure the same people are not responsible for the work of all of the groups?
Accountability and responsibility of staff, leaders, and individual partners	Are there clear expectations regarding the responsibilities of partnership staff, core leaders, and other members? Is there a consequence when someone does not meet expectations?
Mechanisms for internal and external communication*	How are partners informed about events, accomplishments, and other “goings on” of the partnership? Do they feel informed and in touch? How are nonmembers informed?
Mechanisms for soliciting participation and leadership	Is there a clear and satisfactory process for appointing formal leaders?
Perceived benefits of participation	Do partners feel the time and resources they are contributing result in benefits to their agencies and the community’s older adults?
Sustainability	Are the types and sources of funding for partnership sponsored activities growing? Do projects initially sponsored by the partnership continue after start-up without partnership support? Are funders supporting the core functions of the partnership (i.e., staffing) and not just the initiatives?
Cooperation among members*	Are organizations increasing their levels of collaboration with each other through shared resources, Memorandum of Understandings, or joint funding applications?
Centrality (awareness of the partnership)*	Is the Partnership the “go-to” entity for information, problem solving, leadership, or other resources? Are community members aware of the Partnership?
Consistency between planning and implementation*	Are activities and initiatives being carried out as expected? Why or why not? What aspects of the partnership structure are fostering or preventing the successful implementation of planned activities and initiatives?

Note: *Dimension that aligns with the Mitchell and Shortell (2000) typology of community health partnerships.

Data revealed that formative evaluation was an ongoing, iterative process to help members (and staff) facilitate the work required to successfully carry out initiatives and activities and make strategic changes as necessary. Because systems change and community-level impact cannot happen overnight, formative evaluation was used by CPs to measure their capacity to face current and future challenges, remain sustainable, and work collectively toward systems improvements. This approach sometimes required changes in courses of action or organizational structure. As one CP member stated,

Our structure changed based on what we learned [from the assessment] . . . now the board defines work direction. The work groups are organized to enact. They are more focused now and have more direction . . . people see results. This will strengthen the Partnership.

While utilizing formative evaluation for strategic decision making, it was clear from the data that evaluation planning was essential, which in itself is also strategic. A representative from one CP stressed the importance of using an evaluation plan to guide this work, stating, “I am still adamant about a strong evaluation plan. It serves as work plan . . . It’s our accountability plan and is our performance plan.”

Finally, respondents noted a critical linkage between formative and summative evaluation strategies in partnership work. According to the data, there was a common perception that if process characteristics such as governance and management of the partnerships were aligned with members’ needs, CPs would be viewed as effective, demonstrated by satisfaction among members, successful program outcomes, and sustainability. Some CPs used existing partnership assessment tools for this purpose. Reflecting on how formative evaluation was used to better align partnership process with member needs, a CP staff member described that their partnership used a “collaborative factors inventory to document satisfaction with the Partnership and make improvements where needed.” This CP learned that members needed more diverse communications methods for sharing information, so improvements were made to this partnership’s website to increase members’ online access of materials and information.

The NPO introduced to all CPs the Mitchell and Shortell (2000) typology of community health partnerships, which identifies seven indicators for examining partnerships. This typology, explicitly used by three CPs as a conceptual framework for evaluation, facilitated communication between all partnerships by offering a shared platform for teaching and learning from each other. For example, despite the variety of approaches to measure community awareness of their work, CPs were able to use the typology to collectively describe this concept as centrality and use this common language to share lessons. As shown in Table 2, among the formative evaluation indicators adopted in CP evaluations, six of them align with Mitchell and Shortell’s typology. Moreover, their typology links formative and summative evaluation in that the indicators integrate governance and management characteristics of partnerships to its overall effectiveness.

Summative Evaluation

Partnerships typically approached summative evaluation work by tracking immediate outputs or outcomes, not by measuring long-term impacts of systems change. Examples of outcome analysis and tracking ranged from complex cost–benefit analysis to simple documentation of meeting and event attendance. One partnership, for example, conducted a cost–benefit analysis of a home modification program, comparing it to costs of nursing home placement. This analysis showed significant cost savings resulting from the intervention. A less technically advanced example consisted of one CP that counted the number of phone calls received by a local TV station following each broadcast of a program produced by the CP. An important point to be made regarding both of these examples is that in each case, the intervention’s design and implementation, as well as its evaluation, required multisectoral collaboration, thus illustrating the added value of partnership work.

Data revealed the importance of member inclusivity in approaching summative evaluations. When all partnership members were responsible for designing and implementing evaluation plans, including members directly carrying out initiatives as well as those in leadership roles or on evaluation committees, CPs were better prepared to undertake summative evaluation. According to our data sources, this participatory involvement builds capacity for conducting evaluation. As one CP representative stated, “Every partner has learned about evaluation. They now know how to find out, not just whether a goal has been met, but how a goal was met, and how they know this.”

Finally, data sources revealed that creativity and flexibility of partnership members and other stakeholders, as well as flexibility of the organizational structure, were necessary to successfully design and implement summative evaluations. Each CP designed its evaluation to meet the needs of the individual community, incorporating various ways to measure outputs and outcomes so that they could be understood across diverse audiences such as partnership members (or potential members), funders (or potential funders), or policy makers. Stakeholders were required to view evaluation differently than they had in the past. One CP member stated,

The act of evaluation really is different than simply doing progress reports. The RWJF grant has truly allowed us to tackle the big initiative of testing a new model that was not tried before. Implementing new and unique strategies means you really have to answer the question, ‘Is this really better?’ using measurable indicators, or markers of success.

Summative evaluation strategies capitalized on resources within the CP, and sought external expertise when needed. Two creative approaches which mined the collective wisdom of existing partners and did not require outside evaluation expertise to collect or analyze data included: The development of an ongoing compendium of partnership accomplishments, challenges, and lessons learned; and a “story bank” to provide concrete illustrations of a CP’s work and accomplishments. More traditional approaches which sought resources from external stakeholders included pre-post tests of interventions, cost–benefit analyses, and longitudinal surveys of random, representative community samples. A telephone survey instrument created by MPR for the purpose of the national evaluation was used by one partnership to design follow-up community phone surveys to measure changes over time. Because the work of the CP was unprecedented in each community, flexibility, and creativity were required in approaching summative evaluation to allow for strategic changes as necessary.

Challenges of Evaluation and Strategies for Overcoming Them

Evaluation work is challenging, particularly when it requires interorganizational collaboration. The challenges experienced by CPs and strategies for overcoming them will be discussed along the three emergent themes: limited resources, prioritizing objectives and activities, and gaining buy-in from stakeholders to support evaluation work.

Limited Resources for Evaluation

An environment of limited resources is a familiar one for community agencies serving older adults and such an environment is particularly evident when demands for collaboration and evaluation of collaborative efforts are added to the mix. Resource limitations related to evaluation included limited knowledge and skills around evaluation as well as limited reliable data sources for evaluation purposes. Some partnerships had members with expertise in research and evaluation, and some were able to identify and use data over time to begin to measure more long-term results; however, most partnerships in their early stages of implementation were not prepared to design evaluations that would measure long-term outcomes or impact of their work. All partnerships recognized short-term outcomes as the necessary “stepping stones” toward more systematic, long-term impact they were trying to achieve, but needed knowledge, skills, and data to build on their evaluation efforts.

One approach CPs took to overcome these challenges was to develop relationships with new partners who were not traditionally connected to the aging services network. These nontraditional partners sometimes helped to support evaluation efforts. Examples of new partners included universities for data and research expertise (including mapping skills), state or city offices related to zoning or planning, local libraries, think tanks, advocacy organizations, utility companies, business associations, and private research, public relations/communications, or architecture firms.

Partnering with external stakeholders most often allowed partnerships to approach evaluation in new and creative ways not anticipated in their initial planning of initiatives. One CP reached out to the corporate sector to help facilitate a community forum in which data were presented and discussed: “An IBM consultant facilitated the discussion of the data with the questions: What did you hear? What do you think is most important” And what else should we consider?” Nearly all CPs developed relationships with academic partners. Sometimes, however, challenges arose in university–partnership collaborations whereas timing between academic calendars and partnership timelines were not in-sync or consensus on metrics could not be reached. One partnership collected community data from a random telephone survey using some measures from an instrument designed by the national program evaluators. The purpose of this survey was to capture repeated measures over time. This was the one case in which the CP’s evaluation strategies built on MPR’s involvement as the outside evaluator of the national program.

The knowledge and skills gained from new stakeholders ultimately was shared across partnerships. The CPFOA principle of “teaching and learning” across communities was embraced by all partnerships in this study and emerged from the data as a significant strength of the national model. Sharing information across communities allowed CPs to learn from each other and take a more informed approach to their own individual efforts. One CP staff member noted, “One thing we learned was how much we could learn from other CPFOA grantees . . . ” Each community carried this practice into their individual partnership work and promoted teaching and learning principles within their multiple initiatives and organizations. Therefore, benefits of the teaching and learning model were noted both within and across partnerships.

A training-enriched environment was raised by a majority of CPs (n = 9) as a way to successfully design and implement successful evaluation work. Training partnership members about basic evaluation concepts often led to a collective ability and willingness to critically examine the structure, process, and outcomes of the partnership. It also empowered members to use their new knowledge and skills within their own organizations. This left them better prepared for planning future initiatives as well as better equipped to document strengths within their own agencies, thus building member capacity.

Prioritizing Objectives and Activities

A second theme, prioritizing specific evaluation needs, was a challenge faced by nearly all CPs. First, all partnerships faced the common challenge of deciding whether to focus efforts (and limited resources) on formative or summative evaluation strategies. Although each community approached evaluation differently, it was evident at some point that both formative and summative evaluation strategies were deemed necessary to sustain their work. Respondents indicated that formative results provided a mechanism for keeping partners informed of their progress, including successes and challenges, which maintained the momentum of the work. Formative evaluation strategies also were more feasible to implement given the limited resources described above. Summative evaluation results, on the other hand, offered tangible results of the partnership work to outside audiences or stakeholders (e.g., policy makers or potential funders).

In addition to prioritizing formative versus summative efforts, each CP faced the challenging realization that it was not possible to examine every process and outcome identified during the planning phase of the work. The CPs had to prioritize (a) process and outcome indicators most important to document and measure (and which not to measure) and (b) key audiences and stakeholders with which evaluation findings would be shared. Evaluation was used to inform multiple, diverse internal and external audiences about the work of the partnership.

To overcome the challenge of prioritizing evaluation objectives and activities, it was common for partnerships to form a committee to oversee the implementation of evaluation activities. Evaluation committees varied in terms of membership representation from core leadership only to a more diverse membership, including service recipients and direct service providers. Evaluation committees always included paid partnership staff, either as a facilitator or committee member. As articulated by an interview respondent, a major purpose of an evaluation committee is to pose the question, “In the end, what are the two or three most important things we want to share and with whom do we want to share this information?”

Securing Buy-In From Partnership Members

Raising awareness among partners that evaluation is a necessary component of partnership work was a common challenge in all CPs, and is a persistent challenge in collaborative community work. Some communities overcame this challenge by promoting and integrating evaluation into all CP functions, where all members participated in some form of evaluation work. Partnerships approached this in several ways including: educating all members around how to design and carry out evaluation activities and giving each workgroup an opportunity to develop its own evaluation plan (n = 6), identifying champions within the partnership who served as evaluation leaders within the partnership (n = 8), and including the topic of evaluation on each meeting agenda (n = 2).

Although each CP approached evaluation differently, integrating evaluation activities into all CP functions increased “buy-in” among members that evaluation was an essential component of partnership success and introduced unexpected benefits. One representative shared that “The process of gathering data enabled collaboration among institutions that had never worked with one another and the information itself guided and motivated the Partnership’s decision-making and future plans.”

Discussion

The purpose of this study was to examine the evaluation efforts of 15 CPs working to improve long-term supportive service systems to better meet the needs of older adults, and draw lessons regarding effective evaluation practices. This discussion will first describe how the integrative framework for collaborative capacity (Foster-Fishman et al., 2001), introduced earlier in this article, may be applied specifically to examine partnership evaluation. Next, four overarching lessons drawn from the qualitative findings of this study will be discussed along with their implications to practice and research.

Application of the Integrative Framework of Collaborative Capacity

The four elements of Foster-Fishman and colleagues’ integrative framework may be a useful lens to identify characteristics that may facilitate successful evaluation work. Table 3 illustrates how each element of collaborative capacity (member, relational, organizational, and programmatic) can be used to conceptually interpret the results of this study.

Table 3.

Collaborative Capacity Characteristics that Promote Successful Evaluation.*

Elements of collaborative capacity	Partnership characteristics for successful evaluations
Member capacity	CP members are taught evaluation skills. Evaluation methods undertaken by CPs are feasible and accessible so all members can participate in the work. CP members understand purpose and utility of evaluation.
Relational capacity	External resources in the community are tapped by the CP to assist with evaluation efforts, including “non-traditional” partners. CP identifies key audiences and stakeholders with whom to communicate evaluation results, then prioritizes essential activities accordingly.
Organizational capacity	A flexible organizational structure is adopted to integrate formative evaluation results into strategic, collective decision making. A teaching and learning model is supported both internally (within the CP) and externally (with other partnerships). Internal assets and resources are identified and capitalized (e.g., nontraditional data sources).
Programmatic capacity	Measurement indicators are prioritized based on feasibility (i.e., realizing the CP “cannot measure everything”). Less traditional or alternative data sources are used (e.g., the story bank). The CP believes that successful program outcomes are achieved when members have positive perceptions of the governance and management of the partnership.

Note: *This table applies Foster-Fishman and colleagues’ integrative framework of collaborative capacity to the findings of this study (see Foster-Fishman et al., 2001).

The first element of member capacity posits that members demonstrate skills and attitudes that facilitate collaborative evaluation work. The partnership also needs access to these competencies. For example, educating members around evaluation design and techniques will build current knowledge and skills around evaluation, thus building member capacity that the partnership will then have access to.

The second element of relational capacity values relationships both between members and with external networks. This may involve facilitation of trust-building between organizational representatives who traditionally have competed for limited resources in their community or may require educating members around communications skills for reaching out to new types of partners such as the media or other sectors. In this study, CPs that reached out to nontraditional partners for evaluation expertise exemplified relational capacity.

Third, the partnership demonstrates organizational capacity through effective leadership, policies and procedures, communication, resource acquisition, and the ability to self-correct as the CP as an organization evolves. Applying organizational capacity to evaluation work may involve creating an organizational structure that allows for strategic changes to partnership work based on formative evaluation findings.

The final element in the collaborative capacity framework posits that the coalition must demonstrate programmatic capacity to attain desired outcomes. In this study, an example of programmatic capacity is the successful prioritization of evaluation indicators and strategies, including utilization of less traditional data sources for evaluation work.

Overarching Lessons and Implications for Practice and Research

First, findings of this study suggest that both formative and summative approaches are essential in partnership evaluation. Mechanisms to understand partnership processes and functioning must be integrated into evaluation planning and implementation. Although this supports existing literature stating that effective evaluation planning and implementation are ongoing and iterative activities (Israel et al., 1998; Krueter et al., 2000), partnership agencies are faced with limited resources for research and evaluation and funders or regulatory bodies often cast evaluation in terms of outcomes (i.e., summative) research. In practice, however, the emphasis on outcomes overshadows formative evaluation strategies in an effort to fulfill funding requirements (Kubisch, et al., 2010; Mitchell & Shortell, 2000).

Program evaluation work becomes increasingly complex when the “program” being evaluated is not one organizational entity, but is a network of diverse players and stakeholders representing different organizational entities with various perspectives and perhaps different interests. To add to the complexity, CPs are comprised of individual community members (i.e., unpaid advocates or service recipients) as well as organizational representatives (i.e., paid professionals) from both the private and public sectors. In formative evaluation, this diverse membership creates challenges for consensus building around how to grow all elements of collaborative capacity (member, relational, organizational, and programmatic). In summative evaluation, the two predominant challenges that arise are generalizability or replicability, and determining causality. Implementation of an initiative that may have been successful in one community may not be successful in another unless it is tailored to meet its unique needs. Determining causality is an unattainable goal whereby it is not possible to isolate the work of a partnership from possible confounding variables to explain or predict changes in community indicators such as knowledge, attitudes, behaviors, or service access.

Second, promotion and development of knowledge and skills related to research and evaluation within the partnership may help support successful planning and implementation of evaluation work. The teaching and learning emphasis of the CPFOA program provided opportunities to engage various stakeholders in research knowledge-building, although partnerships differed in the degree to which they involved the staff and volunteers of partner organizations. The partnerships that trained their members in research principles and evaluation practices built member capacity and promoted shared responsibility (Foster-Fishman et al., 2001). These findings support existing literature suggesting that ownership on behalf of stakeholders is essential (Dluhy & Swartz, 2006; Mizrahi & Rosenthal, 2001). Involving stakeholders increases buy-in for evaluation and helps them maintain a sense of “knowing where this is going” and creates a culture of collective responsibility and one, according to Kubisch et al.(2010), “that values candid dialogue and analysis and embraces the idea of learning while doing” (p. xi).

Frontline workers like home care attendants or case managers have the capacity to bring direct client experience to inform evaluation approaches, whereas supervisors or managers may bring facilitation expertise to carry out planned evaluation strategies, and agency or community leaders may garner resources to support evaluation work. An essential player in partnership practice is one that brings these diverse stakeholders together. As Dluhy and Swartz (2006) point out, a neutral convener is generally required in partnership work. Those trained in community practice may bring specific competencies to this role of neutral convener and bring the ability to facilitate strategies for assessing process as well as measuring long-term community impact (Thompson, Minkler, Bell, Rose, & Butler, 2003). Interdisciplinary practitioners could be particularly effective at communicating evaluation findings both within partnerships and with outside stakeholders. The CPs had access to other communities in the CPFOA network for “teaching and learning” opportunities. Seeking information from other communities implementing partnership initiatives, however, was a challenge. The growth of social media may be an opportunity to overcome this challenge in the future.

Third, the results of this study point to the importance of flexibility and creativity in designing and implementing evaluation. This was demonstrated by the CPs use of evaluation measurement strategies not traditionally considered in program evaluation, and by flexible organizational structures that changed as necessary. Although traditional summative evaluation strategies were encouraged by the requirement of a logic model, the NPO created space for alternative research ideas by not mandating a standardized evaluation approach. Creativity was further encouraged by the participation of diverse stakeholders. This finding suggests that perhaps more integration of participatory research principles used in community-based participatory research would fit well in the partnership model (Stahl & Hahn, 2007). Although creative evaluation strategies may be useful in partnership evaluation, this creates a challenge for community practitioners and researchers who are often required by funding and regulatory bodies to complete traditional outcome-based evaluation. This challenge illustrates the need for funders to support evaluation approaches from a variety of research paradigms.

Partnership research is still a nascent field of study with a limited evidence base in the form of rigorous and robust systematic inquiry (Zakocs & Edwards, 2006; Zakocs & Guckenburg, 2007), particularly in the field of aging (Ivery & Akstein-Kahan, 2010; Warburton, Everingham, Cuthill, & Bartlett, 2008). Traditional research methods will not suffice in building our knowledge base given the complex contextual factors involved and the individuality of each local community. Clearly, better use of conceptual frameworks is needed, and the very nature of partnerships—their participation—may be best suited for more innovative research methods such as community-based participatory research, an approach not yet optimized by researchers (Stahl & Hahn, 2007). As the results of this study demonstrate, CPs creatively identified new sources of data and methods to evaluate their work.

To balance creativity and methodological rigor, guiding principles could be developed to assist communities with implementing and evaluating new initiatives through partnerships in a way that allows for individuality while maintaining the fidelity of a new initiative. These principles could be shared using innovative communication media such as blogs or wiki tools updated collectively by stakeholders to document a partnership’s accomplishments. Social media has also emerged as a venue for incorporating new evaluation methods into partnership work, while increasing numbers of older adults are using social media. These innovative dissemination examples require a new approach to evaluation which requires different technical skills to be embedded into the work of CPs and promotes shared responsibility among partnership members.

The fourth lesson suggests that leveraging resources from both within the partnership and outside of the partnership to build capacity for evaluation work is crucial. Evaluation work is not easy and requires additional resources not always available within the partnership. The CPs in this study tapped traditional and nontraditional resources such as private consultants, university-affiliated researchers, and a variety of stakeholders from the private sector in many fields. This may not occur without challenges as nontraditional partners often have different jargon, perceptions of the work to be accomplished and timeline expectations. When these challenges were overcome, stronger working relationships and increased relational capacity of the partnership was built. Indeed, multiple perspectives contributed by different stakeholders may foster easier translation between research, policy, and practice (Feldman, Nadash, & Gursen, 2001; Warburton, Everingham, Cuthill, & Bartlett, 2008).

Effective evaluation of partnership work may help practitioners and researchers tap collective community resources and improve long-term supportive systems. Moreover, partnership work begins to break down disciplinary and funding silos in a way that may lead to more integrated programming and improved access to services for older adults. Social workers, community psychologists, and public health educators may play a key role in the work of CPs, as they are trained to integrate both practice and research into their work and can serve as facilitators of collaborative work. Equally essential, however, are older adults, individuals working directly with older adults, and agency directors and other community leaders with decision-making capacity around home and community-based service delivery.

Finally, partnership evaluation may contribute to the currently inconclusive evidence of the cost-effectiveness of delivering home and community-based services for older adults and people with disabilities (Ivery & Akstein-Kahan, 2010; Warburton, Everingham, Cuthill, & Bartlett, 2008). With health care costs increasing uncontrollably, long-term supportive services for people at home and in community settings rather than in institutional settings may be more cost-effective, but the specific predictors of savings are not yet clear (Grabowski, 2006; Kaye, LaPlante, & Harrington, 2009). For older adults to maintain health and avoid unnecessary hospitalizations, they need better knowledge of and access to home and community-based services. Collaborative interventions such as CPFOA, designed to address the lack of information and other challenges faced by older adults in community settings may help identify specific methods for targeting cost savings.

The study’s methodological limitations do, however, limit the extent the findings may be generalized. A specific limitation is the retrospective approach, in which the data analyzed were not originally collected systematically with the current research questions in mind. A further limitation is that the experience of CPFOA partnerships may be unique, thus making the findings less generalizable.

Future research and more cohesive strategies for incorporating research into practice are clearly needed to identify the most effective means of evaluating partnership work. Better integration of conceptual frameworks, innovative research and dissemination strategies will contribute to our understanding of pathways between partnership structure, process, outcomes, and long-term impact on the lives of older adults.

Footnotes

Acknowledgements

The authors wish to acknowledge national program office staff and technical assistance consultants (Phyllis Bailey, Laura Bly, Elise Bolda, Kathryn Lawler, and Mia Oberlink) of the Community Partnerships for Older Adults (CPFOA) program of the Robert Wood Johnson Foundation for collecting original data, conceptualizing this study, and reviewing early drafts of this manuscript. Additional acknowledgement must go to the many members of local CPFOA partnerships whose documentation and reflections were examined in this study. Our gratitude also goes to the anonymous reviewers whose suggestions refined our thinking about the study and greatly strengthened this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biographies

Nancy Giunta, PhD, MSW, MA, is an Assistant Professor and John A. Hartford Faculty Scholar of Geriatric Social Work at the Silberman School of Social Work at Hunter College, City University of New York. Her research and practice interests are in collaborative efforts to improve long term services and supports for older adults and people with disabilities, particularly those who are underserved.

M. Lori Thomas, PhD, MSW, MDiv, is a John A. Hartford Geriatric Social Work Scholar and Assistant Professor of Social Work at the University of North Carolina at Charlotte. Her research interests include homelessness and housing crises among older adults and community efforts to address the housing needs of vulnerable populations.

References

Alley

Liebig

Pynoos

Banerjee

Choi

(2007). Creating elder-friendly communities: Preparations for an aging society. Journal of Gerontological Social Work, 49(1/2), 1-18.

Bayne-Smith

Mizrahi

Garcia

M. L.

(2008). Interdisciplinary community collaboration (ICC): Perspectives of community practitioners on successful strategies. Journal of Community Practice, 16, 249-269.

Blair

Minkler

(2009). Participatory action research with older adults: Key principles in practice. The Gerontologist, 49, 651-662.

Bolda

E. J.

Lowe

J. I.

Maddox

G. L.

Patnaik

B. S.

(2005). Community Partnerships for Older Adults: A case study. Families in Society, 86, 411-418.

Bolda

E. J.

Saucier

Maddox

G. L.

Wetle

Lowe

J. I.

(2006). Governance and management structures for community partnerships: Experiences from the Robert Wood Johnson Foundation’s community partnerships for older adults program. The Gerontologist, 46, 391-397.

Dluhy

Swartz

(2006). Connecting knowledge and policy: The promise of community indicators in the United States. Social Indicators Research, 79(1),1-23.

El Ansari

Weiss

E.S.

(2006). Quality of research on community partnerships: Developing the evidence base. Health Education Research Theory & Practice, 21, 175–180.

Feldman

P. H.

Nadash

Gursen

(2001). Improving communication between researchers and policy makers in long term care: Or, researchers are from Mars, policy makers are from Venus. The Gerontologist, 41, 312-321.

Foster-Fishman

P. G.

Berkowitz

S. L.

Lounsbury

D. W.

Jacobson

Allen

N. A.

(2001). Building collaborative capacity in community coalitions: A review and integrative framework. American Journal of Community Psychology, 29, 241-261.

10.

Francisco

V. T.

Schultz

J. A.

Fawcett

S. B.

(2004). Making sense of results from collaboration evaluations. In Backer

T. E.

(Ed.), Evaluating community collaborations (pp. 113-128). New York, NY: Springer.

11.

Gajda

(2004). Utilizing collaboration theory to evaluate strategic alliances. American Journal of Evaluation, 25(1), 65-77.

12.

Grabowski

(2006). The cost-effectiveness of nonistitutional long-term care services: Review and synthesis of the most recent evidence. Medicare Care Research and Review, 63(1), 3-28.

13.

Granner

M. L.

Sharpe

P. A.

(2004). Evaluation community coalition characteristics and functioning: A summary of measurement tools. Health Education Research, 19, 514-531.

14.

Greenfield

(2012). Using ecological frameworks to advance a field of research, practice, and policy on aging-in-place initiatives. The Gerontologist, 52, 1-12.

15.

Greenwood

(2012). Doing and learning action research in the neo-liberal world of contemporary higher education. Action Research, 10, 115-132.

16.

Israel

Schulz

Parker

Becker

(1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19, 173-202.

17.

Ivery

J. M.

(2007) Organizational ecology: A theoretical framework for examining collaborative partnerships. Administration in Social Work, 31(4), 7-19.

18.

Ivery

J. M.

Akstein-Kahan

(2010). The naturally occurring retirement community (NORC) initiative in Georgia: Developing and managing collaborative partnerships to support older adults. Administration in Social Work, 34, 329-343.

19.

Johnny

Jones

J. M.

Crook

W. P.

Webb

J. R.

(2008). Collaboration for the provision of services: A review of the literature. Journal of Community Practice, 15(4), 41–71.

20.

Kaye

LaPlante

Harrington

(2009). Do nonistitutional long-term care services reduce Medicaid spending? Health Affairs, 28, 262-272.

21.

Kegler

M. C.

Swan

D. W.

(2011). Initial impact at operationalizing and testing the community coalition action theory. Health Education & Behavior, 38, 261-270.

22.

Krueter

M. W.

Lezin

N. A.

Young

L. A.

(2000). Evaluating community-based collaborative mechanisms: Implications for practitioners. Health Promotion Practice, 1(1), 49-63.

23.

Kubisch

A. C.

Auspos

Brown

Dewar

(2010). Voices from the Field III: Lessons and challenges from two decades of community change efforts. Washington, DC: Aspen.

24.

Lasker

R. O.

Weiss

E. S.

Miller

(2001). Partnership synergy: A practical framework for studying and strengthening the collaborative advantage. Milbank Quarterly, 79, 179-205.

25.

Menehan

(2011, July 7). Community Partnerships for Older Adults: Program results report. Princeton, NJ: Robert Wood Johnson Foundation. Retrieved from http://www.rwjf.org/files/research/CSL.final.pdf

26.

Mitchell

Shortell

(2000). The governance and management of effective community health partnerships: A typology for research, policy, and practice. Milbank Quarterly, 78, 241-289.

27.

Mizrahi

Rosenthal

(2001). Complexities of coalition building: Leaders’ successes, strategies, struggles and solutions. Social Work, 46(1), 63-78.

28.

Preskill

(2010). Response essay. In Kubish

A. C.

Auspos

Brown

Dewar

(Eds.), Voices from the Field III: Lessons and challenges from two decades of community change efforts (pp. 106-110). Washington, DC: Aspen.

29.

Scharlach

A. E.

(2009). Creating aging-friendly communities. Generations, 33, 5-11.

30.

Scharlach

A. E.

Sanchez

(2010). From interviewers to friendly visitors: Bridging research and practice to meet the needs of low-income Latino seniors. Journal of Gerontological Social Work, 54(1), 73-91.

31.

Stahl

S. M.

Hahn

A. A.

(2007). The resource centers for minority aging research (RCMAR): Community/researcher interaction and relationships. Ethnicity & Disease, 17, S1-S2.

32.

Thompson

Minkler

Bell

Rose

Butler

(2003). Facilitators of well-functioning consortia: National healthy start program lessons. Health and Social Work, 28, 185-195.

33.

Warburton

Everingham

Cuthill

Bartlett

. (2008). Achieving effective collaborations to help communities age well. Australian Journal of Public Administration, 67, 470-482.

34.

Weiss

Anderson

Lasker

(2002). Making the most of collaboration: Exploring the relationship between partnership synergy and partnership functioning. Health Education and Behaviour, 29, 683-698.

35.

Wildridge

Childs

Cawthra

Madge

(2004). How to create successful partnerships: A review of the literature. Health Information and Libraries Journal, 21(S1), 3-19.

36.

Zakocs

R. C.

Edwards

E. M.

(2006). What explains community coalition effectiveness? A review of the literature. American Journal of Preventive Medicine, 30, 351-361.

37.

Zakocs

Guckenburg

(2007). What coalition factors foster community capacity? Lessons learned from the Fighting Back Initiative. Health Education & Behavior, 34, 354-375.