Abstract
Collaboration is a widely utilized strategy for addressing complex social issues and for facilitating organizational innovation and performance. Evaluators are uniquely positioned to empirically examine the development and effects of interagency and interprofessional collaboration. In this article, the authors present the Collaboration Evaluation and Improvement Framework (CEIF), an extension of earlier work in collaboration theory development. The CEIF identifies five points of entry to evaluating collaborations and suggests actions that evaluators can take to (a) define and describe the evaluand of collaboration, (b) measure the attributes of organizational collaboration over time, and (c) increase stakeholder capacity to engage in efficient and effective collaborative practices. Use of the CEIF to operationalize and assess the construct of collaboration can enable the evaluator to ascertain how collaborative efforts correlate with indicators of organizational impact and outcomes.
Keywords
Organizational collaboration is embraced across multiple sectors of society as a primary strategy for cultivating innovation, conserving economic resources, building relationships, addressing complex problems, and reaching essential outcomes. It is through collaboration that organizations address societal issues, accomplish tasks, and reach goals that fall outside the grasp of any individual entity working independently. As Friedman (2005) attests, It is this triple convergence-of new players, on a new playing field, developing new processes and habits for horizontal collaboration-that I believe is the most important force shaping global economics and politics in the early twenty-first century. Giving so many people access to all these tools of collaboration … ensures that the next generation of innovations will come from all over Planet Flat. The scale of the global community that is soon going to be able to participate in all sorts of discovery and innovation is something the world has simply never seen before. (p. 181)
Though widely recognized as having the capacity to leverage fragmented systems and produce increased efficiency and innovation, effective organizational partnerships do not emerge spontaneously and cannot be sustained without thoughtful attention to their development (Gray, 1989). Vangen and Huxham (2003) report that, Many organizations aspire to gain collaborative advantage by working in partnerships across organizational, sectoral, and even national boundaries. Such collaborations, however, are difficult to manage, and the likelihood of disappointing outputs is high. To create advantage, practitioners need to engage in a continuous process of nurturing the collaborative processes. (p. 5)
Evaluators are uniquely positioned to empirically examine the development and effects of agency and interprofessional collaboration. In this article, we present a theoretically grounded framework that evaluators can use to inform their efforts to systematically measure, assess, and promote the process and outcomes of collaboration.
Collaboration Evaluation and Improvement (CEIF) Framework
The CEIF is a synthesis and extension of earlier work in collaboration theory development and evaluation research (Gajda, 2004; Gajda & Koliba, 2007; Hogue, 1993; Koliba & Gajda, 2009; Thomson et al., 2009; Vangen & Huxham, 2003; Wood & Gray, 1991). Shown in Figure 1, the CEIF framework is comprised of five entry points for thinking about when, where, and how to engage in the complex task of evaluating organizational collaboration. These entry points include: (1) operationalizing the construct of collaboration; (2) identifying and mapping communities of practice; (3) monitoring stage/stages of development; (4) assessing levels of integration; and (5) assessing cycles of inquiry.
Collaboration evaluation and improvement framework.
The framework is relevant to evaluators working in a range of fields (e.g., education, health, justice), settings (e.g., interorganizational, intraorganizational, and/or interprofessional), and stages of partnership development (e.g., formation, implementation, and cessation). The CEIF also suggests qualitative and quantitative data collection strategies and measurement tools that can be used in a variety of evaluation contexts. As Thomson, Perry, and Miller (2009) report, “Few instruments to measure collaboration exist and those that do are difficult to adapt outside the immediate context of a particular study” (p. 29). CEIF strategies such as monitoring strategic alliance development and tools such as the Levels of Organizational Integration Rubric (LOIR) and Team Collaboration Assessment Rubric (TCAR), referenced in CEIF Phases 3–5, respectively, are designed to be applicable in a range of collaboration evaluation settings and useful for examining both interorganizational and interprofessional collaboration over time.
To develop the CEIF, we have reviewed the literature and research in organizational learning, public administration, and applied behavior management, and piloted the framework and its associated tools and strategies in our own evaluation practice. In addition, we have integrated feedback from evaluation practitioners who have implemented elements of the CEIF and shared their experiences with us at think tanks, workshops, and presentations delivered at the American Evaluation Association (AEA) and AEA-Centers for Disease Control and Prevention (CDC) conferences over the past 6 years. In the remainder of this article we describe, the CEIF, its phases and suggested strategies and tools, and explain how evaluators can use it to inform their decision-making about when, where, and how to assess organizational collaboration.
Phase 1—Operationalize Collaboration
Although the literature in support of organizational collaboration is vast, cross sectoral and replete with case studies, collaboration persists as an underempiricized, misunderstood construct (Thomson et al., 2009). Hence, evaluators that seek to examine organizational collaboration as a dependent and/or independent variable will confront the need to operationalize the concept. Take, for example, the following evaluation research questions: Do increases in collaboration between our two local mental health agencies and the nurse home visitation program lead to a reduction in teen pregnancy? To what extent does collaboration between the CDC, the Association for State and Territorial Dental Directors, and the state oral health departments lead to increases in water fluoridation and delayed onset of caries? What is high-quality teacher collaboration? How should we as district administrators develop teacher collaboration? To what extent does teacher collaboration lead to better instruction and improved outcomes for student learning? What will improve collaboration between the Animal Plant Health Inspection Service and the National Association of State Departments of Agriculture so as to ensure incident response preparedness? At what point do continued efforts to increase collaboration among community organizations and to build a coalition have a minimal or negligible effect on the health and welfare of those we serve? What are “optimal” levels of collaboration within Health Associated Infection (HAI) Prevention Collaboratives? What metrics should we use for understanding HAI Prevention Collaboratives rather than just taking the nebulous “more is better” default position?
In each case, collaboration is a central evaluand which must be characterized by specific attributes and variables so that its development, quantity, quality, and/or effects can be measured and observed.
A synthesis of systems theory and the literature on organizational learning suggests that there are observable attributes about partnership development from which the construct of collaboration can be operationalized (Gajda, 2004; Hogue, 1993; Koliba & Gajda, 2009; Phillips, Lawrence, & Hardy, 2000; Thomson et al., 2009; Vangen & Huxham, 2003; Wood & Gray, 1991). These attributes include: (1) the sine qua non of collaboration is a shared purpose; partnerships form in order to address a shared problem or issue (McCann, 1983); (2) collaboration is a nested phenomenon that takes place within complex open systems (Granovetter, 1973; Stewart & Snape, 1996); (3) collaboration is developmental and evolves in stages over time (Doz & Baburoglu, 2000; Tuckman, 1965; Vangen & Huxham, 2003); (4) collaboration varies in terms of level and degree of integration (Phillips et al., 2000); and (5) interprofessional collaboration entails cycles of inquiry (Gajda & Koliba, 2007, 2008; Phillips et al., 2000).
The sine qua non of collaboration is shared purpose
The sine qua non of collaboration is a shared purpose—two or more entities (organizations or people) come together or stay together for a reason—to achieve a vision, or to do something that could not otherwise be accomplished in isolation. For instance, the AEA and CDC have partnered to deliver a professional development conference for CDC stakeholders each June in Atlanta, GA. Neither the AEA nor the CDC acting alone could deliver a conference of the scale, scope, and quality that they can when they pool resources and deliver it together. The Massachusetts Coalition for the Homeless is an advocacy organization that works to address the broad economic and social issues that lead to homelessness and involves a network of over 800 faith communities and 1,300 service providers and regional advocacy groups (http://www.mahomeless.org/). Distinct intraorganizational groups also form around a common purpose. For instance, the Marketing, Accounting, Sales, and Product Development departments of a local business; or the Pharmacy, Produce, and Deli departments of a large grocer each consist of individuals who come together around a specific task/domain within their respective organizations.
Collaboration is a nested phenomenon that takes place in a complex, open systems environment
A second principle of organizational collaboration is that it is a nested phenomenon that exists simultaneously at the interorganizational, intraorganizational, and interprofessional levels. As Vangen and Huxham (2003) state, “Collaborations are often characterized by complex hierarchies—which may be several layers deep—in which one collaboration is a member of another” (p. 17). There is large-scale interorganizational collaboration that may involve partnerships between federal or state agencies such as the Department of Education and Department of Health and Human Services, between professional organizations such the Association of State and Territorial Directors and the American Dental Association, or between many partners such as the Hawaii State Coalition Against Domestic Violence (HSCADV) comprised of nearly 20 domestic violence agencies and programs throughout the state. Nested within interorganizational collaboration exists intraorganizational collaboration, which includes partnerships that go on between groups within an organizational structure or system, for example, between cancer prevention and health promotion divisions within a state health department, between the colleges of engineering, education, and business at a single university, or between the English, mathematics, science, and social studies departments in a single high school.
There also exists interprofessional collaboration—that is, the presence of individual committees or teams within a singular system, and groups that act boundary spanners between organizations. For instance, the Public Policy Work Group at Blue Cross and Blue Shield of Florida involves officer and director representatives from all areas of the organization who collaboratively develop public policy positions that guide the business of the company. The leadership team of the HSCADV mentioned above is the central group of the coalition that serves as the link between the 20 community organizations working to prevent domestic violence. Ultimately, bricks and mortar do not collaborate, people do. It is through interprofessional collaboration within specific teams and groups where the lived dynamics of communication and action-taking around a shared purpose take place.
Collaboration is developmental
A third principle of collaboration is that partnerships go through predictable stages of development. Stages of collaboration development have been described in various ways, including “forming, storming, norming, performing, and transforming” (Tuckman, 1965; Tuckman & Jensen, 1977), “assemble, order, perform, and transform” (Bailey & Koney, 2000), and “potential, building, engaged, active, adaptive” (Gongla & Rizzuto, 2001). Regardless of the specific stage theory language used, it is important for evaluators and their stakeholders to understand that partnerships will need to navigate stages of development.
Collaboration entails a range of integration
A range of linkages or degrees of connection exist between agencies, within organizations, and between people. Collaborative efforts are often characterized as falling across a continuum, exhibiting various degrees of separation or levels of integration between organizations. Level of organizational integration is directly related to the purpose of the partnership; more complex and higher stakes purposes warrant greater degrees of connection (Vangen & Huxham, 2003; Wood & Gray, 1991). A basic and straightforward description of the various degrees of collaboration is offered by Peterson (1991) who postulates a three-point continuum of collaborative integration that begins with cooperation, whereby independent groups share information with one another, to coordination, whereby separate groups codeliver services or cosponsor events that support the missions of their respective organizations, to collaboration, where individual entities relinquish a greater degree of authority and control to combine structures in an effort to realize a shared outcome. Linkages at each level can be characterized by their shared mission, their organizational structures, and communication and decision-making processes (Hogue et al., 1995).
Interprofessional collaboration entails cycles of inquiry
Groups, teams, and committees of people who engage in interprofessional collaboration are “the basic building blocks of an intelligent organization” (Pinchot & Pinchot, 1993, p. 66). Regardless of a team’s name (e.g., History department, obesity prevention state coalition management team, smoking cessation working group, dental director core infrastructure team, school leadership committee, etc.), where members of the team are located (e.g., at the state health department, in a social service agency, university, etc.) or venue of team meetings (e.g. in person, online, teleconferencing) individuals must work together effectively if their organizations are to successfully adapt, grow, and achieve (Argyris & Schön, 1974, 1978; Pounder, 1998).
The most effective interprofessional collaboration entails an ongoing cycle of inquiry that includes: dialogue, decision making, action and evaluation around a shared purpose (Gajda & Koliba, 2007, 2008; Goodlad, Mantle-Bromley, & Goodlad, 2004) and is depicted in Figure 2. Dialogue is what facilitates a team’s ability to engage in what Preskill and Torres (1999) describe as a process of “reflection, asking questions, and identifying and clarifying values, beliefs, assumptions and knowledge … to guide subsequent behaviors” (p. 53). While low-functioning and nonrigorous forms of interprofessional dialogue tend to confirm present practices without determining their worth; high functioning teams make decisions and take subsequent actions to move the work of the group and the alliance forward (Ambrose, 1987). Finally, evaluation is considered a crucial component of a fully developed teams’ cycle of inquiry; high functioning groups assess their effectiveness on the basis of tangible evidence that they are reaching their goals (Senge et al., 1999).
Interprofessional cycle of inquiry.
In summary, five key principles of organizational collaboration are that (a) partnerships form around a shared purpose, (b) collaboration is a nested and networked phenomenon, (c) collaboration involves predictable stages of development, (d) collaboration exists at varying levels or degrees of integration, and (e) collaboration between people entails cycles of inquiry. These five principles can be utilized to guide choices in the planning and implementation of evaluation data collection methods and analysis and to strengthen collaboration development. It is rarely feasible to evaluate collaboration at every level within complex open systems. Obtaining a clear and theoretically grounded understanding of interorganizational dynamics can help the evaluator to determine and isolate the most appropriate and relevant dependent and/or independent variables related to the process and outcomes collaboration in his or her particular context.
In addition to raising one’s own literacy, evaluators should consider how to facilitate an increase in stakeholder understanding of collaboration. One means to do this is for evaluators to share and discuss literature about collaboration from the organizational context of the stakeholders (e.g., use of collaboration articles from the Harvard Business Review when working with corporate/non-for-profit entities, from the Journal of School Leadership when working in K-12 educational settings, and from the American Journal of Public Health when working with health practitioners, etc.). In addition, we have found the visual depiction and representation of the principles of collaboration, such as the cycle of inquiry shown in Figure 2, to be an effective means of raising shared literacy about the construct of collaboration. The process of operationalizing collaboration increases the capacity of program personnel to negotiate with evaluators about what aspects of collaboration are most useful, appropriate, important, and feasible to measure and assess as part of the evaluation plan.
The process of operationalizing collaboration can result in stakeholders making immediate adjustments and course corrections to their partnerships. For instance, the process of raising collaboration literacy will surface partner reasons for having entered or for considering entering into an interorganizational arrangement. As a result of recognizing that “the sine qua non of collaboration is shared purpose,” one potential partner in a 21st-Century Schools Initiative chose to decline participation in the collaborative, while two other agencies became more invested and allocated additional human resources to the partnership process. Program stakeholders with whom we have worked have reported “Ah-ha!” moments upon realizing that levels of integration in a partnership can (and will) vary depending on the shared mission and that “more integration is not necessarily better.” In one case involving an oral health coalition, collaboration literacy building prompted stakeholders to decide that a more effective use of technical assistance and federal funding was to strengthen interprofessional collaboration within existing partnerships, instead of “trying to increase the number of partnerships and up our number of signed MoUs.”
Operationalizing collaboration as an analytical construct, the first phase of the CEIF, is an important entry point into the evaluation of collaboration. Evaluator facilitated collaboration literacy building through the use of context-specific readings, rubrics, and visuals, will increase future capacity for targeted collaboration evaluation and development.
Phase 2—Identify and Map Communities of Practice
Because teams are the predominant unit for decision making and getting things done in any organization (Barnard, 1938; Dufour & Eaker, 1998; Pinchot & Pinchot, 1993; Senge et al., 1999), it is important to ascertain an accurate picture of the groups at work in an organizational partnership. The goal of Phase 2 of the CEIF is identify high-leverage groups: those teams and committees that are responsible for setting policy and carrying out the tasks and activities most central to the work of the partnership. A thorough inventory and mapping process can reveal: the teams and committees that make up the strategic alliance and/or organization; the purpose and primary task of each of these groups; the individual members of the groups; how often, where, and through what mediums each groups meets; how long each group has been in existence, and the relative importance of the group to the vision and mission of the organization/strategic alliance.
The most effective methods for identifying and mapping groups will depend on the scale and scope of the organization or program, the current degree of collaboration literacy, and preexisting channels of communication. Evaluators can generate a comprehensive list of teams and committees by examining organizational charts, surveying alliance membership, or some combination thereof. The goal is to create an accurate snapshot of who is working with whom throughout the partnership. For example, the first author of this article has done extensive work with school districts to evaluate the quality and effects of teacher collaboration, which has become an educational reform strategy in the United States (Dufour, Eaker, & Dufour, 2005; Jacobson, 2010). An example of the form and type of teacher collaboration data that can be generated through the mapping process is shown in Figure 3. This spreadsheet depicts all the teachers that are members of a single school, the various committees to which teachers’ belong, and the number of people on each team. In addition to the spreadsheet, these data were transformed and represented through simple visual formats using software such as Concept Draw Pro® and Word®. School administrators used the data to better understand who is working with whom, where and how the work of the school is getting done, and to inform their decision making about how to reconfigure committee membership to improve organizational performance.
Example teacher team inventory data.
Evaluators can also employ more sophisticated methods, such as social network analysis (SNA), to map organizational collaboration. SNA can be used to mathematically model the structure of organizational networks, how quickly and through what paths innovation transfers, and to identify key network actors (groups or individuals) who bridge or bottleneck knowledge transfer and innovation within the partnership (Deal Purinton, & Cook Waetjen, 2009). For example, the matrix of teacher collaboration inventory data shown in Figure 3 was analyzed using SNA. The matrix was imported into SNA software including Pajek (Batagelj & Mrva, 2008) and R (R Development Core Team, 2011) and used to generate sociograms to depict various patterns of connections between individual teachers and teams. Figure 4 is an example of a sociogram that mathematically reveals which teacher teams serve as hubs and which teams are on the periphery of the school’s task network. Administrators used these SNA findings to make decisions about how to reconfigure teams so that classroom teachers would be more connected to one another and more clustered at the center of the school’s collaboration network.
Example social network analysis (SNA) teacher team mapping data. Lines indicate connections between groups. Length of line denotes number of shared members between groups. Node size denotes size of team membership. Node color denotes content focus/name of the team.
Regardless of whether the data are collected and analyzed using simple spreadsheets (e.g., Figure 3) and pictures or through more complex mathematical process such as SNA (e.g., Figure 4), a systematic and thorough inventory and mapping process as described in Phase 2 of the CEIF can reveal findings that program stakeholders can use to determine which alliance members or groups might be overextended and/ or underextended and which committees might be too big or too small, and how to target next steps in the collaboration evaluation process. Furthermore, a thorough mapping and identification process will surface high-leverage teams: those groups that appear to focus on substantive issues related to the central goals of the partnership with the greatest capacity to precipitate positive change in the organization. When conducted over time, mapping data can be correlated with other measures to determine what patterns of collaboration yield the greatest outcomes. For example, teacher network patterns can be compared to longitudinal measures of school climate and student learning, tobacco prevention coalition networks can be compared to rates of youth smoking onset and prevalence.
Phase 3—Monitor Stages of Development
As discussed earlier in this article, a key attribute of organizational collaboration is that it goes through the predictable stages of development; partnerships will assemble/form, storm/order, norm/perform, and transform/adjourn. One stage may go by faster than another, an alliance find themselves stuck in a stage for a long time, or a group may find itself moving in and out of more than one phase at a time—but inevitably, partnerships need to navigate and emerge from each stage of development in order to successfully implement tasks and reach organizational outcomes. Evaluators can employ data collection and analysis strategies that will generate data that organizational leaders can use to make informed decisions about how to stimulate successful movement through the stages of development. One effective strategy is to conduct interviews with members of high-leverage teams that have been identified through the inventory and mapping process in Phase 2 of the CEIF. Interviews can be used to surface issues about collaboration quality to be isolated for special attention, correction, and improvement (Thomson et al., 2009). Below are questions that could be used as part of an interview protocol to monitor stages of development.
Assemble/formation stage
In this stage of collaboration, success often hinges on the level of shared clarity around purpose, structures, strategies, leadership, and key tasks. Suggested questions include:
How is/was the leadership identified?
How are/were members recruited and was enough time spent in the recruitment process?
How representative is the partnership membership with regard to its purpose?
Do leaders and members share a common understanding of the alliances’ purpose?
Does the group have the right people and organizations at the table?
Are leaders’ and members’ roles and responsibilities transparent and understood by all?
Does each member understand why they are they and what they are hoping to accomplish?
Are anticipated linkages between the members’ parent organizations and the alliance clearly delineated?
Storm/ordering stage
Once the alliance has been assembled, a critical developmental time period ensues. The conversation about the shared purpose of the alliance or community of practice tends to evoke feelings related to urgency, resources, turf, expertise, and each person’s willingness to take on tasks and responsibilities. Suggested questions include:
What is the purpose of this alliance, coalition, or community of practice?
What outcomes does the group expect to reach?
What are the primary activities?
What will indicate to the group that it is reaching its goals and outcomes?
Has the group established systems and norms for managing consensus and conflict?
Are policies and guidelines in place to achieve the group’s purpose?
Does the group have the appropriate bylaws, contract, or other agreements in place to govern its partnerships and activities?
How is information disseminated to members?
What systems are in place for the budgeting and distribution of resources?
What processes exist to address the issues of membership turnover?
How will new members be incorporated into the alliance?
Have informal leaders begun to emerge?
How are these leaders incorporated into the leadership group?
What benefits and costs does each member expect to accrue as a result of participating in the alliance, this project, and this particular community of practice?
To what extent do the benefits of participation outweigh the costs of membership?
Norm/performing stage
In transitioning from ordering to performing, the alliance focuses on safeguarding its resources and activities from external interference and strengthening (or rediscovering) its internal validity and creative energy in pursuit of the accomplishment of its goals. In this stage, stakeholders are actively implementing the various systems that have been established (e.g., communication, financial, personnel, and evaluation) and are executing the specific tasks and activities necessary to accomplish the alliance’s goals. Suggested questions include:
Do members understand their individual roles in the context of the alliance?
How have roles and responsibilities shifted over time?
How successful have members been in putting the goals of the alliance before their own or their organizations’ needs? (How have members balanced the needs of their home organization with that of the alliance?)
How effectively and/or efficiently are the alliance systems (e.g., information dissemination, resource allocation) working?
Do leaders and members acknowledge and address progress and setbacks?
How are requirements for additional or different resources identified?
How are data being used to inform decision making and to make mid-course corrections?
Are lessons learned used to amend the alliance structures, leadership, and/or process?
Transforming/adjourn stage
Over time alliances and teams will likely reach critical milestones and face planned and unforeseen events (such as the addition of a new member or leader). In the transformation stage, the organization and its members use data related to goals and outcomes to determine whether and how to refine, reconfigure, or dissolve their collaboration. As a result of this process, three possibilities will emerge: the group will choose to formally end; it will continue unchanged; or it will change any or all of its components. Suggested questions include: What goals have been accomplished, and how satisfied is the group with its performance? What activities have been carried out, and how satisfied is the group with these accomplishments? What events (both foreseen and unplanned) have had an impact on the group’s performance? What evidence does the group have about its accomplishments? How committed are each of the partners to the purpose of the alliance? Should membership change? If so how? How is the alliance transforming? What factors are precipitating the transformation? To what extent do the leaders, members, and external linkages agree with the decision to transform the alliance? To what extent does each member/partner believe the purpose of the alliance has been fulfilled? Should the alliance be adjourned? If so, when? If not, why not?
Face-to-face group interviews is an effective means for generating immediate and useful information that stakeholders can use to navigate progression through stages of partnership development. However, depending on the context of the evaluation, it may be more appropriate and/or feasible to use other methods to monitor and stimulate alliance development over time. For example, interviews can be conducted using virtual conferencing technology such as Skype®. In lieu of interviews partnership development questionnaires can be administered through online surveys that ask stakeholders to provide quantitative and qualitative information about their strategic alliances’ stages of development. Regardless of what particular approach is used, partnerships benefit from the systematic assessment of stages of development. As Hughes and Weiss (2007) attest, “Just as partners need to focus on building a strong working relationship at the start of an alliance, so they need to nurture that relationship throughout the life of the partnership” (p. 7). The monitoring of partnership development can enhance alliance capacity to realize their shared purpose.
Phase 4—Assess Levels of Integration
One of the fundamental principles of collaboration theory is that there are levels of integration that exist between and within organizations. More integration is not necessarily better—levels of organizational integration should vary according to the purpose and goals of the alliance. The LOIR 1 shown in Table 1 describes five levels of organizational integration (zero/none to four) and the purposes, strategies/tasks, leadership/decision making, and communication characteristics that tend to be present at each level of integration.
Levels of Organizational Integration Rubric
A rubric is a scoring tool that delineates specific criteria that can be used to promote a more standardized and transparent assessment of performance and behavior. Evaluators can use the LOIR to quantitatively and qualitatively gauge levels of integration over time. When collected in the assemble stage of alliance development, levels of integration data will provide a baseline against which future alliance development can be compared. In addition, such data can inform decision making about the most effective allocation of resources for future growth.
Organizational leaders with whom we have worked have utilized data generated through the LOIR process for annual performance reporting, project marketing, and to communicate issues related to sustainability to federal project officers, funders, health department administrators, and the public. A Massachusetts’ coalition of the state’s university-based nurse preparation programs and the state’s nurse licensing, nursing home regulation, and nursing home administration agencies formed a strategic alliance to increase patient safety and reduce nursing patient medication administration errors. Coalition leaders have credited the use of the LOIR as key to determining additional partners that needed to be at the table, for gaining clarity on what the most appropriate makeup of the leadership group should be, and for the attainment of the alliance’s short-term objectives (Anderson et al., 2011).
To assess levels of organizational integration using the LOIR, it is useful to convene alliance members and orient them to the rubric so that they can become familiar with the characteristics of organizational collaboration at varying levels of integration. Stakeholders will learn, or have it reinforced, that collaboration exists at many levels, and that linkages and relationships are defined by their purpose, strategies/tasks, leadership/decision making, and interprofessional/communication characteristics. Next, evaluators facilitate a process through which organizational representatives determine and record their current and ideal level of integration with each of the other partners in the alliance. During the numerical rating process, alliance members should also be facilitated in the qualitative description and recording of the actions they anticipate needing to take in order to move toward their ideal level of integration. Possible prompting questions for the qualitative component of the LOIR evaluation process include:
What would it look like if they reached their ideal level of integration?
What actions do they need and want to take to bring about their ideal level of integration?
What evidence would indicate that they have reached their ideal level of integration?
Quantitative ratings generated with the LOIR, descriptions of the ideal levels of integration, the planned actions to bring about ideal levels of integration, and a list of evidence that would indicate achievement of their ideal levels of integration should be recorded, collected, analyzed and reported.
The assessment of levels of organizational integration through facilitated group dialogue can take a substantial amount of time, space, and facilitation skills. Depending on the time and resources available, levels of integration could be assessed via online survey or a mailed questionnaire.
In addition to accessible and relatively straightforward tools such as the LOIR, evaluators can employ SNA to determine the nature and degree of relationships between partners. Further, systematically collected and analyzed data about levels of organizational integration can be measured longitudinally and looked at in conjunction with organizational outcome measures. For example, Cross, Dickman, Newman-Gonchar, and Fagan (2009) examined the existence, types, and levels of organizational linkages at work in a Safe Schools/Healthy Student Initiative (SS/HSI) and used SNA to visually depict how these linkages changed over time. In two SS/HS initiatives for which the lead author was the evaluator, changes in levels of organizational integration were tracked and compared to data generated through the Youth Risk Behavior Survey, school-based bullying prevention curriculum surveys, and other measures of child health and safety.
Phase 5—Assess Cycles of Inquiry
The quality of interprofessional collaboration that takes place within individual teams, committees, and groups has a profound effect on organizational performance. Evaluation of team cycles of inquiry builds stakeholder capacity for effective and efficient interprofessional collaboration—the building blocks of organizational performance. Gratton and Erickson (2007) concluded in a large-scale study of organizations that “Some teams had a collaborative culture but were not skilled in the practice of collaboration itself. They were encouraged to cooperate, they wanted to cooperate, but they didn’t know how to work together very well in teams” (pp. 105–106.) Evaluation of interprofessional collaboration helps program personnel avoid “collaboration lite” (lightheartedly referred to as coblaboration), make meetings more meaningful, strengthens the collaboration skills of group members, and maximizes performance. Unless the scale and scope of the partnership is very limited, it is typically not feasible to evaluate the quality of collaboration in every working group in an alliance. Evaluators can use the identification and mapping strategies described in Phase 2 of the CEIF to make decisions about which committees and teams are high leverage and warrant an in-depth examination of their cycle of inquiry.
Evaluators can assess the quality of team functioning using rubrics and survey tools such as the TCAR shown in Figure 5, available online at http://aje.sagepub.com/supplemental, which operationalizes elements of collaboration in team cycles of inquiry. Specifically, the TCAR describes in concrete terms the characteristics of dialogue, decision making, action, and evaluation at three levels of quality (2, 1, and 0).
Seven attributes of high-quality dialogue and decision making and six attributes of action and evaluation are described for a total of 24 separate criteria. Evaluators can adapt the TCAR to include more specific language detailing the purpose and practices of a particular group, and administer it in ways that are most feasible, useful, and appropriate for their particular program evaluation context.
Group members can self-assess their functioning using the TCAR and the data can be aggregated and analyzed to provide an overall snapshot of quality. Evaluators can also review team agendas and minutes and/or observe teams in action and then use the TCAR to rate quality of collaboration. Data generated through the analysis of team meeting documents can be triangulated with group member TCAR self-assessment ratings. The process and findings of assessing quality of interprofessional collaboration enables stakeholders to make evidence-based decisions about how improve team functioning. For instance, the evaluation of interprofessional collaboration in a state department of health–oral health unit revealed consistent weaknesses in dialogue. Evaluation findings (generated via TCAR self-assessment) resulted in group adoption of team norms and the use of a standardized format for creating and documenting meeting agendas and minutes.
When used longitudinally, TCAR data can be correlated with other measures to determine what patterns of collaboration yield the greatest outcomes. For instance, in an impact evaluation of a three-year district staff development initiative, TCAR data were used to statistically determine the relationship between teacher collaboration, quality of instruction, and student achievement. Correlational and multiple regression analyses were conducted to examine the relationships between achievement outcomes (measured by performance on the state’s annual standardized assessments) and TCAR data (Zito, 2011).
Conclusion
The ability to collaborate on both a large and small scale is one of the core requisites of post-modern society … in short, without collaborative skills and relationships it is not possible to learn and to continue to learn as much as you need in order to be an agent for social improvement (Fullan, 1993, pp. 17–18.)
In this article, we have presented the CEIF Framework, which is meant to provide evaluators with a theoretically grounded and actionable framework for approaching the task of evaluating and improving organizational collaboration. The CEIF presents five entry points to consider when engaging in the work of evaluating collaboration, including: (1) Operationalizing the construct of collaboration: the evaluand of collaboration may be characterized by specific attributes and variables so that its’ existence, development, quantity, and/or quality and effects can be measured, observed, and/or otherwise documented; (2) Identifying and mapping alliance teams and groups: because teams are the predominant unit for decision making and getting things done in any organization, it is important to ascertain a clear and accurate picture of the high-leverage groups at work in a strategic alliance; (3) Monitoring stage/stages of development: All partnerships will navigate predictable stages of development; monitoring the stages can enhance organizational performance; (4) Assessing levels of integration: Data about degrees of organizational integration can inform decisions about appropriate allocation of resources; when used longitudinally, integration data can be correlated with other important outcomes measures; and, (5) Assessing cycles of inquiry in high-leverage teams: Assessment of dialogue, decision making, action, and evaluation generates findings that can be used to inform decisions about how to strengthen the interprofessional collaboration and builds interorganizational capacity for efficiency and performance.
The CEIF incorporates the use of qualitative and quantitative data collection strategies and measurement tools appropriate in a variety of evaluation contexts and at any stage of alliance development. Specific tools such as the LOIR and the TCAR and strategies such as monitoring alliance development will generate findings that stakeholders can use to make immediate and ongoing evidenced-based decisions about how to improve the quality of their partnerships. Further, when collected over time, collaboration evaluation data can be analyzed in relationship to indicators of essential organizational outcomes.
Organizational collaboration has become a widely championed strategy for addressing complex social issues in the 21st century, and as collaboration scholars Thomson et al. (2009) note, “an idea that carries considerable rhetorical appeal” (p. 51). Evaluator use of the Collaboration Evaluation and Improvement Framework can help make the rhetoric more of a reality.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
