Abstract
A method called multi-attribute utility analysis (MAUA) provides a decision-making framework that facilitates comparative analysis of multiple real-world decision alternatives with unique complex attributes. Utility analysis as a measure of effectiveness has been minimally used by educational researchers to date, despite clear relevance in complex decision-making. To illustrate its viability, the application of MAUA was modeled for two example academic programs with diverse partnership priorities as a form of assessing academic–clinical partnership alignment. Simulated application indicates MAUA may be successfully utilized as an evidence-based methodological framework. The presented example is illustrative of the wide-spanning potential for this approach in different contexts, as predicted and recommended by experts in the field. Evaluators are encouraged to collaborate in new ways and strive to produce tangible, solution-oriented approaches to address key challenges and demonstrate the value of sound evaluation practices.
Keywords
A Potential Evaluation Framework: Multi-Attribute Utility Analysis
A method called multi-attribute utility analysis (MAUA) (Levin et al., 2018; Sarin, 2013), which is rooted in multi-attribute utility theory (MAUT), provides a decision-making framework that facilitates comparative analysis of multiple alternatives with unique complex attributes (Dyer et al., 1992). This approach offers a process for identification and quantification of relevant options in the absence of a shared metric. Multi-attribute utility analysis is needed to handle the complexity of real-world decisions involving multiple decision criteria. In the absence of MAUA, decision makers rely on heuristic techniques in an attempt to expedite practical, efficient decision-making (Wallenius et al., 2008). However, heuristic approaches may be overwhelmed when the list of decision criteria is long and there is not a simple way to identify value preferences or reconcile those in conflict, given that there is typically no single effectiveness measure (Levin et al., 2018; Wallenius et al., 2008).
The MAUA approach elicits a decision-maker's preferences and values to identify and rank priority attributes, simplifying the task of arriving at a rational decision and reducing inconsistencies in decision-making (Dyer, 2005; Levin et al., 2018; Wallenius et al., 2008). A utility function is applied to clarify the relative value of each attribute, and an overall utility is derived for each option by combining attribute utilities. The decision-maker compares this overall measure of effectiveness across options and reviews the trade-offs to determine the most preferable option (Kailiponi, 2010; Levin et al., 2018; Wallenius et al., 2008).
Levin et al. (2018) identify three levels of stakeholder engagement with MAUA. It is theoretically most preferable for the entire community to be involved in the decision-making process. However, group selection should be aligned with the intent of the evaluation or research. In some cases, only the population directly impacted by a decision may be included, and in others, a smaller representative group of select stakeholders may be most appropriate. Levin et al. (2018) indicate that if ranking and weighting preferences vary significantly across groups, then the overall utility measure may not be valid. The stakeholders engaged in MAUA must be intentionally and comprehensively selected to promote successful decision-making.
A concrete example of MAUA utilization in a real-world context is presented later in this report to clearly demonstrate the application of its methodological steps and the transition of idiosyncratic value metrics to a common utility metric. To preface this simulation, the four clearly defined steps for conducting MAUA must be understood (Levin et al., 2018). The first is to operationalize the set of attributes that best represent priority considerations in the decision-making process (Levin et al., 2018). This is an iterative process that involves both grouping attributes under broad categories and subdividing those categories into explicit attributes. Abbas (2018) recommends choosing attributes with meaningful scales to avoid misinterpretation by others, avoiding arbitrary scores and probabilities as measures. Abbas (2018) also advises careful consideration of whether attributes are direct (a key priority to compare) or indirect (related to a key priority but only secondarily so) values, including only direct values in the decision-making process. This may be determined through root cause analysis techniques (Voehl, 2016), beginning with an initial value statement and repeatedly asking “why?” until the core direct value remains. Another approach is to assess whether a given value would remain important if other attributes were fixed regardless of the decision (Abbas, 2018).
The second step is to determine the single utility, or value, for each attribute (Levin et al., 2018). Because attributes are measured in their unique inherent units, a common utility scale is needed to compare the relative strengths of preference. The low and high values on a single attribute scale are assigned as the bottom and top values on an arbitrary utility scale (e.g., 0–100). Remaining attribute score conversions are performed via one of three methods. The proportional scoring method assumes a linear relationship so uses a linear formula to simply rescale attribute scores onto the utility scale. The direct method asks individual stakeholders to rate their utility preferences for each score on the attribute scale relative to the chosen utility scale endpoints, independently of their other attribute score preference ratings. The variable probability method uses a similar approach, but individuals rate attribute scale scores on the utility scale relative to each other.
The third step in conducting MAUA is to assign importance weights to each attribute (Levin et al., 2018). This step can occur using different variations of the direct method or variable probability method. In the direct method, individuals allocate a total of 100 points among the identified attributes based on their perceived relative importance. The importance weights of all attributes within the overall utility sum to 1.0. The variable probability method is similar, but individuals are asked to choose between two options until the utility balance is determined.
The fourth step in the MAUA process is to calculate the weighted utility of each attribute (Levin et al., 2018). The importance weight for each attribute is simply multiplied by its calculated utility score. The final step in the process is to apply the multi-attribute utility function using the additive approach to calculate the total utility, or effectiveness, of a given option. This is done by summing the weighted utilities across all attributes included for that option (Levin et al., 2018). The final utility scores across multiple decision alternatives may then be compared. This process for identifying and evaluating complex priorities allows effective decision-making that otherwise would be challenging to achieve.
Across fields, scholars consistently conclude that MAUA serves as a flexible yet objective measurement framework. The methodology has been utilized successfully in a variety of applied fields, including public health (Selameab & Yeh, 2008), engineering (Abbas & Sun, 2015; Alencar & de Almeida, 2015), public infrastructure and management (Arif et al., 2016; Dabous et al., 2020; de Almeida, 2012; Eweda et al., 2015; Garcez & de Almeida, 2014; Kaddoura et al., 2018; Mohamed & Zayed, 2013; Rochat et al., 2013; Vucijak et al., 2016), transportation (Ramani et al., 2011; Rios Insua et al., 2019), law (Trevino, 2018), and industry safety (Beaudouin, 2015; Dee et al., 2019). Positive outcomes of beneficial partner identification are also reported when applying MAUA to partnership selection processes in areas such as branding alliance (Chang, 2008), virtual enterprise (Sha & Che, 2005), and private–public partnerships in city projects (Lam & Yang, 2020).
Expert scholars in the field of decision analysis have identified a growing importance in the behavioral components of decision-making (Wallenius et al., 2008). They anticipate that as such, multi-attribute decision-making approaches will continue to expand into different practice fields (Wallenius et al., 2008). Utility analysis as a measure of effectiveness has been minimally used by educational researchers to date, despite clear relevance in complex decision-making (Levin et al., 2018). Levin et al. (2018) assert that if MAUA has not yet been used as a measure of effectiveness in a given field, then researchers may need to execute their own data collection to determine priority attributes and relative weights.
Application of MAUA in Partnership Evaluation
In the field of educational evaluation, scholarly exploration is needed to assess and strengthen academic–clinical partnerships. Strong partnerships between academic institutions/programs and community/clinical settings, termed academic–clinical partnerships, are essential in providing quality training experiences to learners (Jensen et al., 2017; Mehigan et al., 2019). Strong partnerships allow both parties to contribute expertise and resources to achieve more together than they could accomplish on their own (Buffet, 2018). For example, academic–clinical partnerships in health professions education have been shown to improve learner performance in safety and subsequently improve patient outcomes (Bvumbwe, 2016). To maximize outcomes, both partners must recognize their shared purpose to provide quality care to and improve the health of the communities served (Applebaum et al., 2014).
What is unclear, however, are the precise conditions and factors that promote strong academic–clinical partnerships. While researchers have begun to explore the general outcomes resulting from academic–clinical partnership engagement (Davies et al., 2016), there is scant information regarding how one might objectively assess the quality of an existing academic–clinical partnership (Brinkerhoff, 2002; Mayer et al., 2017). A limited number of studies drawn from the fields of evaluation (Brinkerhoff, 2002), government (Charles et al., 1998), health professions education (Jensen & Royeen, 2001; Nabavi et al., 2012), and community-based participatory research (Duran et al., 2019; Mayer et al., 2017; Mayo-Gamble et al., 2017) highlight the paucity of current evidence in partnership evaluation as a gap in knowledge.
There is a need to investigate the application of systematic, scholarly methods to the problem of understanding and weighting academic program values and priorities regarding academic–clinical partnerships. First, it is not known which partnership factors academic programs prioritize. Such constructs have not been clearly operationalized to date and there is no model by which an individual program may assess its priorities. Second, the inherent variability in academic program missions, resources, operations, learner needs, and faculty skills and experiences suggest that partnership priorities are likely not the same for every program. Third, any given combination of factors in an academic–clinical partnership may be viewed as highly effective by one program or partner but ill-fitting by another. A partnership therefore cannot be rated broadly as “good/bad” or “desirable/undesirable” given the degree of subjective interpretation involved.
This contextual variability across partnerships, which is expected and appropriate (Wheeler et al., 2018), precludes the ability to generate a shared set of measurable characteristics contributing to partnership effectiveness. Any potential evaluation framework must, therefore, be flexible and adaptable to a given partner's unique priorities and allow direct comparison of specific factors and conditions across partnerships.
Application of MAUA concepts to academic–clinical partnership evaluation was piloted in one academic health profession program with initial success (North & Sharp, 2022). This Doctor of Physical Therapy clinical education team sought the ability to quantitatively evaluate partnership data to determine the extent to which existing clinical partners supported the program's goals (i.e., partnership effectiveness). Because literature in the health professions did not offer a systematic process to conduct this evaluation, the team applied select MAUA concepts as an early exploration of the feasibility and utility of MAUA in this new context. The pilot evaluation processes and outcomes detailed by North and Sharp (2022) resulted in the use of an institution-specific Clinical Partner Prioritization Rubric © (CPPR) to guide initial actions to improve overall partner effectiveness. Because the full MAUA evaluation process was not conducted in the feasibility study, the authors indicated that further research is needed to assess the utility of MAUA for academic–clinical partnership evaluation across institutional contexts.
A process is needed to identify and quantify priority values within an institutional context and use findings to make decisions and render judgment in academic–clinical partnerships. An absence of such guidance leaves partners ill-equipped to consider the effectiveness of their partnerships due to difficulty evaluating whether the factors and conditions present across each partnership are optimal or in need of improvement. The purpose of this report was to examine the extent to which MAUA may be applied as an existing, evidence-based methodological framework to perform values-based partnership evaluation.
Simulated Application of MAUA in Partnership Evaluation
Given the complexity of real-world variability in decision-making regarding partnership prioritization, the application of MAUA to academic–clinical partnership evaluation requires each academic program to conduct its own program-specific calculations. The proposed evaluative framework blends a shared methodological approach for quantifying subjective partnership attributes with a flexibility for each academic program to align the process with their contextual values-based priorities. The analysis in this study applied the steps in conducting an MAUA as described by Levin et al. (2018).
To illustrate its viability, the application of MAUA was modeled for two example academic programs with diverse partnership priorities, presented in Table 1. Program A is a large urban institution committed to addressing state health and wellness needs, with 40 students per cohort, while Program B is a small suburban institution serving a high volume of nontraditional students, with 80 students per cohort. Their interests and therefore priority partnership characteristics vary accordingly.
Two Diverse Profile Examples of Academic Clinical Education Program Priorities in Academic–Clinical Partnerships.
Table 2 presents the characteristics of three example clinical partners (Clinical Partners A, B, and C), hypothetically shared by the two examples Academic Programs A and B. Given their diverse partnership priorities, it would be expected that example Programs A and B would evaluate the characteristics of Clinical Partners A, B, and C differently and reach different conclusions regarding the effectiveness of each partnership. However, the steps to determine such comparative evaluation outcomes and intentionally align partnership initiatives have not been demonstrated to date.
Three Diverse Profile Examples of Clinical Partners Engaged in Academic–Clinical Partnerships with Academic Programs.
The first step of simulated MAUA application to academic–clinical partnership evaluation was to operationalize the attributes that represent priority considerations in academic–clinical partnership evaluation. For simplicity, simulated Programs A and B utilized the comprehensive list of 48 values-based prioritized attributes generated in a dissertation study (North, 2022) informed by results of the pilot study applying MAUA in clinical partnership evaluation (North & Sharp, 2022). It is assumed that academic program added, modified, or removed any of the attributes in the list.
In the second step of applied MAUA simulation, the single factor utility, or value, for each attribute was calculated. Individual attribute scales were determined following the identification of prioritized characteristics in the first step of MAUA. For example, an attribute for “clinical partner loyalty/reliability in supporting the academic program's placement needs” may be measured on a scale from 0 to 3, representing high (3), medium (2), low (1), or no (0) loyalty/reliability. Given the nominal nature of most attributes, simple scales were assigned based on the author's expertise. Scales were framed with the assumption that each attribute was presented in desirable terms, with higher points allocated to a “yes” or “high” option and fewer or no points assigned to a “no” or “low” option. In subsequent steps of MAUA, the simulated academic programs may choose to indicate 0% preference for any attributes they find undesirable.
The values placed on each attribute were assigned by Academic Programs A and B based on their perceptions of the specific clinical partnership they are evaluating. For example, in the scenario above, a program would assign a score of 1 to a clinical partner perceived to have a low level of loyalty/reliability in supporting clinical placement needs. However, given the variability in measurement units for individual attributes, conversion to a common utility scale was needed to compare the relative strengths of preference across attributes. To convert measurements from their inherent attribute units, the low and high values on a single attribute scale were assigned as the bottom and top values on an arbitrary utility scale. While a scale of 0–100 is commonly used, any scale values may be used to complement the needs of a given analysis. Because the values for all individual attribute scales in this simulation were factors of 12, a common utility scale of 0–12 was selected for ease in calculations.
Score conversions were performed via the proportional scoring method using a linear formula to simply rescale attribute scores onto the utility scale:
The third step of MAUA requires academic programs to assign importance weights to attributes based on their unique partnership priorities. In this simulation, Program A and Program B both used the direct method to first allocate a total of 100 points among 10 categories based on their perceived relative importance. Any categories deemed not important were weighted with a value of 0 of 100 points. The importance weights of all attribute sets within overall utility initially summed to 100. These weights were then converted to fractional weight equivalents that sum to 1.0. The two simulated academic programs were provided with a set of sample aggregate mean weighted values for each partnership category as an example, drawn from the aforementioned dissertation study (North, 2022). Each program then increased or decreased the proposed values to best reflect their program priorities. For example, if the example weighted value of the category “Practice Setting” was 0.22, a program may decide to raise the value to 0.26 (an increase of 0.04) to place greater importance on that category of partnership characteristics. Because the final summed weighted value across all categories must still total 1.0, the program would need to reduce the weighted value by 0.04 across one or more other categories.
The weighted importance selected by Programs A and B for each category is presented in Table 3 (bolded partnership category title rows). Program A, with 40 students per cohort at a large urban institution in Minnesota, expressed commitment to state health and wellness needs. Their clinical education faculty placed the greatest value on the partnership categories Placement Volume (0.15), Placement Level (0.14), Mission/Values (0.13), and Relationship (0.13).
Importance Weights Assigned to Academic–Clinical Partnership Categories and Attributes by Two Simulated Academic Programs.
Program B has 80 students per cohort at a small suburban institution and serves a high volume of nontraditional students. They placed the greatest importance on the categories Practice Setting (0.17), Placement Volume (0.15), Placement Location (0.14), and Administrative (0.12).
Programs A and B then allocated their chosen priority weighted value for each category across the attributes within that given set, based on their perceived relative importance (Table 3, nonbolded partnership attribute rows). The weighted values for the attributes in each category summed to the weighted value assigned to that category. For example, Academic Program A weighted the category “Cost” as 0.05 out of 1.0, allocating 0.03 to the attribute “Low cost to the academic program for placement/site requirements,” 0.01 to the attribute “Low cost to the student for placement/site requirements,” and 0.01 to the attribute “Placements with free or reduced cost housing for students.” The weighted values for the three attributes in this category sum to the weighted value assigned to the category (0.03 + 0.01 + 0.01 = 0.05). The values assigned to each attribute are termed importance weights. This was repeated for every category. The total sum for all attributes’ importance weights across all categories was 1.0.
In the fourth step of MAUA, simulated Programs A and B considered each specific partnership and calculated the weighted utility of each attribute for each clinical partner by multiplying its utility score on the common utility scale (0–12) by its assigned importance weight (0–1.0). The single factor utility for each attribute was calculated in the second step of MAUA, and the importance weight for each attribute was calculated in the third step of MAUA. As an example, the calculated weighted utility of an attribute with a single factor utility of 20 and an importance weight of 0.15 would be calculated as (20 × 0.15) = 3. Weighted utility scores are presented for Program A (Table 4) and Program B (Table 5) for partnership attributes of Clinical Partners A (column 5), B (column 8), and C (column 11).
Academic Program A Weighted Utility Scores Calculated for Clinical Partners A, B, and C.
Academic Program B Weighted Utility Scores Calculated for Clinical Partners A, B, and C.
In the fifth and final step, the multi-attribute utility function was applied using the additive approach to calculate the total utility, or effectiveness, of each of the three academic–clinical partnerships. This was done by summing the weighted utilities across all attributes included in that partnership. For example, the weighted utilities of 10 attributes may sum to (2.8 + 3.1 + 0.5 + 6.2 + 7.1 + 4.4 + 3.8 + 1.1 + 0.8 + 2.2) = 32. Summing the weighted utilities across all attributes, the partnership scores for Program A were 9.38 for Clinical Partner A, 6.72 for Clinical Partner B, and 4.12 for Clinical Partner C. The partnership scores for Academic Program B were 5.66 for Clinical Partner A, 9.9 for Clinical Partner B, and 3.46 for Clinical Partner C (Tables 4 and 5).
Implications for MAUA Application in Academic–Clinical Partnership Evaluation
Simulated application indicates that MAUA may be successfully utilized in the field of education as an evidence-based methodological framework for values-based partnership evaluation.
Simulation findings demonstrated that the MAUA methodology supported identification of a number of notable practical differences in partnership attribute priorities based on academic program preferences. For example, in MAUA step 2, the location attributes for simulated clinical partners A, B, and C were scored differently for each academic program based upon geographic proximity to the academic institution. The greatest differences in partnership prioritization began to appear in MAUA Step 3, in which simulated Academic Programs A and B demonstrated distinct differences in their priority weights across categories and attributes, based upon their institutionally specific partnership priorities. It is important to note that even if overall category weightings were similar or the same between Academic Programs A and B, the suballocations assigned to attributes within each category allowed each academic program to further differentiate their contextual priorities (refer to Table 3). For example, though both simulated academic programs allotted 0.15 importance weight to the partnership category Placement Volume, Academic Program A allocated greatest importance to the category attribute “Offers placements slots every year” while Academic Program B allotted greatest importance to the attribute “Provides a high number of slots.” This demonstrates the flexibility of the MAUA methodology to allow a partner to adapt their scoring to the institutional context. Differences in partnership prioritization between simulated Academic Programs A and B further increased in MAUA step 4 as both the individual attribute utility score and importance weight were factored into total attribute scores for simulated clinical partners A, B, and C.
The final numerical partnership score calculated in MAUA step 5 represents the values-based effectiveness of an academic–clinical partnership from the perspective of the academic program. Given that higher scores indicate a greater degree of alignment between the academic and clinical partners in practice, the MAUA methodology enabled Academic Program A to quantify Clinical Partner A as its most effective partner (score 9.38 of 12) and Academic Program B to quantify Clinical Partner B as its most effective partner (score 9.9 of 12). Clinical Partner C was quantified as the least effective partner for both Academic Programs A and B (score 4.12 and 3.46 of 12, respectively). It is within reason to expect that a third simulated academic program may score Clinical Partner C as most aligned with their partnership priorities and Clinical Partners A and B as lower in alignment. Results demonstrate that it is not practical to expect that a partnership evaluation tool with standardized scoring could yield a single partnership score applicable to all academic programs.
The evaluation process was both simplified and more consistent across clinical partners by using the MAUA methodological framework. Without such a methodology, the academic programs would be left to consider a given partnership as a whole, unable to weight or evaluate different attributes. Further, the quantified values placed on individual partnership attributes enabled each academic program to identify and justify strategic interventions to improve low scores, enhance high-scoring partnerships, or discontinue a partnership.
Multi-Attribute Utility Analysis as a Methodological Framework in the Field of Evaluation
The use of MAUA as a methodological framework in the field of evaluation extends beyond the health professions academic–clinical partnerships simulated in this report. The presented example is illustrative of the wide-spanning potential to use this approach in different contexts, as predicted and recommended by experts in the field (Levin et al., 2018; Wallenius et al., 2008). It is likely that any academic–clinical partnership, in any context, would benefit from analysis of the many context-specific factors contributing to partnership alignment or challenges. Further, any evaluation scenario requiring the identification and quantification of relevant options in the absence of a shared metric may benefit from MAUA as a decision-making framework that facilitates comparative analysis of multiple alternatives with unique complex attributes (Dyer et al., 1992). Multi-attribute utility analysis may be particularly well positioned as a methodological approach to evaluating system interdependencies that are partner-coalition-collaboration based (Renger, 2022; Renger et al., 2017; Souvannasacd et al., 2022).
As such, it is important to consider the theoretical framing of this evaluative approach in the context of evaluation theory and literature. Each theory typically guides a select aspect of an evaluation, such as the methods, values, intended use, or philosophical framing. Three theories are especially relevant in the context of academic–clinical partnership evaluation.
First, values-engaged theory (Greene, 2012, 2013; Hall et al., 2012) posits that values are at the root of decision-making in the evaluation process. There should be equity in access to the evaluation with all stakeholder perspectives considered. Some elements of this theory may inform the study of academic–clinical partnerships: (1) direct engagement with stakeholders will produce a deeper understanding of the contextual needs and interests of the program; (2) the evaluator's positionality must promote authentic engagement and trust of the participants; (3) context plays an important role in the evaluation and its interpretation; (4) the relevance, utility, and perceived impact of the evaluation should be considered; and (5) the subjective nature of values should be embraced through dialogue and a mixed-methods approach.
However, values-engaged theory adds a prescriptive element that predetermines the priority values rather than allowing stakeholders to identify their own values. The approach does not outline specific methodological steps and does not allow stakeholders to complete assessments themselves, dependent instead upon the evaluator's interpretation. Feasibility is, therefore, a challenge for a larger volume of study participants due to the demands of qualitative inquiry (Miller, 2010). Reproducibility would also be difficult in the case of more than one evaluator. Campbell's validity taxonomy (Shadish et al., 2002) identifies additional challenges for this approach in analytical, internal, construct, and external validity. Analytic validity is threatened by unreliability of measures, unreliability of implementation, extraneous variance in program settings, and heterogeneity of participants. Other threats to validity include variance across settings, complex social history, and lack of values operationalization. Overall, values-engaged theory offers useful philosophical considerations in research design but does not provide a solid framework for the study of academic–clinical partnerships.
A second theory considered from evaluation literature is realist evaluation (Astbury, 2013; Pawson, 2006). This approach seeks to understand and evaluate programs based on the formula “Context + Mechanism = Outcome.” Some elements of this theory are aligned with the study of academic–clinical partnerships: (1) Realist evaluation is focused on identifying the most impactful priorities from the many contextual elements present for a specific program; these factors may enable or constrain program operations and are interwoven to include individual, interpersonal, and system-level considerations; (2) realists look for patterns in these variable combinations; (3) collective analyses from multiple sources are preferred over an individual evaluation as a better indicator of “good” methodology and practice; and (4) the results of the variable interactions in different contexts and mechanisms define the outcomes, seeking an explanation for why certain outcomes occurred rather than whether they occurred. The realist principle of abstraction, a conceptual approach used to connect variable language across evaluations, is applicable for generalizability across partnerships.
However, realist evaluation views the “mechanisms” component of the formula as elements of a program that define its success based on participant change, rather than considering the logistics (processes) of program operations. The interpersonal and system-level factors are therefore reduced to focus on the outcomes of the individual. The originating theorist admitted that the principles outlined for realist evaluation are very difficult to achieve and do not result in solid knowledge because they are so complex within a given context (Astbury, 2013). There are no specific evaluation tools to achieve the realist principles or “context + mechanism” outcome variations, and there is no clear methodology outlined in the form of actionable steps. Operational specificity is lost in the absence of guidance for practitioners regarding when, how, and what evaluation questions are pursued (Miller, 2010). Additional challenges for realist evaluation as critiqued using Campbell's validity taxonomy (Shadish et al., 2002) include threats to analytical validity due to unreliability of measures, unreliability of implementation, extraneous variance in program settings, and heterogeneity of participants. Internal validity is significantly impeded by the lack of consistent methodology, precluding the ability to control history, selection, testing, and instrumentation threats at a minimum. Without internal validity, there can be no claim of external validity, as there is no control over interactions with proposed causal relationships at the subject, intervention, or outcome levels. Overall, realist theory is philosophically aligned with academic–clinical partnership evaluation but does not provide the necessary rigor as an evaluation framework.
Utilization-focused evaluation (UFE) is the third approach considered from the field of evaluation (Patton, 2012; Ramírez & Brodhead, 2013). Studies employing a UFE design are primarily focused on whether and to what extent the outcomes of a study will be useful in practice, as determined by the primary stakeholder or intended “user.” As with the previous two theories, UFE demonstrates some conceptual alignments with partnership evaluation: (1) the intent of the evaluation is to explore what works and what doesn’t work based on usefulness to the stakeholder (user); (2) the stakeholder (user) is responsible for conducting and interpreting the evaluation, which promotes ownership and investment in the process; (3) the role of the evaluator is to create a learning process and serve as a facilitator; and (4) the evaluation is used to guide decision-making.
However, in practice, UFE has significant limitations as well. The theory specifies 17 prescriptive steps in the evaluation process but lacks a defined methodology. Utilization-focused evaluation is meant to be learned through practice, but the 17 steps are abstract and cumbersome to define and implement. As an example, the proposed checklist for UFE evaluations includes the theoretical premises, primary tasks, and evaluator (facilitator) challenges for each step but totals 19 pages (Patton, 2012, pp. 406–424). Busy academicians and clinicians likely do not have the capacity to invest in this formal process. Additionally, because academic–clinical partnership evaluation has not historically been conducted in the field of education, the programs (users) on their own may not be fully aware of the options or parameters for applied use of the evaluation outcomes, violating a key premise of UFE. They may first require additional modeling or the opportunity to interpret their results to determine their unique desired actionable responses. Applying Campbell's validity taxonomy (Shadish et al., 2002), analytic validity is threatened by unreliability of measures, unreliability of implementation, extraneous variance in program settings, and heterogeneity of participants. Internal, construct, and external validity are threatened by complex history interactions in partnerships, lack of consistent instrumentation, absence of defined constructs, and interactions between setting-specific partnership elements and partnership outcomes. Overall, UFE aligns with the practical, stakeholder-driven element of partnership evaluation but does not provide an efficient, guided process by which programs may assess their partnerships.
Each of these three theories from evaluation literature has the potential to offer limited insight into one component of the conceptual design of partnership evaluation studies. Principles drawn from each—stakeholder values exploration, realistic applications within a program's context, and a utilization-focused mindset—may in combination be used as a practical set of philosophical considerations in research design. Collectively, these theories suggest that a study should pursue stakeholder values exploration within a specific context with direct involvement of the stakeholders and their self-defined needs in the evaluative process.
However, even together, these theories demonstrate significant constraints in analytic validity due to the lack of consistent instrumentation and methodology available to academic programs. The additive and interactive effects of the threats to internal validity described above leave programs unable to consider priority partnership factors within their context to make partnership judgments and decisions. They also do not acknowledge the interconnectedness between the many partnership factors and conditions, specifically the human choice variable found in decision-making processes. Programs are in need of guidance via a sound methodological process to mitigate construct confounding because a failure to thoroughly define and consider all constructs may result in poor interpretation of evaluation outcomes.
This study serves as an early foundation for a series of studies exploring the applicability of the MAUA methodology in academic–clinical partnership and other complex evaluations. Future research related to academic–clinical partnerships may pursue stakeholder engagement in MAUA methodology in a values-based context through contemporary, applied exploration of feasibility, priorities, and implications for education and practice within a single health profession, across multiple health professions, and beyond to also include other forms of academic–clinical partnerships. This line of research may lead to the development of a shared approach to measure subjective partnership attributes across multiple academic programs within a given profession and over time. There is tremendous potential for such approaches to be consistently implemented for academic–clinical partnership evaluation across many health profession programs, and beyond to other types of community and service partnerships, allowing peer comparison to drive quality improvement initiatives and promoting overall quality in these vital educational relationships. The application of MAUA in systems evaluation also holds tremendous potential and highlights the flexibility of this methodological approach.
In certain cases, this approach may not be feasible. In addition to its benefits, MAUA has limitations (Jansen, 2011). First, the approach assumes that each factor is considered independently of other factors when determining its weighted value. Instead, a decision-maker may find that they would prioritize one factor differently in the presence or absence of another factor, or that they would prioritize a factor only when it falls within a certain value range but not when outside of that range. This factor independence may conflict with the systems evaluation assumption of attribute interdependence, such that additional strategies may be necessary to apply the MAUA framework in a systems evaluation context. Second, the priorities determined through the MAUA process may not ultimately reflect the decision if the constraints in selection outweigh the desired factors. Third, MAUA assumes that the decision-maker is rational, but the human element in decision-making may override the practical prioritization due to intuition or emotion. Finally, the structured methodology of MAUA poses potential drawbacks in the ability of stakeholders to navigate the prescribed steps in the face of resource constraints such as time, training, or motivation.
Multi-attribute utility analysis holds tremendous promise for the continued advancement of applications in a variety of fields. It is the responsibility of educational researchers to propose and establish innovative connections between areas of evaluation theory and practice, across a variety of fields and disciplines, to promote applications that enhance impact in the field. Evaluators are encouraged to collaborate in new ways and strive to produce tangible, solution-oriented approaches to address key challenges and demonstrate the value of sound evaluation practices.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
