Applying the eight principles of developmental evaluation: Complex Care Hub as a case example

Abstract

Evaluators learning about developmental evaluation (DE) may struggle with the need for guidance on the concrete specifics of how to “do developmental evaluation” because of the need to tailor the use of the method to the context of the program being evaluated. Case examples in the literature provide illustrations of how others have applied this approach. This article adds to this growing body of knowledge by detailing the experience of the first year of a multiyear developmental evaluation of Complex Care Hub. This healthcare program is a hospital-at-home model where patients receive hospital-level services but sleep in their own home and receive case management as needed. The article ends with a discussion of challenges and lessons learned.

Keywords

Complex Care Hub complexity and evaluation developmental evaluation hospital-at-home innovation

Developmental evaluation (DE) as a formalized approach has developed over the past decade (Dozois et al., 2010; Gamble, 2008; Patton, 2011; Patton et al., 2016). Its relevance as a modality is increasing in response to increasing complexity in programs, organizations, and society. Healthcare is an environment with abundant complexity. Healthcare research and evaluation often involve intensive stakeholder engagement, process development, and adaptation to an evolving evidence base to improve services or produce new models of care. Traditionally, health care based its evaluation primarily on controlled (e.g. experimental) and time-bound approaches as the gold standard. However, these approaches are not always agile enough to provide teams with rapid feedback when innovating or navigating complexity.

While DE is now accepted among evaluators, in our experience it remains a new frontier for healthcare stakeholders who are only familiar with traditional approaches. While case examples of applying DE exist—such as in Patton et al. (2016)—we aim to contribute to this literature by detailing our first year evaluating a program called Complex Care Hub (CCH).

CCH

CCH was launched in February 2018 at a university-affiliated community hospital in Calgary, Alberta, Canada. Based on the hospital-at-home (HAH) model (Caplan et al., 2012; Leff et al., 2005; Montalto, 1997; Shepperd et al., 2008), CCH is a virtual hospital inpatient unit where patients receive selected acute care services while remaining in their own home. The program also leverages allied health services, including Community Paramedic and Home Care programs to provide care in the home. General Internal Medicine (GIM) specialists oversee patient care and assess patients in the hospital’s Day Medicine clinic, via telemedicine (secured videoconferencing) or home visits.

CCH is unique because, to our knowledge, it is the only HAH program that combines the expertise of GIM specialists in dealing with patient acuity and complexity, with Community Paramedics as their “eyes and ears” in the home to provide interventions, assessments, and case management support. Community Paramedics have been selected for their clinical acumen and have extra training in helping vulnerable populations beyond their core first responder skills. Another central component of the program is case management, led by CCH nurse navigators to develop complex care plans for patients and connect them to primary care and appropriate community resources.

The program aims to reduce the health care utilization of patients who are high users of acute care. This is based on the literature suggesting that HAH reduces cost while improving outcomes including readmission rates and mortality versus conventional hospitalization (Caplan et al., 2012; Frick et al., 2009; Shepperd et al., 2009). While CCH provides care from young adults to the oldest old (age range to date of 19–101 years), the program targets vulnerable older adults with complex and chronic health conditions. Vulnerable seniors can experience poor health outcomes when physically admitted to a conventional hospital setting and are at risk of functional decline and iatrogenic adverse events, such as delirium and infections (Caplan et al., 2005, 2006, 2012; Fong et al., 2012; Leff et al., 2005; Montalto et al., 2010; Shepperd et al., 2008, 2009; Shepperd and Iliffe, 2005). They often benefit from less aggressive treatment and a holistic evaluation of their disease trajectory. CCH also fills a gap for patients with lower acuity issues who require hospital-level interventions, but could not previously access medically supervised care outside of hospital walls. Therefore, typical CCH patients’ acuity level is lower than average hospital admissions, but higher than in primary care settings. In its first year, CCH saw 90 individual patients (101 total admissions).

Applying the eight principles of DE: A case example

The evaluation team became involved after the provincial health organization, Alberta Health Services (AHS), requested an evaluation of CCH in November 2017. It soon became apparent that an adaptive evaluation approach was necessary to manage the complexity of the program and its environment. In this article, we delve deeper into the context that led to us using DE, and we share our experience of using a DE approach.

Below, we describe our alignment with the eight “essential elements” of DE delineated by Patton (2016b; see Table 1 for an overview). We do so in response to Patton’s (2016a: 3) concerns about the “fidelity challenge,” which requires evaluators conform to these principles. We aim to demonstrate both the suitability of DE for CCH and how the evaluation aligned with DE theory.

Table 1.

Developmental evaluation (DE) principle examples in practice.

DE principle	Case-study examples
Innovation niche “Elucidate how the change processes and results being evaluated involve innovation and adaptation, the niche of developmental evaluation.” (Patton, 2016b: 301)	• CCH is a new and innovative program based on the HAH model. • CCH is one of the first of its kind as it intertwines the HAH model, GIM specialists, and community paramedicine. • Change processes: While developing its model, CCH had to innovate and change the system in which it works. • Innovation: See example on page 6 where CCH created a shared platform on the existing electronic medical record to increase communication. • Adaptation: See example on page 6 in which CCH adapted its operations to fit the rotational schedule of GIM physicians.
Co-creation “The innovation and evaluation develop together—interwoven, interdependent, iterative, and co-created—so that developmental evaluation becomes part of the change process.” (Patton, 2016b: 306)	• When the evaluation team identified DE as the most appropriate evaluative approach for CCH, the program team responded with enthusiasm and commitment to learning about and using DE. • The DE was designed based on consultation with the program team. Methods were organically designed based on program need. See the Developmental Evaluation Rapid Feedback (DERF) cycle on page 12. • Inherently, the DE helped co-design the program as utilization meetings and DERF cycles helped inform action steps and program change.
Evaluation rigor “Ask probing evaluation questions; think and engage evaluatively; question assumptions; apply evaluation logic; use appropriate methods; and stay empirically grounded– that is, rigorously gather, interpret, and report data.” (Patton, 2016b: 296)	• Multiple methods were used to collect empirical data. See observation logs (Supplemental Table S1), interview guides (Supplemental Figure S1), and critical/systems thinking. • Thematic analysis was consistently used to understand what the DE was capturing. • The evaluation and program teams questioned their assumptions periodically to ensure CCH was achieving its proposed outcomes and the evaluation was accurately assessing the needed processes and outcomes.
Developmental purpose “Illuminate, inform, and support what is being developed, by identifying the nature and patterns of development (innovation, adaptation, systems change), and the implications and consequences of those patterns.” (Patton, 2016b: 291)	• The evaluation team used a DE learning framework to record and manage DE data (See Table 2). This framework enabled us to tell the story of the program’s development both chronologically and thematically. • The evaluation team adopted the mantra of “being the framework,” which emphasized understanding CCH well enough to provide evaluative feedback and informal recommendations in or near real-time (page 13).
Complexity perspective “Understand and interpret development through the lens of complexity, and conduct the evaluation accordingly. This means using complexity premises and dynamics to make sense of the problems being addressed; to guide innovation, adaptation, and systems change strategies; to interpret what is developed; to adapt the evaluation design as needed; and to analyze emergent findings.” (Patton, 2016b: 304)	• It was apparent from the beginning that CCH was a dynamic program working in a complex environment. For examples of nonlinearity and uncertainty, see page 3. • The developments the evaluation team collected captured both intended and emergent outcomes, two key elements of emergence discussed by Patton (2011: 150). See Figure 1.
Systems thinking “Think systemically throughout, being attentive to interrelationships, perspectives, boundaries, and other key aspects of the social system and context within which the innovation is being developed and the evaluation is being conducted.” (Patton, 2016b: 306)	• To capture the system that CCH was working in and the various actors within CCH’s social system, the evaluation team included system mapping elements in interview guides, asking: 1. What parts of the healthcare system were involved? 2. How or why that part of the system was involved? 3. What leverage points were available that could help CCH create shifts in the system? This allowed us to start mapping the system and how CCH was interfacing with it • See page 6 for more examples
Timely feedback “Time feedback to inform ongoing adaptation as needs, findings, and insights emerge, rather than only at predetermined times (e.g., quarterly, or at midterm and end of project).” (Patton, 2016b: 308)	• The evaluation team developed a method called the DERF cycle to expedite information collection and feedback for the program team. • An evaluation member was also embedded part-time at the program site to provide real-time feedback and capture developments. • See page 13 for more examples of providing real-time feedback.
Utilization focus Focus on intended use by intended users from beginning to end, facilitating the evaluation process to ensure utility and actual use.” (Patton, 2016b: 299)	• The evaluation team facilitated utilization meetings after DERF cycles to discuss action steps with the program team and review the summary reports highlighting challenges and opportunities in each cycle (page 13). • The program team allocated 15 minutes in their weekly project management meeting to discuss developmental results and utilization. • The evaluation team, in conjunction with the program team, also reflected on the value of the DE work and changed our methods when needed to adapt to the information requested from funders, executive leaders, and clinicians.

DE: developmental evaluation; CCH: Complex Care Hub; HAH: hospital-at-home; GIM: General Internal Medicine.

Rationale for employing DE to evaluate CCH

Innovation niche

The evaluation team began collaborating with the program team as early as possible to design a 3-year evaluation. The DE team comprised a senior research and evaluation consultant, a research and evaluation consultant, and two research and evaluation coordinators (1.25 total full-time equivalent (FTE) budgeted with higher actuals due to project requirements). The core CCH program team included the CCH medical lead, a paramedicine manager, a program manager to help oversee operations, and a project manager to support the tracking, development, and reporting needs of CCH. Through frequent meetings and rapport-building, a strong relationship between the teams formed early. The initial months of work focused on defining the expected outcomes of CCH and to build a corresponding evaluation framework. Prior to the evaluation team’s involvement, the program team had begun preparations for a clinical research study of the program’s cost-effectiveness and clinical outcomes. Thus, we worked to develop the evaluation framework in tandem with the research protocol to ensure the evaluation would complement the proposed study. Within the first few months after launch, however, the evaluation team identified a number of “complexity-sensitizing concepts” (Patton, 2011: 146) as applying to CCH and its environment.

CCH targeted a medically complex population group. Furthermore, the program was designed to mimic inpatient processes to facilitate a smooth transition for clinicians to the program’s workflows. The program team grappled with the challenges of innovation and establishing itself within the existing healthcare system. The program team described CCH’s path to success as not following a straight line. Similarly, early results were not always proportional to program inputs (nonlinearity). Optimal processes and outcomes were often uncertain and not always apparent in advance (uncertainty).

For example, since the program was targeting a wide patient population, admission criteria and processes for CCH were kept broad at the outset. The program intended to admit patients from the Emergency Department (ED), anticipating this population to be the appropriate level of acuity. Significant time was invested in understanding the ED environment, training personnel and searching for patients, but ultimately the ED did not have enough eligible patients. This led the program team to explore alternative admission sources. Aside from admission processes, program funding, staffing resources, and interdepartmental negotiation are just a few other sources of uncertainty the program team often faced.

In addition, new program developments were unfolding regularly (emergence). First, CCH used an established model (HAH), but expanded upon it by forging new clinical partnerships (GIM physicians and Community Paramedics). Second, virtual inpatient care was a new concept in Alberta (and most of Canada). CCH was conceptualized such that it leveraged existing processes and infrastructure to create a new model of care.

As an innovation, CCH ambitiously aimed to make changes within its healthcare context. This required the program team to make adjustments to well-established system processes. For instance, a variety of pre-existing system constraints prevented effective information sharing between CCH’s hospital-based doctors and nurses and the Community Paramedics providing care in patients’ homes. Prior to the program’s implementation, Community Paramedics used an electronic medical record (EMR) that did not interface with the EMR used by the hospital. The program team engaged AHS’ information technology team early in the program’s design to create a shared platform for these two provider groups to exchange clinical information (orders, investigation results, and clinical documentation). This required Community Paramedics to learn the hospital EMR and for certain functionalities to be adapted to meet CCH’s needs.

However, the program team also had to adapt to factors they could not change. For example, to enable GIM physicians to work on CCH and provide continuous clinical coverage, the program had to conform to the GIM physicians’ predetermined on-call schedule, which required a weekly rotation through CCH. This schedule was established prior to the program’s design and dictated the timing and duration of each physician’s time on the program. This had important operational implications, as CCH staff and patients worked with a new physician each week. The frequency of these rotations required significant training and efforts to standardize CCH’s processes. However, the CCH model was flexible in adapting to immutable systemic context. By doing so, the program was able to become one of many rotations for physicians, rather than an isolated entity.

The program team was also participating in systems change as part of a large-scale effort aimed at augmenting community-based care to provide care closer to the patient’s home and ease demands on hospitals. These efforts required the GIM physician group’s culture of care provision to shift. Traditionally, these specialists worked within the walls of the hospital or offered sporadic clinics in the community. Having them partner with Community Paramedics to provide virtual home visits—and, in some cases, in-person home visits—was novel in this clinical context. CCH aimed to support the healthcare system by enabling the hospital to increase its census without additional physical infrastructure. CCH straddled the inpatient and outpatient sectors to provide this care outside of hospital. As a result, the program team, and at times the frontline clinicians, continually ran into barriers. CCH’s leaders often found themselves having to engage hospital and health system leaders to resolve issues and enable CCH to continue its mission.

CCH’s aims to strengthen the continuum of health care included developing partnerships across acute, primary, and community care. This involved an array of relationships within the hospital, AHS’ administrative departments, and community and primary care stakeholders outside of AHS. For example, acute care services involved in the program’s operations included GIM, Transition Services, ED, Family Medicine, Hospitalists, Pharmacy, Day Medicine, Laboratory, Diagnostic Imaging, and Health Records. With an expansion to another site planned to occur shortly after the first anniversary of the program’s launch, these departmental synergies were expected to be duplicated at the second site. Overall, the stakeholder map was complex. An expansive steering committee of over 50 members was established at the hospital with participation from community leaders to guide the program’s planning and implementation. A research steering committee concurrently supported ongoing research and evaluation tasks to inform knowledge generation and translation. These interactions between site, organizational, and provincial collaborators were contributing to multilevel system changes in the culture of acute care service delivery and the inpatient experience (dynamism, adaptation, coevolution).

These are but some of the many innovations and complexities the evaluation team observed as the program team refined operations. Out of necessity, a developmental mindset surfaced organically within the evaluation team. The program’s very nature forced the evaluators to be agile, flexible, and closely integrated with the program team to keep pace with updates. For example, in addition to innovation and integration work, CCH was still responsible for providing evidence-based care for patients and producing similar health outcomes as standard care delivery. These accountabilities warranted research and/or evaluation to assess the merits of CCH. A formal research study formed part of this effort. However, neither traditional research nor formative evaluation approaches could sufficiently meet the program’s needs. A DE approach was needed to fulfill the innovation niche (Patton, 2016b: 301) of CCH’s development. DE enabled the evaluation team to document the program team’s changes and adaptations while informing decision-making. When the evaluation team proposed this direction, the program team embraced it with enthusiasm and a commitment that grew stronger over time.

Designing and implementing DE

Evaluation rigor and co-creation

With the consent and collaboration of the program team, we crafted our DE strategy and designed new data collection methods as needed. Although constantly evolving, the program had been admitting patient research participants since its launch date. Thus, administrative and clinical data were being collected accordingly. As a result, the evaluation team was also involved in supporting the research study by collecting survey, interview, and administrative data. The evaluation team also supported health economists in evaluating CCH’s value for money. At times, these data were woven together with the DE interviews and observation results, but they were most often viewed in parallel due to the time required for the methods involved in the research study.

The evaluation team targeted its DE efforts to document developments, analyze and synthesize data, and feed them forward to the program team in an effort to illuminate, inform, and support program development and decision-making. Patton (2016b: 296) explains DE is more of a mindset than a method, and that methods should be selected and used in ways that best fit the evaluation situation. For CCH, discussions among the evaluation and program teams informed which tools were needed to provide feedback for advancing the program’s goals while ensuring evaluation rigor (Patton, 2016b: 296).

Despite everyone’s busy schedules, all involved were committed to keeping connected to enable the DE approach. The evaluation team participated in regular meetings, both virtual and in-person, with the program team to discuss important updates or suggestions relevant for the DE. Importantly, communication was not limited to scheduled interactions. The evaluation and program teams kept in contact via phone calls, emails, and text messages. Connections were often spontaneous when there was a sudden need to connect. Through this frequent, bidirectional communication, we embraced the spirit of co-creation (Patton, 2016b: 306): we successfully collaborated to move the DE forward and develop methods that would inform program decisions.

One of the first tools the evaluation team developed was a DE observation log adapted from other online resources (Dozois et al., 2010; Spark Policy Institute, 2014) (see Supplemental Table S1). The log was used to record observations made when the evaluation team was interacting with the program team in weekly and ad hoc meetings. A few months into the DE, a member of the evaluation team was embedded with the clinical program team on a half-time basis. This coordinator was well-positioned to not only collect data, but was also essential for building relationships, understanding CCH’s clinical context, providing in-the-moment feedback to the program team, and recording real-time observations. These observations were logged in an ongoing manner and time-stamped.

As the DE progressed, the evaluation team found it useful to have regular brief phone interviews with the CCH medical lead. Informally, these unstructured interviews were referred to as “brain dumps” in which the CCH medical lead updated the evaluation team on recent opportunities and challenges that had arisen for CCH and potential next steps. The lead evaluator conducted and recorded these interviews and analyzed them for key developments. The evaluation team eventually expanded these brief interviews into a more comprehensive process, which is discussed later on as the Developmental Evaluation Rapid Feedback (DERF)cycle. Prior to taking part in any DE activity, all participants signed a written informed consent form.

Systems thinking

The evaluation team engaged in systems thinking (Patton, 2016b: 306) by remaining attentive to the various actors within CCH’s social system, the interactions between them, and how CCH was both impacting and impacted by them. To capture the systems change processes that CCH was undertaking, the evaluation team included systems change elements in the DE interview guide (Supplemental Figure S1) and in the DE observation logs. Specifically, the evaluation team sought to capture three main elements for each major development: (1) What parts of the healthcare system were involved; (2) how or why that part of the system was involved; and (3) what leverage points were available that could help CCH create shifts in the system. In this way, the evaluation team gathered information that was helpful in starting to map the system elements with which CCH was interfacing and how the program was working to change the system (Dozois et al., 2010).

The evaluation team captured findings from these sources in a DE learning framework (see Table 2 for excerpt), which enabled them to manage a large volume of data (Dozois et al., 2010). All entries were date-stamped and codified in the framework which, over time, enabled the evaluation team to tell the story of the program’s development, both chronologically and thematically.

Table 2.

Developmental evaluation learning framework excerpt.

Theme	Date	Subtheme	Development type	Description	What the project team wants to know/learn/pay attention to	Data source	What parts of the healthcare system contribute to, influence, or are affected by the challenge/opportunity	How or why the part(s) of the system are involved in the challenge or opportunity	Leverage points that can help to create shifts in the system
Case-finding	23 February 2018	Emergency Department	Challenge	Case-finding challenges in the ED include the following: no referrals at night, ED doctors unaware of appropriate referrals, IT challenges that complicated doctor’ referrals, time and length of ED consults can prevent a CCH admission as the patient may get transferred to a ward, some patients too sick to leave ED	Help to communicate to the ED MDs about what kinds of patients are being admitted to CCH so that they get a better sense of type of patients	1:1 Interview	ED, ED doctors, IT	Consult/referral processes	Posters to communicate changes to ED
Case-finding	20 April 2018	Hospital Ward	Opportunity	Opportunity to admit to CCH from the ward. Work has gone into understanding patient flow through the wards. Some anxiety about CCH and nursing capacity if there are too many ward admissions. However, RGH leadership supports ward admissions	Instead of funneling admitted patients to Day Medicine might funnel them to inpatient units	1:1 Interview	RGH GIM units, Day Medicine, hospital leadership	Consult/referral processes	Leadership support

ED: emergency department; CCH: Complex Care Hub; GIM: General Internal Medicine; RGH: Rockyview General Hospital.

Complexity perspective

The data collected also sensitized the evaluation team to the need to maintain a complexity perspective (Patton, 2016b: 304) and map intended and emergent outcomes as CCH’s program team experienced complexity and was forced to adapt and move within it. For example, the complexity of the population served by CCH and the fragmentation of the healthcare system quickly became obvious, prompting the program team to stratify patients by acuity and biopsychosocial needs at different times in their CCH admission. It also became apparent that some patients required a shorter admission to the program with daily visits and interventions, while others required a longer, less intensive admission to complete their case management and foster a smooth transition back to the community with increased supports. As such, the service was reorganized to provide two levels of care that were progressive and adapted to patients’ needs over time. Such dynamic program design led us to focus evaluation efforts on both intended and emergent outcomes, following the model in Figure 1 adapted from Patton (2011: 49). The DE approach allowed us to adapt our methodologies and indicators as the program changed so rapidly: while we originally collected outcome data, such as length of stay, for one level of CCH care, we had to quickly review our indicators to monitor and report on all levels of CCH’s new stratified approach. Close collaboration between the program and evaluation teams and CCH’s research steering committee ensured the new indicators’ relevance to CCH’s clinical context.

Figure 1.

Model of intended and emergent outcomes, adapted from Patton (2011: 49).

Developmental purpose and utilization focus

The ultimate rationale for this range of data collection was to fulfill a developmental purpose (Patton, 2016b: 291) and support the CCH program team in innovating and developing the program. The evaluation team verbally communicated the findings to the program team through weekly project management meetings, in-person meetings, and phone calls to illuminate key developments; discuss implications; and support appropriate action planning. The evaluation team generated summary handouts and reports and chronicled how CCH evolved over time, using the metaphors of building a honeycomb or cobblestone path to visually and conceptually represent the program’s most significant developments and their implications. Where appropriate, the evaluation team made recommendations and/or solicited the program team’s feedback regarding additional information they needed, what actions they wanted to take as next steps, and what they felt should be prioritized in order to target its efforts based on bidirectional feedback. This ensured the evaluation maintained a utilization focus (Patton, 2016b: 299), tailored to the program team’s needs as the primary users, while being additionally useful for external stakeholders (e.g. funders, administrators).

For example, 5 months into the DE the evaluation team organized a presentation to the program team, discussing five main thematic categories of developments that had been captured to date. The first two categories were positive and focused on successes and opportunities: a successful launch of the program and momentum created within the organization to spread and scale. The three remaining thematic categories were areas for improvement identified by the program team: case-finding, staffing models, and challenges during the discharge process.

CCH had experienced a successful launch and 6-month run-in period and was actively admitting and discharging patients. Given the program’s complexity, this was no small feat. (Two years of stakeholder engagement was conducted prior to the launch, and the steering committee of more than 50 members was guiding the work.) Second, there was significant enthusiasm within and beyond AHS for the program model. Another hospital in Calgary was interested in having CCH expand there in the subsequent year. Discussions were occurring, committees were being formed, and overall efforts to plan the expansion were underway.

However, as the evaluation team’s review identified, the CCH model still needed to overcome other themes representing significant challenges to ensure its sustainability. First, case-finding was a challenge in terms of finding the most appropriate patient population (see Figure 2). Patients needed to be at the right level of acuity and functionally safe to receive care at home. This required clinical acumen and more time for assessment than usual medical admissions to conventional inpatient wards. In addition, hospital physicians (in ED and Hospitalist units) needed to be encouraged to refer patients to CCH. These physicians invested significant time and effort into patient admissions and understandably felt some concern about subsequently transferring patients to CCH. This required building processes and relationships that took significant clinical time and impacted CCH’s ability to case-find.

Figure 2.

A visual representation of CCH’s case-finding challenges.

Second, challenges in the staffing model emerged over time. For example, there was a need for a Clinical Assistant to support case-finding, as well as greater Pharmacist support for processes related to medication management and reconciliation. The program’s rapid growth also necessitated more project management staffing.

Finally, discharging patients from CCH was complex in that patients needed to be connected to community services appropriate for their particular needs. Community services in Calgary were both numerous and fragmented with no central database outlining all available services or their intake criteria. These, combined with gaps that were uncovered in managing complex patients in the community, were some of the numerous challenges related to discharging patients, contributing to longer lengths of stay in CCH than would otherwise have been required.

The evaluation team shared the findings from the above five themes with the program team during an in-person meeting and made recommendations as part of its presentation. Although they shared their perspectives and creative problem-solving, the evaluation team primarily based these recommendations on themes and solutions directly communicated by DE interview participants. Importantly, half the meeting time (approximately 45 minutes) was allotted to facilitate a discussion, with the evaluation team probing for feedback about the five themes and how this information could be applied to potential action and learning steps.

This was an important way that the DE contributed to the CCH’s decision-making processes. For example, case-finding had been a challenge since the creation of CCH. The evaluation team offered an evaluative lens to problem-solving; they gathered and organized relevant feedback about case-finding from multiple sources, tracked what had been attempted, the outcomes of these efforts, and finally presented possible options. Together, the group identified that their next step to resolve case-finding challenges would be to engage the sponsorship of the Rockeyview General Hospital (RGH) Senior Operating Officer and reach out to the head of hospitalists to begin negotiating mutually agreed upon processes. This sponsorship support ultimately served as a vital leverage point for relationship-building efforts; eventually new processes for these clinicians and CCH were built to support the program’s case-finding. This highlights CCH’s system-level influence; by engaging multiple groups to create process changes, CCH helped challenge the status quo for acute care services in RGH.

Timely feedback

While this DE result meeting was fruitful, the evaluation team was not able to deliver and discuss all of the data in 1 day. The early stages of the DE work were heavily focused on refining data collection processes, and thus a large backlog of information had accumulated. It became clear that the evaluation team had not yet successfully aligned with the DE principle of timely feedback (Patton, 2016b: 308), which triggered the creation of a rapid feedback process.

The evaluation team challenged themselves to see how fast they could collect empirical data, analyze and report it back to the program team, and then follow up to see what the program team’s next steps would be in terms of a learning or action plan. They communicated this goal to the program team who, despite their busy clinical schedules, reciprocated by making themselves available for interview sessions. The evaluation team called this process a DERF cycle.

The DERF cycle initially included six main components:

The evaluation team conducted 15- to 45-minute interviews with members of the program team to capture recent developments. As the program evolved, the scope of these interviews was expanded to include the Community Paramedic team, project management team, and any subsequent new hires to ensure that a range of perspectives were included. Each interview cycle had a sample size of approximately seven and varied based on participant availability; each evaluation team member completed up to three interviews. Interviews were not recorded, but detailed notes were taken.

Interviews were then rapidly analyzed. Each interviewer reviewed their notes for key themes and entered the findings into a shared master excel file. The evaluation team then met to discuss and refine themes as a group. Once the themes were finalized, data were consolidated into a two-page summary with main challenges on one side and opportunities on the other (see Figure 3).

The program team was given time as needed to review the handout (from a few days to 2 weeks).

The evaluation team conducted 15- to 45-minute follow-up interviews with members of the program team to capture action and learning steps, as well as the system mapping elements for the DE learning framework.

The results of the second round of interviews were rapidly analyzed using a similar method of analysis as in step 2. Results were summarized into a second handout listing actions already in progress or completed, potential actions, and things to explore for each category of challenges and opportunities (see Figure 4).

The DE learning framework was updated and preparations were made for the next DERF cycle.

Figure 3.

An excerpt of a round 1 DERF cycle summary of CCH’s challenges and opportunities.

Figure 4.

An excerpt of a round 2 DERF cycle summary of CCH’s potential action steps.

The DERF cycles were not scheduled at predetermined times. Rather, they were an effort to quickly collect, analyze, disseminate, explain, and use empirical data regarding program developments. Initially, the evaluation team aimed to complete the first cycle within 4 weeks, but the creation and refinement of processes and templates, amid continuing traditional evaluation duties, resulted in a 7-week turnaround time for the full process.

As the DE work progressed, the evaluation team checked in with the program team as to whether the results of the cycle were being used to support development. Following the first DE rapid feedback cycle, the program team signaled they wanted facilitation support from the evaluation team to discuss follow-through and use of the DE results. As a result, the evaluation team added a facilitation step to the end of the rapid feedback cycle and used the DERF results to inform an in-person session with an emphasis on use of the DE results. In addition to these in-person sessions, we also agreed to add 15 minutes to weekly CCH project management meetings to check in on progress regarding DERF results.

Ultimately, while these DERF cycles led to turnaround times that were faster than what may commonly occur in traditional evaluations, the program team was rapidly developing the program and the evaluation team found that, without even faster processes, they were still missing the mark at times. The evaluation team therefore adopted the mantra of “being the framework,” a commitment to build a deep understanding of the program, its emerging developments, and the breadth of data on hand, without relying solely on formal data collection and reporting processes. This enabled the evaluation team to usually respond to simple DE data-related requests within 24 hours and more involved requests within 1 week. For example, later in the year, the program team began reaching out to the evaluation team for targeted feedback on specific topics, such as the CCH staffing model. Upon request, the evaluation team was able to pull together relevant data from the various data sources to quickly respond to requests. In addition, even without a DE data-related request, the bidirectional communication that was established within the teams allowed evaluation members to provide valuable input from a DE lens wherever appropriate, embodying the “be the framework” mantra. This enabled the program team to quickly modify their processes or even re-examine the responsibilities of each team member to address emerging challenges. This flexible and well-prepared mindset consistently served both teams well, and the evaluation team felt that they had finally modeled what real-time decision support could look like.

Challenges and lessons learned: Reflections from the evaluation team

While we felt that the application of DE during CCH’s first year was an overall success, the evaluation team encountered a number of challenges and learned a great deal about applying the principles. Adopting a DE mindset requires the ability to be nimble and responsive in a dynamic environment, which can conflict with traditional research requirements a fledgling program may need to determine its implementation success.

Tension between timely feedback and evaluation rigor

Patton (2016b: 310) explains that the DE principles are intended to be “interrelated” and “mutually reinforcing.” However, the evaluation team experienced some ongoing tension between rigor and timeliness. In other words, to “rigorously gather, interpret, and report data” (Patton, 2016b: 298) often felt at odds with “timely feedback to inform ongoing adaptation” (Patton, 2016b: 306). As the program team was rapidly modifying CCH to adapt to emerging circumstances, the evaluation team grappled with ensuring accurate DE findings were shared in time to inform decisions. This regularly pushed the evaluation team to scrutinize methods traditionally associated with rigor (e.g. interview transcription and coding). Patton (2015: 701) posits that methods alone do not ensure rigor; deep critical, creative, evaluative, inferential, and practical thinking is also required. Yet for the evaluation team, it was at times difficult to forgo the time and processes that had previously exemplified rigorous thinking, while understanding the evaluation was meant to rapidly inform the program team. Another facet to this challenge was a sense of discomfort in giving feedback that was timely, but, at times, not as polished as the evaluation team was accustomed to providing. The evaluation team often discussed the DE mindset required for this approach and how they might need to seek out other methods and tools as needed. In an era of smartphones and rapidly developing technology, new possibilities are emerging regularly for evaluators to consider.

Maintaining objectivity

The evaluation team’s close working relationship with the program team added another challenge in terms of maintaining objectivity. Strong rapport and close collaboration had been built between the two teams, and a member of the evaluation team worked onsite with the program team nearly every day. This provided a valuable in-depth understanding of CCH for the DE, but required constant diligence by the evaluation team to maintain objectivity. The evaluators navigated this by establishing boundaries with the program team early on and sensitizing them to expect both positive and negative results. The program team embraced this and were receptive to constructive feedback. Independently, the evaluation team met often to reflect on the DE findings together and check in with each other about potential biases. These conversations helped maintain impartiality, but added another layer of process to feedback provision. Similarly, the evaluation team and medical lead regularly reflected on the relationship of the evaluation and program teams to avoid and/or mitigate any potential biases that could have arisen as a result of a major change in the program or research processes. Ultimately, the evaluation team also worked to identify methods that would enable them to provide useful data and keep pace with the rapidly innovating program team while routinizing objectivity check-ins into day-to-day operations.

Capacity constraints for the program team and evaluation team

We found DE to be more time-intensive than traditional evaluation—both for the evaluation team and for the program team. The evaluators were regularly involved with the program team, tracking developments, and otherwise regularly adding value with our evaluation lens. However, the program team was also regularly engaged—for example, in DE interviews, discussions, and feedback sessions. Given how busy the program team often was with rapidly building out a complex program in addition to their usual clinical activities, the addition of DE requirements was, at times, challenging. The additional time burden never ultimately prevented strong communication and never unduly prevented our ability to move the DE forward. However, we found it important for all parties to fully appreciate what would be required.

The custom-built nature of DE work also sometimes meant design work sprints when there were new developments. This was a challenge at times, especially for work that required a lot of time to set up. For example, the back-end coding for a complicated CCH dashboard required significant time invested by a dashboard analyst. When the program or evaluation shifted in a different direction, it compromised a significant amount of work already done on the dashboard, as well as required still more hours to accommodate the new data needs.

The additional time required for conducting the DE work also meant budget implications. The evaluation team is fortunate to be funded in a way that is enabling a team of four members (1.25 FTEs) to work on the CCH evaluation for 3 years, along with some in-kind support from a dashboard analyst. Even still, during the first year, the time required often pushed the limits of the time available. Doing this kind of work in contexts where the same level of funding or in-kind support is not available could be prohibitive.

DE and traditional research ethics boards

One of the contextual factors for the evaluation team’s DE efforts was that they were operating within the structure of a traditional research ethics board (REB) approval. Because there was a desire to publish research articles on CCH, and therefore a need to disclose findings outside of AHS, the study qualified as a research project requiring REB and related administrative approvals. This created a complicated project with a blend of traditional research and DE components.

As Patton (2011: 10) describes, the evolving nature of innovative programs often requires evaluators to make rapid and unexpected adaptations to the DE’s design and methods. Although there are emerging discussions about REBs in relation to innovation (e.g. Stryjewski et al., 2015), our REB was not able to accommodate the rapidly changing nature of our evaluation without requiring modifications to the existing approval when new developments warranted adapting our data collection strategies. We submitted multiple modifications to the REB protocol in CCH’s first year. Often, by the time a modification was completed, another was already required, often leading to significant administrative and cognitive burden. Furthermore, this process caused delays that compromised the evaluation team’s flexibility as DE practitioners and their ability to meet the timely feedback principle.

We want to emphasize that we are fully in support of ethical practice, but propose that traditional REB processes proved challenging because they generally require a project to be fully specified at the outset, and, in our experience, REBs may not have processes in place to sufficiently support developmental projects without ongoing modifications to the approval. Therefore, there is a need for REBs, innovators, and developmental researchers and evaluators to re-assess the current processes to develop more efficient ways to ensure compliance with both ethical standards and principles of DE.

Continuing to socialize DE with funders and stakeholders

“DE—what’s that?” is a common response we have encountered when talking about the DE work for CCH. DE is spreading, but we have found a need to continually socialize DE with funders and other stakeholders. In addition to the REB example discussed above, CCH also has accountability requirements to its funders. The funders take a traditional approach, and, as a result, the reports often ask for regular updates to predetermined indicators. As CCH and our DE work evolved over time, there were some indicators we would choose to abandon, as well as some we would choose to add. But as long as traditional accountability models view unrealized outcomes as failure and emergent outcomes as mission drift (Patton, 2015, as cited in Snow, 2015), there will be a mismatch between DE studies and reporting requirements. However, as we continue to have conversations with REBs, funders, program leaders, decision-makers, and others, new ways of working together will emerge—such is the nature of complexity.

Conclusion

DE, in our experience, is a dynamic approach to evaluating rapidly evolving programs such as CCH. While striving to adhere to the eight principles of DE, the evaluation and program teams worked closely, enabling us to capture the evolutionary model of CCH in its first year. Through the DE, the evaluation team was able to document the chronology of the program as if it were a blueprint, which we believe will be essential for the scale and spread of the program as it continues to grow in the healthcare system. While the first year was nothing short of fascinating, challenging, and fulfilling, it required a large time commitment and cost and could not have been done without strong support from multiple parties from the outset. Despite the large time commitment and the challenges faced, DE was a suitable and worthwhile approach to evaluate CCH’s development, particularly in its first year of implementation and preparation for spread to a second site. As the program, the DE, and their co-existence were all innovative ways of working within AHS, the evaluation team had the privilege of breaking new ground and supporting the growth of a promising program.

Supplemental Material

Supplementary_Materials – Supplemental material for Applying the eight principles of developmental evaluation: Complex Care Hub as a case example

Supplemental material, Supplementary_Materials for Applying the eight principles of developmental evaluation: Complex Care Hub as a case example by Jason R. Goertzen, Ashley D. Fraser, Marysia E. Stasiewicz and Michelle N. Grinman in Evaluation

Footnotes

Acknowledgements

The authors thank Jill Veenendaal; the CCH program, clinical, and leadership teams; the Transition Services project management team; and Members of the Health Systems Evaluation & Evidence Department of Alberta Health Services

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This evaluation work was funded internally by Alberta Health Services.

ORCID iD

Jason R. Goertzen

Supplemental material

Supplemental material for this article is available online.

Jason R. Goertzen is a Senior Consultant with Alberta Health Services (AHS). He holds a doctorate in the History & Theory of Psychology from York University, Toronto, Canada.

Ashley D. Fraser is a Research and Evaluation Coordinator with AHS. She holds a master’s in Public Health from the University of Victoria, Canada.

Marysia E. Stasiewicz is a Research and Evaluation Consultant with AHS and holds a Bachelor of Arts from the University of Alberta, Edmonton, Canada.

Michelle N. Grinman is a Clinical Assistant Professor and Internal Medicine physician, Cumming School of Medicine, University of Calgary, Canada. She is also Medical Lead for Seniors, Palliative and Continuing Care, Calgary Zone, AHS.

References

Caplan

Coconis

Woods

(2005) Effect of hospital in the home treatment on physical and cognitive function: A randomized controlled trial. Journals of Gerontology Series A 60(8): 1035–8.

Caplan

Coconis

Board

, et al. (2006) Does home treatment affect delirium? A randomised controlled trial of rehabilitation of elderly and care at home or usual treatment (The REACH-OUT trial). Age and Ageing 35(1): 53–60.

Caplan

Sulaiman

Mangin

, et al. (2012) A meta-analysis of “hospital in the home.” The Medical Journal of Australia 197(9): 512–9.

Dozois

Langlois

Blanchet-Cohen

(2010) A practitioner’s guide to developmental evaluation. The J.W. McConnell Family Foundation and the International Institute for Child Rights and Development. Available at: https://mcconnellfoundation.ca/wp-content/uploads/2017/07/DE-201-EN.pdf (accessed February 2019).

Fong

Jones

Marcantonio

, et al. (2012) Adverse outcomes after hospitalization and delirium in persons with Alzheimer disease. Annals of Internal Medicine 156(12): 848–56.

Frick

Burton

Clark

, et al. (2009) Substitutive Hospital at Home for older persons: Effects on costs. The American Journal of Managed Care 15(1): 49–56.

Gamble

JAA

(2008) A developmental evaluation primer. Montreal, QC, Canada: The J.W. McConnell Family Foundation. Available at: https://mcconnellfoundation.ca/wp-content/uploads/2017/07/A-Developmental-Evaluation-Primer-EN.pdf (accessed February 2019).

Leff

Burton

Mader

, et al. (2005) Hospital at Home: Feasibility and outcomes of a program to provide hospital-level care at home for acutely ill older patients. Annals of Internal Medicine 143(11): 798–808.

Montalto

(1997) Hospital in the Home. Australian Prescriber 20(4): 88–9.

10.

Montalto

Lui

Mullins

, et al. (2010) Medically-managed Hospital in the Home: 7 year study of mortality and unplanned interruption. Australian Health Review 34(3): 269–75.

11.

Patton

(2011) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: The Guilford Press.

12.

Patton

(2015) Qualitative Research and Evaluation Methods: Integrating Theory and Practice. Thousand Oaks, CA: SAGE.

13.

Patton

(2016a) State of the art and practice of developmental evaluation: Answers to common and recurring questions. In: Patton

McKegg

Wehipeihana

(eds) Developmental Evaluation Exemplars: Principles in Practice. New York: The Guilford Press, 1–24.

14.

Patton

(2016b) The developmental evaluation mindset: Eight guiding principles. In: Patton

McKegg

Wehipeihana

(eds) Developmental Evaluation Exemplars: Principles in Practice. New York: The Guilford Press, 289–312.

15.

Patton

McKegg

Wehipeihana

(eds) (2016) Developmental Evaluation Exemplars: Principles in Practice. New York: The Guilford Press.

16.

Shepperd

Iliffe

(2005) Hospital at Home versus in-patient hospital care. Cochrane Database of Systematic Reviews 3: CD000356.

17.

Shepperd

Doll

Angus

, et al. (2008) Admission avoidance hospital at home. Cochrane Database of Systematic Reviews 4: CD007491.

18.

Shepperd

Doll

Angus

, et al. (2009) Avoiding hospital admission through provision of hospital care at home: A systematic review and meta-analysis of individual patient data. Canadian Medical Association Journal 180(2): 175–82.

19.

Snow

(2015) Webinar notes: “Emergence: Detection and interpretation in a leadership program” by Michael Quinn Patton. Dr. Beth Snow. Available at: https://www.drbethsnow.com/2015/01/27/patton-webinar/ (accessed February 2019).

20.

Spark Policy Institute (2014) Developmental evaluation toolkit. Available at: https://sparkinsight.com/courses/developmental-evaluation-toolkit/ (accessed 6 June 2018).

21.

Stryjewski

Kalish

Silverman

, et al. (2015) The impact of institutional review boards (IRBs) on clinical innovation: A survey of investigators and IRB members. Journal of Empirical Research on Human Research Ethics 10(5): 481–7.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB