Abstract
Intervention programs are open systems that must withstand the disturbances of the environments in which they are implemented if they are to thrive. Stakeholders can benefit from proactive insights related to potential program malfunction or failure. Evaluators have not developed approaches to serve this preventive function. This study proposes the program stress test as a holistic, preventive approach to program evaluation, aiming to identify vulnerable areas at risk of failure, and to suggest preventive strategies. The stress test can be used in the planning stage of a program (called ex-ante stress test), or during the implementation stage (called ex-post stress test). This study illustrates an ex-post test. The authors introduce its concept, methodology, research steps, and application to a student retention program, including findings of structural and functional issues, as well as solutions proposed by participants. The authors call the evaluators’ attention to the need to expand the traditional scope of evaluation, to include future-focused assessments.
Keywords
Introduction
This article proposes the program stress test as a new tool for program evaluation. Two observations are foundational to this proposition. First, the stress test concept is built upon an observation that intervention programs are open systems that must withstand the disturbances of the environments in which they are implemented if they are to thrive (Chen, 2015; Patton, 2011). Many industries and disciplines use stress tests to detect whether their programs maintain structural and functional parameters during stressful conditions, and to identify the parts or circumstances that allow system vulnerability.
Historically, stress tests were developed in response to real system malfunction or failure events and have since been widely used in multiple disciplines to prevent repeated failures. For example, a cardiac stress test in health care measures the heart’s ability to respond to high functional demand through exercise on a treadmill (King, 2017). In banking, a stress test uses simulation software to assess the required capital necessary to withstand unfavorable financial and employment market conditions such as a recession or high employment rate (Riebl & Gutierrez, 2018). In bridge construction, a stress test overloads a bridge to observe its maximum capacity, unity, and stability (Chang & White, 2008). In computer technology, stress testing is used to determine satisfactory software and hardware performance under extreme load or volume conditions (Ghosh et al.,1998). All types of stress testing propose to increase stress within a system, whether in reality or virtually, with the purpose of exposing structural and/or functional issues of the system, in the context of shifting environmental conditions.
A second observation is that agencies and community-based organizations are accustomed to building programs, not diagnosing or deconstructing them to discover structural or functional weaknesses. Evaluators should help stakeholders understand and document what holds the program together in synergy with its environment, and what could break it apart, without having to actually experience failure. Such information would be of great interest to stakeholders, indeed. However, program evaluation has not yet proposed a tool that serves such a proactive assessment purpose. For example, formative evaluation identifies and recommends troubleshooting solutions for immediate implementation problems (Scriven, 1991; West, 2012); however, formative evaluation cannot assess a program holistically to explain its underlying structural and/or functional problems. Process evaluation examines whether a program is implemented according to its plan (Rossi et al., 2004). Evaluability assessment examines whether a program is organized in a way that allows for rigorous outcome evaluation (Wholey, 2004). None of the existing evaluation approaches respond to the concept and purposes proposed by the stress test. This article will introduce the concept of program stress test, outline the methodology, illustrate its application to a retention program for nursing students, and discuss the unique contribution of the program stress test in contrast to other program evaluation tools.
Program Stress Test: Concept, Strategy, Methodology, and Application
Concept and Strategy
Intervention programs, like other systems, are created with a defined structure to produce intended outcomes (function). Ideally, intervention programs’ structure, function, and context should be defined by a model (i.e., a logic model, program theory, theory of change, the action model/change model schema) that can guide planning, implementation, and evaluation. Especially useful are models that clarify the underlying causal mechanisms and contextual factors of the program. Through decades of practice and experience, the authors found the action model/change model schema (Chen, 2005, 2015) most useful in this respect.
The schema (Figure 1) includes the components necessary to initiate action, the environment and target population that will benefit from the program, and specific instructions for service delivery. The action model, conceptualized as the driving force behind the change model, is unique to the implementing organization and implementation environment. Small changes to the ingredients of the action model can produce a rather large effect on the implementation and outcomes of the program. The change model proposes causal mechanisms and explains what interventions act upon which leverages to produce the desired outcomes, as informed by social, behavioral, or stakeholders’ theories.
Action model/change model schema.
As contextually derived open systems, intervention programs are likely to have structural weaknesses (e.g., inadequate staff, hostile environment) or functional issues (e.g., poor coordination, ineffective communication) that are not immediately visible in the models. Failure to recognize weaknesses before they disrupt the program can lead to program ineffectiveness or failure. Stakeholders need to identify and remedy these problems as soon as possible. However, because problems are likely to run beneath the surface and take time to manifest, they are not easy to predict. As a result, typical evaluation approaches have difficulty detecting them. How can these problematic or dysfunctional areas become obvious for diagnosis and remediation before damaging the program?
One potential solution is a stress test that intentionally magnifies weaknesses or overloads the program. The main question then becomes: How may we adapt the stress test idea to program evaluation? The existing literature indicates two stress testing strategies: actual stressing and simulation of stress. Actual stressing of a system (i.e., intentionally causing real tension or overload) has been used in other disciplines (e.g., bridge stressing through overload, cardiovascular stressing through treadmill exercise); however, actual stressing is not acceptable for intervention programs since stakeholders would never agree to such an approach because it could be the fastest route to program failure. Rather, program stressing must be done through simulation, much like the banking stress test (e.g., running simulations of financial market crises to estimate whether the bank has enough capital to withstand them). While banks run such simulations on computer software using electronic data, to our knowledge, such computer programs specific for program evaluation do not exist. This is not to say that virtual simulations using statistical modeling with interim data could not be useful; on the contrary, virtual simulation may become a promising stress testing method, once sufficient program data become available from the program itself or from similar implementation sites. However, such data have not been reported in the literature yet. Furthermore, it is questionable whether the great majority of intervention programs have such rich data required for computer simulation or statistical modeling at this time.
For our exploration of stress testing for program evaluation, we defined the program stress test as an evaluation tool that aims to proactively examine the resilience of the structural and functional integrity of a program, when challenged through stress scenarios facilitated by experienced evaluators. The following Methodology section explains how to conduct an ex-post program stress testing using focused discussion and mental simulation.
Methodology
The ex-post stress test, as the authors approached it, consists of three phases: preparation, execution, and report findings. The preparation phase is the most time consuming and important for the test. External evaluators are likely the best option for stress test facilitators because they are not part of the program but can think critically and objectively about program assessment, and they can quickly accustom themselves with the organization and program structure and function.
Phase 1: Preparation for the Program Stress Test
Phase 1is a three-step process of preparation for the actual stress test. During this phase, evaluators introduce the stress test as an evaluation tool, facilitate the selection of a program model, and plan the actual stress test meeting.
Step 1: Introduce the program stress test to the program management team (PMT)
In the first step, the evaluators introduce the program stress test to the PMT and discuss how the stress test can benefit the program. This conversation will expand the evaluators’ understanding of the program and organization structure, and it will prime stakeholders to start thinking in terms of program shortcomings. The PMT and evaluators should agree at this point on appropriate methods to look for underlying structural or functional deficiencies.
Step 2: Select a program model that will guide the stress test assessment
The next important conversation is about selecting a model that can discriminate the program structure, functions, and context to guide the stress test. The authors explained their preference for the action model/change model schema in the Introduction section. Some programs create a model during the planning phase, as it is often a funding requirement. However, these planned models do not always clarify the structure and function of the program and rarely explicitly consider the characteristics of the context in which the program will be implemented. These models can serve the purpose of a stress test if they meet, or can be adjusted to meet, these requirements.
Step 3: Create a stress test plan
The plan of the stress test is the most important part of the preparation phase because the success of obtaining relevant and useful data depends on it. Once the PMT and evaluators agree on the program model, the plan of the stress test may consider the following elements:
(a) Determine which program components will be stressed
All or some of the program components can be selected to stress test. The program model greatly simplifies this task since it provides the big picture of the program. The action model of the schema (discussed previously) provides a good representation of the program structure, its target population, environment, and proposed interventions. In selecting components to stress, evaluators should consider each stakeholder’s expertise and familiarity with different components of the program. The change model of the schema represents the necessary causal processes that lead the intervention to the desirable outcomes (function). A preliminary discussion or polling on program components that are most likely to fail under stress should help clarify the selection. Alternatively, the PMT can decide to stress test the entire structure of the program.
(b) Design the meeting
The design of the program stress test can be selected from a variety of formats including focus groups, working group meetings, team-building style meetings, brainstorming sessions, or other meeting formats. Regardless of the format, participants should do some preparatory work. For example, participants can enhance each other’s understanding of the entire program through short presentations of their own perceptions related to program components most familiar to them. The length of the group meeting should be sufficient for extensive discussions. Evaluators and stakeholders have to agree on meeting length and other logistic details (place, time of the week/day, serving refreshments/lunch, etc.) based on the size of the program, the number of selected program components, the number and schedule of participants, and so on. Small programs can benefit from a single iteration of the stress test; however, additional iterations may be necessary for large or multisite intervention programs to catch issues arising in different regions or populations.
(c) Develop the stress test protocol
Evaluators and the PMT should develop a set of preliminary “what if scenarios” for stimulating discussion among stress test participants and a detailed protocol for conducting the stress test. A safe environment for discussion secures participants’ honesty without fear of punishment or consequences; therefore, the protocol should include a confidentiality statement of shared responsibility to protect participants’ identity and the exchanged information.
(d). Identify and invite participants
Once the format and protocol are developed, evaluators and the PMT select participants for the stress test. Best candidates have good knowledge of the program and diverse backgrounds and roles in the program (leadership, managing, coordinating, and/or implementing). Subject matter experts may also be considered. The ideal group size is 6–12 members, to allow sufficient time for simulation and discussion.
One important question is “Should program managers participate alongside implementers in stress testing?” The answer is “yes” if implementers do not subordinate but rather collaborate with managers for the purpose of the program. The managers could answer questions that implementers could not. However, in many organizations, program managers also supervise implementers. Fear of any form of punishment can deter implementers from freely expressing genuine thought and opinions about the program in the presence of managers; thus, they should not participate in stress testing alongside implementers. In this situation, evaluators may consider gathering their input during previous stages and/or invite them at the end of the stress test meeting for further discussion and solution brainstorming.
Phase 2: Execution of the Stress Test
Conducting the stress test is a three-step process including the stress test introduction, facilitation, and debriefing.
Step 1. Introducing the stress test
Introducing the stress test to the participants is important for two reasons: to secure participants’ buy-in, therefore their active engagement in the stress test, and to prime them to think in terms of deconstructing the program. At a minimum, participants should understand that they will engage in a mental simulation exercise meant to exaggerate the probability of program dysfunction in order to identify its structural and functional problems. All participants need to sign the confidentiality statement and consent to audio recording the meeting. The stress test facilitators conclude this step with an overview of the program model to ensure a common understanding of the program and to reiterate the program components that will be stressed.
Step 2: Facilitate stress testing
The facilitators will follow the established protocol to engage participants in mental simulations using the following techniques: (1) amplify an issue and discuss its implications for other program components, (2) note concurrence of participants on the issue, and (3) explore the chain of consequences (secondary effects, either positive or negative). The simulation exercise can start with an existing issue, barrier, difficulty, or problem that is analyzed from the perspective of different participants with different roles in the program. Participants may also volunteer to report their observations. It befalls facilitators to observe the established protocol at all times and to skillfully channel the discussion to obtain useful data, until all topics are exhausted. Once a program component has been adequately stressed, the group should brainstorm for solutions and discuss the extent to which they are feasible, given the program resources.
Step 3: Debriefing
Given the purpose of the program stress test, all topics discussed during the meeting have a negative connotation. This focus could overwhelm participants with negative impressions about the program’s future. Facilitators need to de-stress participants at the end of the session by restating the purpose of the stress test to amplify potential problems and by reminding participants about the current accomplishments of the program.
Phase 3: Data Analysis and Reporting
Accurate analysis and reporting of stress test findings highlight the stress test results and provide stakeholders the necessary impetus for program adjustments. All audiotapes, artifacts, analytic memos, and/or field notes should be transcribed and analyzed following standard qualitative data analysis (Krippendorff, 1980; Miles & Huberman, 1994). The findings should be organized around structural and functional problems uncovered through the stress test and should include the solutions proposed by the participants or elucidated by the evaluation team. The final report should be informative to the PMT to support structural adjustments and enhance program function.
When Is the Ex-Post Stress Test Recommended?
To answer this question, we considered the type of program (newly designed or adopted from existing programs) and the implementation stage of the program (before or after initial implementation) to provide some guidance for adopting the stress test as an evaluation tool.
New Programs
A new program often relies on many assumptions about the soundness of its components, such as implementers’ competency and commitment, partners’ willingness and availability to collaborate, clients’ acceptance of services, and community support for the program. When a new program is designed for success by depending on optimal structure and function, it is bound to suffer when these resources lapse or are challenged by the program’s environment. The program stress test can assess resilience, anticipate issues, and proactively design solutions that shield new programs from failure.
Adopted programs
Oftentimes, organizations must adopt a program that already exists. Funding agencies commonly give preference to evidence-based interventions (EBIs; documented through randomized control trials) for funding community programs. However, community-based organizations rarely have the expertise, capacity, culture, and/or local resources to replicate EBIs. As a result, they struggle to adapt EBIs to their local conditions. The program stress test can identify structural and functional issues arising from adapted programs. The above principle can be applied to other adopted programs or promising practices.
Application of Program Stress Test to a Retention Program for Nursing Students
The faculty of a nursing college developed a homegrown student retention program funded by the federal government. The program targeted students from backgrounds underrepresented in the nursing workforce. By homegrown we mean that, while the program was developed based on best practices described in the literature with the target students in mind, the program as a whole was not an established evidence-based program. The program had many structural components that stakeholders designed without actually knowing how they were going to function in reality. Both a logic model and an action model/change model schema were developed for the program. A stress test was conducted as proposed in the evaluation plan at the end of the first year of implementation.
The following combines the process used and experience gained from the stress test to demonstrate the potential for stress testing as a novel evaluation tool. In order to protect the program and participants, the results shared are general findings and serve as an exemplar of potential findings from a stress test.
Application of the Ex-Post Stress Test
The stress test was implemented according to the described methodology.
Phase 1: Preparation for the stress test
Evaluators explained the stress test to the PMT in detail and the potential value it could bring to the program. The evaluators recommended to use the schema created during the planning phase of the program (Figure 2) as a guiding framework and recommended a focus group method for the stress test. Evaluators and the PMT examined the schema and agreed that the program’s change model was sound, and the program’s potential weaknesses were within the action model. As a result, the stress test focused on the action model of the program.
The action model/change model schema of the retention program.
The format selected was a 4-hr focus group with brainstorming activities. Stakeholders completed a short survey to identify program components for stressing. Evaluators and the PMT developed the stress test protocol which was shared with participants and which included an introduction, questions, and the action model/change model schema. Examples of questions designed to generate discussion included the following: Why is this component a stress issue? Provide examples. What components of the program burden students or faculty unintendedly? What components of the program could become sustainable at the organizational level?
An activity was designed to involve all participants to collaborate in completing a table linking program components to monitoring activities, outcomes, and sustainability potential. The PMT identified key stakeholders to participate in stress testing the program, all with extensive experiences in managing activities and/or providing services to students in the retention program. Because PMT did not have supervision authority over the implementers at the college level, they participated in the stress test alongside implementers.
Phase 2: Execution of the stress test
The evaluators (facilitators) explained the purpose and procedure of the stress test, provided an overview of the schema for the program, and secured signed confidentiality agreements. The brainstorming session explored the structure and function of each program component, responsibilities in the program and other job responsibilities of participants, the support received from the program, and structural and/or functional issues encountered since the beginning of the program. Occasionally, facilitators had to push the participants to elaborate on the negative aspects of the program using the stressing techniques discussed previously. At the end of the exercise, the facilitators de-stressed all participants.
Phase 3: Data analysis and reporting
The audiotaped meeting was transcribed verbatim. Evaluators identified potential problems with high concurrence among participants along with their recommendations. Next, the evaluation team organized the participants’ statements into categories based upon semantic and conceptual similarity and then abstracted major themes for each stressed component. Two evaluators met 4 times to discuss the analyses until they reached consensus. A report of the findings was sent to the PMT and then finalized by incorporating their comments/suggestions and feedback. Evaluators reiterated caution for the appropriate use of results, such as for improving the program and preparing contingency plans or for scaling up the program.
Major Findings and Recommendations
With permission from the PMT, this article reports aggregate and general findings of the stress test as an exemplar to illustrate what kind of information may be provided by a stress test and how the information may be useful to stakeholders.
Structural issues
One structural issue discovered was related to turnover among implementers. There is a critical nursing faculty shortage nationwide with a faculty vacancy rate estimated to be 7.9% (American Association of Colleges of Nursing, 2019). In each year of the program, the PMT experienced turnover among faculty implementers and leaders who were difficult to replace. However, the PMT was interested in a more sustainable staffing plan. For example, one proposed solution was a succession plan in which coordinators for each service area were identified and groomed to fill leadership vacancies when they arose.
Another structural issue was noted in the recruitment of the target group. The PMT did not receive sufficient applications from students eligible for the program despite the program’s attractiveness. During previous formative evaluations, students expressed appreciation for all interventions and reported that many nonparticipating students would have liked to qualify for the program as well. However, early identification of a potential pool of applicants had multiple barriers including late applicant matriculation decisions, applicant selection time lines, marketing, and application processes. A solution discovered through the stress test was a comprehensive, streamlined recruitment strategy.
Functional issues
An example of a functional issue was related to communication. While the PMT met every other week, team meetings were difficult to schedule and attend due to the high workload of faculty, different teaching and clinical schedules, and increased responsibilities associated with delivering program interventions. Consequently, the PMT received sparse feedback from implementers to inform decision making, and implementers had difficulty keeping current with management decisions. Therefore, the use of team meetings as the main vehicle for communication was ineffective. Multiple communication strategies were suggested during the stress test and have been piloted. The most effective strategy to date has been using the full-time grant manager as the central point of communication for all stakeholders. As a conduit of information, the grant manager makes sure all communication loops are closed and conveys updates to the PMT weekly.
Conclusion and Discussion
This article introduced the concept, purpose, timing, methodology, and procedures of the ex-post program stress test. Furthermore, we illustrated its application to a retention program for nursing students. Guided by evaluators, participants were able to stress test the program and identify several structural and functional vulnerable areas, as well as provide concrete recommendations for addressing these problems.
Many evaluators will ask the following question: What kind of data is the stress test providing that formative evaluation cannot? We provide an example to clarify the difference.
Having conducted three formative evaluation waves (Chen et al., 2019) for the nursing retention program, we were very clear on program structure and service delivery protocols. We found some issues, but we couldn’t see clearly upstream to understand the underlying factors and conditions fueling these issues. Neither could these issues be uncovered unless stakeholders, representative of each program component, voiced their opinion in open dialogue, asking and answering questions in an exaggerated scenario of “what if” during the stress test. For example, during formative evaluations, mentors expressed frustration with lack of a communication platform other than PMT meetings. To address this issue, PMT archived meeting minutes and made them available to all implementers. However, communication with and from implementers did not improve a great deal. At the end of the academic year, the stress test discussed the leadership turnover issue that apparently had nothing to do with communication within the structure of the program. However, one of the proposed solutions—grooming new leaders by assigning coordinators to each type of service area—responded to future leadership turnover, provided a liaison between implementers and PMT, which addressed communication issues, and contributed to closing the feedback loop between different service areas. Formative evaluation or organizational capacity assessment could have not provided such comprehensive solutions because they would have not found the problem, which, apparently, was unrelated to the downstream issues.
We applied the action model/change model schema for describing the program because the schema provides components that are useful for stakeholders to diagnose problematic areas. Although the application of the schema was highly beneficial, we consider it is just one of the available models; we encourage evaluators to try different modeling tools for this purpose. Similarly, we applied a mental simulation to conduct the program stress test in this study, but we encourage evaluators to exploit other approaches such as computer simulation or mixed methods.
After decades of practice, many evaluators agree that program evaluation could benefit from additional practical tools. Many practical issues encountered and solved ad-hoc by implementers never make it into reports. This practice hinders the development of the body of knowledge for program implementation or program adaptation. Without a rich and detailed literature, the process of improving programs becomes a slow and rather frustrating task. Practical tools such as a program stress test are useful to document and cumulate implementation knowledge and practice from communities.
It is worthy to note that the concept and methodology described in this article represent an ex-post evaluation because the stress test was conducted after the initial implementation, since the program was new (homegrown). As the authors developed the concept, it became clear that the stress test can also benefit EBIs that undergo an adaptation process by the implementing organization, to fit the local conditions of their communities. These adaptations can be tailored during the planning phase through an ex-ante stress test, similar to the Regulatory Impact Assessment (Kirkpatrick & Parker, 2004). Such an approach would clearly inform whether the program plan lacks consistency with available resources, existing infrastructure, and/or capacity to implement the EBI with the necessary adaptations, as well as identify subsequent functional pitfalls. Stakeholders can use this information to strengthen the planning process. Currently, the evaluation team is working on the next paper discussing the strategy, methodology, and application of the ex-ante stress test for program planning assessment.
This study calls the attention of the evaluation community to differences between past- and future-oriented evaluation approaches. Many traditional approaches focus on a program’s past activities and performance. For example, outcome evaluation assesses whether a program was effective. Process evaluation assesses how a program was implemented. Past-oriented evaluations provide information for stakeholders to improve the program reactively. This study illustrates that proactive evaluation is also possible by anticipating potential breaking points, structural or functional. The program stress test uses a future-oriented lens to uncover, identify, and support proactive action to address these weaknesses. The authors encourage evaluators to develop more future-oriented or proactive evaluation tools to further enhance the evaluation usefulness to stakeholders.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: Health Resources and Services Administration (Grant No. D19HP29808).
Acknowledgement
The authors greatly appreciate the anonymous reviewers’ valuable comments.
