Abstract
Intensive interventions are time- and resource-demanding interventions designed to be implemented with a single student with unique learning needs. Economic evaluation provides a methodology for evaluating the time and material resource costs of implementing these interventions to provide detailed feasibility information for educators considering their implementation. This study presents a cost analysis of the time and materials required to implement an intensive intervention, I-Connect, and compares those costs to the reported effects, from the perspective of the intervention agent and recipient (i.e., teacher and student) using time as the primary cost metric. The results suggest the total time cost of implementing I-Connect falls within the time teachers and students are likely to have available. Furthermore, teachers will likely find I-Connect to be a cost-effective option for generating a 50% increase in on-task behavior if they have an initial 40 min of available time to become familiar with the intervention procedures, prepare the intervention, train the student to engage in self-monitoring, and initiate a 10-min session of I-Connect. Limitations and future research directions for the economic evaluation of intensive interventions are discussed.
Economic evaluation is used to evaluate the specific personnel, training, materials, facilities, and other resources that are consumed when preparing and implementing an intervention (Levin et al., 2018). These methods provide valuable feasibility information for educators to determine if they have access to or can readily acquire the necessary resources to implement the intervention (Detrich, 2020; Scheibel et al., 2022). Economic evaluation typically begins with an initial cost analysis to catalog the various resources used during the implementation of an intervention and the calculation of the total cost of these resources (i.e., implementation costs; Levin et al., 2018). Beyond the initial cost analysis, additional economic evaluation methods can be applied to compare implementation costs with the effects achieved by the intervention (i.e., cost-effectiveness analysis) or long-term outcomes (i.e., cost–benefit analysis; Belfield & Bowden, 2019). Cost-effectiveness analysis provides implementors not only with information about the likelihood of effectiveness but the resources needed to achieve those effects. Despite the use of these methods becoming increasingly popular in education research, economic evaluation methods remain largely unapplied to intensive interventions received by students with or at risk for disabilities. Given the potential utility of these methods in supporting educators to select feasible and effective interventions, there is a need to examine the utility of applying economic evaluation methods to intensive interventions.
Economic Evaluation
Cost analysis
Cost analyses in education traditionally utilize the Ingredients Method to conduct an initial cost analysis (Institute of Education Sciences [IES], 2020; Levin et al., 2018). The Ingredients Method includes four steps: (a) determine the decision-maker perspective from which the costs will be analyzed (i.e., who will be paying the intervention cost?), (b) review implementation scenarios to determine all resources consumed during the intervention, (c) assign a value to those ingredients specific to the implementation context, and (d) calculate a total cost of all ingredients consumed during the intervention (Levin et al., 2018). Most often in education research, intervention cost is considered organizationally from the department, district, or state perspective using a dollar cost metric. See Cost Analysis: A Toolkit (Institute of Education Sciences, 2020), “Cost Analysis Methodology,” for an in-depth guide to these methods. Cost analyses provide a foundation for further evaluations of costs and intervention outcomes, including the cost-effectiveness analysis (Belfield & Bowden, 2019). See Table 1 for a summary of economic evaluation methods.
Economic Evaluation Methods and Single-Case Research Design (SCRD) Considerations Summary.
Cost-effectiveness analysis
Cost-effectiveness analysis is increasingly popular in education research as it provides a method of comparing the total cost required to implement an intervention with gains in student outcomes (Levin et al., 2018). Comparison of the total implementation cost to the effect an intervention produces (i.e., cost-effectiveness analysis) allows decision-makers to determine if the effect of the intervention is comparable with consumed resources and the time spent by teacher and student during implementation (Levin et al., 2018; Scheibel et al., 2022). Conducting a cost–effect analysis requires both an accounting of the total costs of the intervention using a common cost metric (e.g., dollars) and an analysis of a single outcome using a statistically derived effect size that allows for comparison across interventions (Belfield & Bowden, 2019). Cost–effect analysis yields a ratio of total cost to effect (i.e., cost–effect ratio).
Further analysis of cost-effectiveness
Relationships between total cost and effect can be further analyzed using cost-effectiveness planes and cost-effectiveness acceptability curves (CEACs) (Hoch et al., 2019; Levin et al., 2018). Cost-effectiveness planes allow for the examination of unique implementation cost exemplars (i.e., implementation across environments or participants) compared with different outcome possibilities (e.g., more or less effect or time availability; Levin et al., 2018). The cost-effectiveness plane then allows for further comparison with a willingness-to-pay threshold, or a contextualized comparison of the intervention effect to a desired cost threshold (Hoch et al., 2019). For example, an intervention may be likely to demonstrate an effect (e.g., an increase in test scores to meet state standards) but requires at least a minimum amount of cost to achieve this effect. Willingness to pay can be captured using a CEAC to visualize the joint distribution between the costs and effects of an intervention (Fenwick et al., 2006). Furthermore, findings from the CEAC translate into highly useful practical guidance for educators regarding the minimum amount of resources needed to achieve a desired effect, allowing educators to determine if the intervention is feasible in their classroom.
Economic Evaluation of Intensive Interventions
Economic evaluation methods of interventions for students with and at risk of disabilities are beginning to emerge in the research literature. Recent cost studies have examined service delivery systems (e.g., Burgin et al., 2020), comprehensive treatment models (e.g., Cidav et al., 2017; Scheibel et al., 2022), and individualized initiatives (e.g., Barrett et al., 2020) to support students receiving special education services. These cost studies, like general education cost studies, have examined multi-student interventions from the cost perspective of a school- or state-level administrator considering the cost to the district when implementing across large groups of students (Detrich, 2020). However, intensive interventions differ from multi-student interventions in how they are prepared (for a single student), delivered (grouped together in individualized education programs [IEPs]), and selected (by teachers instead of administrators). In addition, the individualized nature of these interventions often presents a challenge to educators who often report a lack of material resources, time, and training as significant barriers to implementing these interventions (Locke et al., 2015; Suhrheinrich et al., 2020). Given the critical role of resource availability when selecting an intervention, a full accounting of intensive intervention cost would assist special educators and IEP teams in selecting intensive interventions that can be effectively and feasible implemented (Leko et al., 2019; Scheibel et al., 2022).
Considerations for intensive interventions
Calculating the total implementation costs of intensive interventions requires adaptation to the cost perspective and metric used in traditional economic evaluation methods. First, traditional economic evaluation considers intervention cost from a school or agency cost perspective using a monetary cost metric (i.e., dollars) to inform administrators or policymakers seeking to maximize funds. However, given that intensive interventions are selected and implemented by teachers, consideration of these costs from the perspective of the implementor (i.e., the teacher) would produce more relevant and useful information to inform intervention selection decisions. Though this shift in perspective is unlikely to substantially change the methods under which costs are gathered, this adjustment impacts the metric by which costs are measured.
As previously noted, traditional economic evaluation uses the dollar cost metric to estimate costs. Though funding to purchase material resources is a perennial concern in special education and can affect implementation decisions (Suhrheinrich et al., 2020), we propose that implementors may consider time to be a valuable measure of cost when selecting intensive interventions when presented alongside dollar cost. Time is an essential indicator of feasibility for many implementors, who are often considering not only the intervention but also the availability of staffing coverage to implement an intervention or planning time to prepare interventions. Adequate time to prepare and implement interventions is critical to the implementation of intensive interventions as lack of time can impact implementation quality and limit intervention effectiveness (Locke et al., 2015; Zhang et al., 2021). Furthermore, to address these concerns, we propose analyzing implementation time costs in conjunction with the dollar cost of material resources. This adaptation provides valuable intervention details to facilitate implementor selection of interventions that can be feasibly implemented with available time and material resources (Scheibel et al., 2022).
Considerations for single-case design research methods
Intensive interventions present an additional complexity when comparing intervention cost to the anticipated effects. These interventions are likely to be investigated using single-case design (SCD) methodology and direct observation outcome measures, as opposed to the standardized measures employed in group design research methods. The latest standards for SCD research in special education highlight the role of economic evaluation in a call for improving the validity of SCDs through considerations of “transparency, ethics, and cost” (Ledford et al., 2023, p. 8). Furthermore, the standards emphasize the use of endogenous settings and implementers to improve understanding of how these interventions will perform in typical contexts (Ledford et al., 2023).
The use of endogenous settings and implementors provides an approximation of the costs a real-world (i.e., authentic) implementor may anticipate when implementing an intensive intervention. To do so, researchers must calculate the costs of the intervention planning and implementation to understand the time, material, and personnel costs associated with the intervention (Ledford et al., 2023). When it is not possible to conduct intervention research in endogenous settings with endogenous implementers, cost data should be reported to support the consumer in understanding the costs associated with conditions under which the desired “optimal outcomes [were] demonstrated in tightly controlled clinical settings” (Ledford et al., 2023, p. 11). In other words, interventions evaluated in tightly controlled research conditions must include reports of the time, material, and personnel costs of achieving results like those in research studies for consumers in conditions dissimilar to research (e.g., educators in classroom settings not equipped with intervention creators or data collectors). Given reporting and analyzing intervention costs now falls under primary study reporting guidelines and is required by education funding agencies (e.g., IES, 2020), methods for calculating the cost of intensive interventions using SCD research must be created to support understanding the requirements and feasibility of adopting interventions in typical settings.
Reporting and interpreting these cost data for intensive intervention studies can serve multiple purposes. First, it can provide a measure of the social validity of the intervention through an objective measure of the feasibility and/or acceptability of an intervention based on the time, material, and personnel costs associated with the intervention. The cost–effect plane/acceptability curve, detailed above, may provide a visual representation of a different perspective or index of social validity called for since the initiation of SCD research (Wolf, 1978). Second, cost data, including time cost analyses and cost-effectiveness, may be used to support intervention syntheses so consumers can understand the predictable bounds of how much cost (time, material, personnel) can be anticipated to achieve an overall effect using intervention X for outcome Y with population Z in a specific context (Ledford et al., 2023). Incorporating cost data in evidence-based syntheses of intensive interventions can provide a generalizable cost–effect finding to support teachers addressing IEP goals or administrators considering staff training on specific intensive interventions.
Providing a generalizable cost effect may be particularly critical given current advancements in SCD meta-analysis, the method most used for synthesizing outcomes across intensive interventions in education research. The use of parametric effect size estimates to quantify the magnitude of intervention effects is emerging in SCD synthesis. However, interpretation of these values is limited to very few outcomes using the log response ratio index (see Chow et al., 2023). While quantitative indices of non-overlap between conditions are interpretable, these fail to capture the magnitude of intervention effects (Moeyaert et al., 2018). However, the use of direct observational measurement in SCDs is a unique asset when examining implementation costs of intensive interventions as direct behavior change is captured in a time series and reported in each intervention study. These data allow for implementation costs to be compared with the change in a specific behavior or skill over time. Comparison of the time and material costs of an intervention with direct behavior change provides a more interpretable metric for implementors to determine if the outcome justifies the time and materials required to implement the intervention (Scheibel et al., 2022).
Study Purpose
This study applies economic evaluation methods to an exemplar intensive intervention, I-Connect, a technology-based self-monitoring (SM) intervention with a rigorous single-case evidence base indicating positive academic engagement outcomes. The study uses economic evaluation methods to examine the total costs associated with achieving positive outcomes for students with I-Connect from the highly focused perspective of the target end-user (e.g., a special education teacher) and target beneficiary (e.g., students with disabilities). I-Connect was selected as an exemplary intervention for this methodology as it is a manualized intervention, which facilitates cataloging of ingredients, and is supported by a substantial evidence base of single-case research that has been meta-analyzed to estimate an average effect size. The following research questions were addressed: RQ1. How much time is needed to implement I-Connect? RQ2. How does implementation time compare with the effect of I-Connect under research conditions?
Method
Description of I-Connect Technology
I-Connect is a technology-based SM platform that allows educators to design and track the progress of SM interventions for multiple students within a web portal. The web portal allows educators to set up student accounts, create individualized SM interventions to be used in various locations (e.g., math, language arts, and home), view student progress charts, and share chart access with relevant stakeholders (e.g., parents, co-teachers, and community support staff). Students then implement the SM intervention using the I-Connect app which can be downloaded on desktop or mobile platforms to best fit student needs. Furthermore, comprehensive implementation supports are freely available on the I-Connect website to support educators to implement I-Connect with fidelity, including a user guide, self-paced study guide, and student training and implementation checklists (I-Connect, n.d.).
I-Connect implementation procedures
Implementation of I-Connect requires the teacher to first become familiar with I-Connect and SM interventions (i.e., teacher training), select the SM prompt and interval schedule for the prompt to appear, and set a monitoring goal for the student on the I-Connect web portal (i.e., intervention preparation). Next, the teacher trains the student to use the I-Connect app to self-monitor their behavior accurately (i.e., student training). Once the student monitors accurately, the teacher directs the student to start monitoring to improve their on-task behavior (i.e., pre-intervention activities), then the student uses I-Connect during academic tasks (i.e., during intervention activities). Finally, at the end of the session, the teacher directs the student to end the session and checks in with the student (i.e., post-intervention activities). Though I-Connect implementation procedures are manualized, the intervention allows for multiple adaptations to individualize the intervention to student needs resulting in varied student training and implementation procedures across the evidence base (I-Connect, n.d.).
I-Connect evidence base
A recent synthesis of the I-Connect was conducted to determine for whom and understand what conditions I-Connect is effective (Scheibel et al., 2023). Results of this study indicate I-Connect demonstrates positive effects with students across special education eligibility categories (e.g., autism spectrum disorder [ASD], learning disabilities, and intellectual disabilities), medical diagnoses (e.g., attention-deficit hyperactivity disorder [ADHD]), settings (e.g., special and general education settings), and under varying task conditions (e.g., lecture, small group instruction, and independent work; Scheibel et al., 2023). On-task behavior was the most common outcome measured using I-Connect across the evidence base. Meta-analysis of these studies found an average effect of I-Connect for on-task behavior is LRRi = 0.86 (95% confidence interval [CI] = [0.68, 1.04]), and an unweighted average percentage change across conditions was found to be 198% (range = 19%–834%) across intervention packages (Scheibel et al., 2023). Variations were noted across intervention packages, including (a) student training session durations and methods, (b) monitoring intervals, and (c) session lengths. Additional intervention descriptions and synthesis details can be found in the work of Scheibel et al. (2023).
Cost Analysis
To address our first research question, we conducted a cost analysis of I-Connect by adapting the Ingredients Method (Levin et al., 2018) to address data produced from SCDs, the methodology employed to evaluate I-Connect. The Ingredients Method steps and adaptations are described in detail below.
Step 1: Select a cost perspective
I-Connect is an intensive intervention designed to be implemented with individual students and adapted to student needs. SM interventions, including I-Connect, are regularly used in special education to address individual IEP goals or objectives (Briesch et al., 2019; Scheibel et al., 2023). These intervention features indicate the teacher would serve as primary cost bearers likely to evaluate time and material resource availability. The primary decision-maker role of teachers and students in selecting the intervention would suggest a teacher and student cost perspective would be more useful than one of administration (i.e., systems) or state agency (i.e., society).
Step 2: Generate ingredients list of consumed resources
To examine the average implementation costs to achieve the effects described in the I-Connect evidence base, we gathered a list of ingredients specific to the implementation of I-Connect under research conditions. First, we reviewed I-Connect implementation supports (I-Connect, n.d.) to identify the following implementation steps: (a) teacher training, (b) student training, and (c) intervention use. Next, we identified the following cost categories within the intervention steps: teacher training activities, teacher intervention preparation activities, and student training activities. Using the I-Connect evidence base identified in Scheibel et al. (2023), we collected personnel time estimates, training needs, material and facility usage, and student time for each implementation step reported in primary studies. We considered the 14 included SCDs as 14 individual implementation exemplars to reflect the unique implementation conditions and ingredients reported across participants. To identify materials used exclusively to implement I-Connect, we cataloged all materials and then reviewed these materials to identify those used for the exclusive purpose of implementing I-Connect or consumed during the intervention procedures (i.e., use of ingredients during the intervention prohibited other or future use). We considered all ingredients exclusive to the intervention as ingredients. In the absence of reported cost data, we reviewed implementation supports (I-Connect, n.d.) and confirmed with the intervention developer. Finally, we differentiated I-Connect ingredients into one-time costs associated with initiating the intervention with the student (i.e., startup costs) and repeated costs that were expected to occur over the lifetime of the intervention (i.e., repeated costs).
We identified I-Connect startup costs as teacher training activities, student training activities, and intervention preparation activities. Each of these activities was identified or described at the start of the intervention but was not repeated once the intervention was in place. We defined teacher training time as the time a teacher spent learning how to implement I-Connect, including how to navigate the technology portion of the intervention (e.g., using a web platform to set up accounts and SM intervention) and reviewing the components and hypothesized mechanism of change of SM interventions. We defined teacher intervention preparation activities as the time the teacher spent setting up teacher and student accounts in the I-Connect web portal, designing the SM intervention for specific student needs, and ensuring the I-Connect application was downloaded on a student device.
We identified I-Connect repeated costs using intervention procedure descriptions from primary studies and implementation supports to be pre-, during, and post-intervention activities, which were reported to occur each time I-Connect was used (i.e., each monitoring session) and repeated through the lifetime of the intervention. We defined teacher-led pre-intervention activities as delivery of the device to the student and provision of direction or supervision to ensure the student opened the I-Connect app, logged in, and started the monitoring session. Student-led pre-intervention was defined as opening the I-Connect app, logging in, and starting the monitoring session. We defined teacher-led during intervention activities as supervision of the student throughout the monitoring session. For exemplars investigating the effects of I-Connect without supplemental reinforcement (i.e., I-Connect), we determined student supervision to be consistent with typical classroom supervision duties and thus was not a unique intervention ingredient. For exemplars describing the use of I-Connect with supplemental reinforcement (i.e., I-Connect+SR), reinforcement was provided contingent upon accurate monitoring (e.g., recording “yes” when on-task) indicating the teacher was required to observe or co-monitor with the student at each monitoring interval. We defined student-led during intervention activities as the time required to acknowledge, navigate to, and respond to the prompt to record their behavior and return attention back to the task. We defined teacher-led I-Connect post-intervention activities as directing the student to stop the monitoring session, retrieving the device or supervising the termination of the app, and engaging in a short discussion of student performance. We defined student-led I-Connect post-intervention activities as ending the monitoring session and engaging in a short discussion regarding student performance. For I-Connect+SR exemplars, post-intervention activities were defined as consistent with I-Connect exemplars. See Table 2 for a list of intervention ingredients and descriptions.
Ingredients List and Average Cost Valuation From the I-Connect Evidence Base.
Note. Time cost values represent average time costs reported or estimated in the evidence base including both I-Connect and I-Connect with Reinforcement. NE = material determined to be non-exclusive to intervention and cost value not estimated.
Step 3: Valuate cost ingredients
For teachers and students considering the use of I-Connect, time is suggested as the primary cost metric most meaningful to decision-makers, as time is of critical importance when determining intervention feasibility for the primary cost bearers (i.e., teachers and students; Bettini et al., 2016; Vannest & Hagan-Burke, 2010; Zhang et al., 2021). We valuated startup time costs for teacher training, intervention preparation, and student training costs using the duration reported in exemplars or implementation supports. In the absence of reported durations, estimates were obtained from intervention developers. We recorded durations separately for teachers and students engaging in the same activity (e.g., student training duration times were reported for both student and teacher startup costs). In addition, if multiple personnel were reported for startup costs, we reported duration for both individuals (e.g., teacher and paraprofessional).
We valuated repeated time costs of pre- and post-intervention activities using the duration reported in primary manuscripts or estimated by intervention developers. We used the product of an estimated duration of during intervention activities and the number of monitoring opportunities that occurred during the monitoring session to valuate time cost of during intervention activities. We defined monitoring opportunities as the number of times the student responded to the prompt during the monitoring session (i.e., the monitoring interval length). Though SM interventions are typically used in practice for varying lengths of time, primary studies often report consistent durations across monitoring sessions that vary across studies, for example, I-Connect evidence-based exemplars reported monitoring sessions ranging from 10 to 30 min. To allow for comparison across these exemplars, we scaled the number of monitoring opportunities to 10-min monitoring session for all exemplars. This was done by determining the number of monitoring opportunities that occurred within a 10-min observation window at the reported monitoring schedule for each exemplar. For example, an exemplar with a participant who monitored at a 30 s monitoring interval for a 15-min monitoring session would have 30 monitoring opportunities during the reported monitoring session, and 20 monitoring opportunities during a 10-min observation window. We totaled pre-, during, and post-implementation time costs and reported these totals as active implementation time (AIT) cost for each 10-min session. Finally, we valuated materials exclusive to I-Connect implementation separately using a dollar cost metric to allow for a dollar and time cost comparison.
Step 4: Total cost calculation
We calculated the total time cost for startup costs by totaling the durations of teacher training, intervention preparation, and student training time costs and reported in minutes for both the teacher and student. We reported the total time costs for repeated costs in minutes by session, reflecting pre- and post-intervention activity durations and during intervention durations for 10-min observation windows to better allow for comparison across exemplars.
Cost-Effectiveness Analysis
To address our second research question (How does implementation time compare with the effect of I-Connect under research conditions?), a cost-effectiveness analysis of I-Connect was conducted. This analysis was selected as it provides intervention decision-makers (i.e., teachers) with a comparison of the anticipated outcomes of this intervention with implementation costs to allow decision-makers to determine if implementation is feasible in their classroom (Detrich, 2020; Scheibel et al., 2022). The individual-level cost data for this cost-effectiveness analysis combined total costs calculated in the aforementioned cost analysis, and individual-level effect data from an earlier meta-analysis of I-Connect SCD studies (Scheibel et al., 2023). Student outcomes reported by Scheibel et al. (2023) included “on-task” behavior using operational definitions that varied topographically across designs but consistently measured the academic or task engagement of an individual engaging in an academic task during an instructional period. On-task behavior was measured using momentary time sampling and was reported as a percentage of time on task in all designs (Scheibel et al., 2023). A meta-analysis of on-task behavior outcomes using I-Connect indicated positive effects across all SCDs, and similar effect size estimates when I-Connect was used in isolation (i.e., I-Connect) or combined with supplemental reinforcement (i.e., I-Connect+SR; Scheibel et al., 2023).
To conduct the cost-effective analysis, we used the average effect across the demonstration as our effect estimate for the SCD (i.e., exemplar) within SCDs that had multiple demonstrations of effects per design. To compare individual-level effects more meaningfully across studies, the sizes of effects for each exemplar were quantified using the Log Response Ratio-increasing (LRRi; Pustejovsky, 2018). The LRRi is a parametric effect size that measures outcomes using a natural logarithm of proportion change between SCD phases and is translatable into a measure of percentage change from baseline to intervention (Pustejovsky, 2018). The benefit of this effect size metric is that it is less sensitive to variation in study procedures. These methods were consistent with those employed by Scheibel et al. (2023), and more details about the effect of data use in this study can be found in this study.
To conduct a cost-effectiveness analysis, we first created a cost-effectiveness plane depicting costs and effect data for all 14 exemplars. The cost-effectiveness plane provides a visualization of the relationship between individual implementation exemplars (i.e., participants within the evidence base) and estimated effects associated with each implementation exemplar. To allow for more practical interpretation, the LRRi percent change metric (LRRi%) was used to compare total time costs with intervention effects. To allow for a comparison of time cost and effect within the context of likely available time, we provided a reference line for the estimated time available to teachers and students during the lifetime of the intervention. We calculated likely available time estimates using percentages of time spent engaging in tasks across the school day reported by special educators in the work of Vannest and Hagan-Burke (2010). Tasks included in the work of Vannest and Hagan-Burke (2010) included meetings, non-academic instruction, academic instruction, and planning tasks. For this study, likely available estimates were made using daily planning time (5.4%) and non-academic instruction time (4.4%) estimates reported by Vannest and Hagan-Burke (2010) to calculate a daily planning time value (n = 19.44 min) and daily non-academic instruction time value (n = 15.84 min) based on a 360-min school day. We calculated teacher available time by multiplying non-academic instruction time by the number of exemplar sessions and adding a single planning time value. We calculated student available time by multiplying non-academic instruction time by the number of sessions. We calculated available time individually for each exemplar, then we averaged these times to reflect the average time a teacher and student would likely be available.
Next, we evaluated the probability that I-Connect will generate the desired effect at an acceptable cost. In this cost-effectiveness analysis, the desired effect was a 50% improvement relative to baseline (e.g., on-task behavior increases from 60% to 90%). Incidentally, we chose 50% improvement to be the desired effect for generality, but we provide an annotated SAS script on the Open Science Framework at https://tinyurl.com/5n82eryr that interested readers can use to rerun this same cost-effectiveness analysis with any desired effect value. To implement cost-effectiveness analysis, we fitted a net benefit regression model to the sample data using a Markov Chain Monte Carlo (MCMC) simulation procedure in SAS software (PROC MCMC; SAS 9.4) and second, using the output, graphed the CEAC (Hoch et al., 2019). This curve shows the probability that I-Connect will generate the desired effect at an acceptable cost change as the amount of available time increases.
In CEAC analysis, the “acceptable cost” is defined by the parameter λ (i.e., the acceptable cost ceiling or the maximum amount of time a teacher or student has available) such that any cost (i.e., time cost estimate) for I-Connect exceeding the value of λ falls outside the cost a teacher or student is willing to “pay” for the intervention. Specifically, λ represents the maximum amount of time that teachers or students are willing to allocate toward I-Connect to obtain the desired effect. The CEAC compares λ (i.e., the acceptable cost ceiling) along the x-axis to the conditional probability a desired effect is achieved on the y-axis. This comparison allows readers to select an acceptable cost ceiling that represents their local implementation context to determine if I-Connect will probably prove to be a cost-effective option for them by producing desired effects within their context.
To be thorough, we graphed the probability that I-Connect will be cost-effective for teachers or students with λ values starting at 0 min and extending to 80 min to ensure the curve had sufficient time to stabilize toward an asymptote. In this case, the extremes can be interpreted such that when λ = 0 the teacher or student has no available time to implement or receive I-Connect to obtain the desired effect, and when λ = 80, the teacher or student is willing to spend 80 min to use I-Connect to obtain the desired effect. It is important to note that the λ value is reflective of the total time spent implementing the intervention which would amount to small installments of costs throughout the intervention period in activities, for example, training, setting up, and AIT. While the desired effect is dependent on the baseline performance of an individual student, a minimum desired effect of a 50% improvement in outcomes made sense in this context and was selected as the desired effect for graphing purposes. This desired effect indicates a student would demonstrate at least a 50% increase in baseline performance when using I-Connect (e.g., a student with on-task behavior of 40% at baseline would demonstrate an improvement to 60% on-task behavior with I-Connect). I-Connect is considered cost-effective when acceptable cost ceilings have at least a 50% probability of producing the minimum desired effect.
Results
Cost Analysis
Ingredients list
Table 3 presents an ingredients list with cost valuations for I-Connect, which were sourced from implementation exemplars reported within the I-Connect evidence base (Scheibel et al., 2023), freely available implementation supports (I-Connect, n.d.), and, as necessary, confirmed by intervention developers. Of the 14 implementation exemplars, seven were determined to include supplemental reinforcement; ingredients were initially gathered for both forms of I-Connect and then differentiated during the cost valuation step. These costs were reported as an average total across the evidence base in the ingredients list (see Table 2) and then reported by exemplar and differentiated by intervention form and cost bearer in Table 3. Personnel, training, and material ingredients were largely consistent across implementation exemplars, with a few notable exceptions. Teacher training detail was not reported in the manuscripts of any primary studies, as such this was gathered from implementation supports and applied across all implementation exemplars. Intervention preparation activities reported in primary manuscripts were largely implicitly described (e.g., a device with a downloaded app was used, indicating an app had been downloaded). To ensure these activities were described comprehensively, implementation supports were reviewed and compared with primary manuscript details to identify if additional ingredients were employed. Implementation preparation ingredients as described in implementation supports were applied across all implementation exemplars as an initial time estimate and additional ingredients were included only when reported in primary manuscripts (e.g., the use of a preference assessment). Student training ingredients were largely reported within primary manuscripts with noted variation across instructional methodologies (e.g., use of online modules or discussion and role-playing to train students). This review of intervention procedures found variation was reflected only in reports of material use and valuation of the individual ingredients, as all student training descriptions included the use of a single teacher to train a single student. Pre- and post-intervention activities were not consistently reported in primary manuscripts and implementation supports, as such the intervention developer was contacted to identify expected pre- and post-activities. These activities were applied across all implementation exemplars. During intervention, activities were noted to vary considerably across implementation exemplars, due to variations in the session use and monitoring schedule. The average session duration was 17 min (range = 10–30 min), with monitoring schedules varying between 15 s and 5 min fixed intervals, though 30 s fixed intervals were reported in 8 of the 14 implementation exemplars.
I-Connect Implementation Costs From the Evidence Base.
Note. Min = minute; AIT = Active Implementation Time; ID = Implementation exemplar identifier; IC = I-Connect only implementation exemplars; SR+ = I-Connect with supplemental reinforcement implementation exemplars; Avg = Average across implementation exemplars.
Cost valuation
Materials
A review of materials consumed during teacher training, intervention preparation, and student training activities indicated all materials, except the I-Connect app, were commonly occurring and durable materials that did not require exclusive purchase or replenishment dependent upon implementation of the intervention. The I-Connect app is a free app, available for download across all major technology platforms on mobile or desktop devices. As such, teachers implementing I-Connect would be unlikely to request funding or need to acquire new technology devices for this intervention. Furthermore, I-Connect runs in the background of most devices allowing the student to utilize I-Connect and simultaneously complete academic tasks using the device, eliminating the need for a dedicated device for I-Connect. Given the likelihood that I-Connect that I-Connect could be downloaded on a technology device readily available in the classroom, cost values were not assigned.
Startup costs
The average valuation of teacher and student startup costs across implementation exemplars can be found in Table 2 and broken down by intervention form and cost bearer in Table 3. The average teacher startup time across implementation exemplars was 85.7 min, this estimate included 30 min in teacher training time and 11.4 min of preparation time, and 44.3 min of student training time. I-Connect+SR exemplars reported longer preparation time (m = 13 min) and student training time (m = 60 min), resulting in higher startup costs (m = 102.9 min) than I-Connect (m = 68.6). The average student training time for all exemplars was 44.3 min spent by the teacher.
Repeated costs
The average valuation of teacher and student repeated costs across implementation exemplars can be found in Table 2 and broken down by intervention form and cost bearer in Table 3. All AIT-reported estimates are scaled to 10-min observation windows to allow for comparison across implementation exemplars. Pre-, during, and post-implementation ingredient costs were valued using time estimates. For all implementation exemplars in the evidence base, Teacher AIT was estimated to be 8.5 min/session, including 1 min for pre-intervention, 5 min during, and 2.5 min post-intervention activities. Student AIT for all implementation exemplars was estimated to be 6.5 min/session, with 1 min of pre-intervention time, 3 min during, and 2.5 min post-intervention. However, these estimates varied depending upon the intervention form (shown in Table 3). Teacher AIT for I-Connect consists only of 1 min pre- and post-intervention activity duration estimates. Student AIT of I-Connect includes an average of 1 min pre-intervention activities, 4.5 min of during intervention activities, and 1 min of post-intervention activities. Teacher AIT estimates for I-Connect+SR include an average of 1 min pre-intervention, 10 min during, and 2 min of post-intervention activities. Student AIT of I-Connect+SR includes an average of 1 min pre-intervention, 7 min during, and 3 min of post-intervention activities.
Total cost
The total startup time cost for teachers is 86 min, with repeated costs of 7.5 min AIT/session. The total startup time cost for a student is 50 min, with repeated time costs of 5.52 min AIT/session for students. When I-Connect is used in isolation, the total time cost for teachers is 69 min in startup costs and 2 min of AIT/session, and 40 min of startup time costs, and 4.5 min of AIT/session for students. When I-Connect is combined with supplemental reinforcement, the total time cost increases for teachers (103 min in startup costs and 13 min of AIT/session) and students (60 min in startup costs and 7 min of AIT/session). The total time cost of I-Connect across the lifetime of the intervention reported in the evidence base was an average of 167.8 min in teacher startup and repeated costs, and 131.4 min in student startup and repeated costs for an average of 11 sessions. All total cost estimates can be found in Table 3.
Cost-Effectiveness Analysis
The cost–effect ratio across implementation exemplars was 167.8 min for a 198% increase in on-task behavior, which roughly translates to a 70% increase in on-task behavior for every hour spent implementing. To compare the time cost of implementing I-Connect with intervention effects, the total costs of I-Connect were plotted on a cost-effectiveness plane (CEP; see Figure 1). The total lifetime cost of each implementation exemplar is represented on the x-axis. The intervention effect is represented on the y-axis using the average intervention LRRi% for initial and secondary AB comparisons. All implementation exemplars were clustered in the northeast CEP quadrant, indicating high cost and high effect. Teacher implementation exemplars were noted clustered in two distinct groups, differentiated by intervention form. The first group of 7 implementation exemplars reflected the use of I-Connect produced intervention effects between LRRi% = 105% and 541% (m = 236%) with a total time cost ranging between 84 and 122 min, which was well below the teacher available time estimate of 192.5 min over a 4-week period. The second group consisted of 7 I-Connect SR+ implementation exemplars that produced intervention effects between LRRi% = 55% and 610% (m = 161%) with a total time cost ranging from 170 to 206 min (m = 184 min), clustered around the teacher available time estimate. Comparison of student total time cost with intervention effect resulted in less differentiation between intervention form and all exemplars falling below the student time available estimate of 173.1 min over 4 weeks.
Teacher and student cost-effectiveness planes.
Figure 2 presents the results of net benefit analysis represented in a CEAC. Importantly, the two curves shown in this figure come at costs from the respective perspectives of the intervention agents (teachers) and immediate beneficiaries (students). For both teachers and students, CEAC analysis shows how the conditional probability that I-Connect will likely be cost-effective based on data changes as a function of the maximum number of minutes needed to see a 50% increase in outcomes. For example, the model indicates that I-Connect will likely not be a cost-effective option for a teacher with only 20 min available in their schedule to see even a 50% increase in outcomes. However, when the teacher can commit at least 40 min (and students 25 min), then the model is confident it will be a cost-effective option for them. Based on the results of CEAC analysis, readers can select which acceptable cost ceiling represents their context to determine if I-Connect will probably prove to be a cost-effective option for them. For example, in some contexts, a lower time investment (e.g., 15 min) yielding a lower percentage change in effect (e.g., a 25% increase from 60% at baseline to 80% with I-Connect) may be more feasible for the teacher and still acceptable for the student.
Cost-effectiveness acceptability curve.
Discussion
To support intensive intervention adoption decision-making, this study aimed to apply economic evaluation methods to an intensive intervention supported by a single-case research evidence base. We identified the costs of an intensive intervention, I-Connect, from evidence base implementation exemplars and valuated using reported or estimated durations of startup and repeated costs to calculate a total cost using the Ingredients Method. Next, we calculated the intervention effects from the evidence base reported in an earlier meta-analysis, compared these effects with the total costs, and then determined a cost-effectiveness threshold. The total costs of I-Connect indicate an upfront investment in time (i.e., startup costs), result in a strong intervention effect, and require a relatively small time investment over the lifetime of the intervention (i.e., repeated costs). Teachers electing to implement I-Connect with a student will likely find I-Connect to be a cost-effective option for generating a 50% increase in outcomes if they can spend at least 40 min of time to become familiar with the intervention procedures, prepare the intervention, train the student to engage in SM, and initiate a 10-min session of I-Connect. Importantly, the next time they initiate a 10-min session, they will only take a fraction of the original time as they only need to spend the AIT for training, setup, and establishing protocols that have already been paid.
Cost Analysis
The startup and repeated costs of implementing I-Connect were examined using a cost analysis from the perspective of the teacher and student using time as the primary cost metric to calculate a total time cost. We deliberately ignored the fiscal costs of purchasing that technology to highlight the cost of primary concern to teachers and students, time cost, or how much time teachers and students would need to invest to initiate I-Connect to see positive results. We felt confident that educators would not need to request administrative purchases for the specific use of I-Connect, given that I-Connect is readily available across software platforms and devices and the free availability of I-Connect app and student and mentor accounts. The cost analysis using time as a cost metric provided practical insight into the impact of intervention package features on the total time cost of the intervention. The time cost metric further allows for easy translation into a dollar cost metric as needed. This cost analysis indicates I-Connect costs $68.50 (85.7 min) in teacher training time and $6.00/I-Connect session (using national average prices for special education teachers; Chang & Head, 2022). Students received an average of 11 sessions of I-Connect for an average total cost of $65.99 (82.5 min)/student over the lifetime of the intervention.
We found the total time cost of I-Connect to be sensitive to variations in intervention packages, specifically training methodology, use of supplemental reinforcement, and length of the monitoring interval. The use of online training modules to train the student to self-monitor in implementation exemplars from Rosenbloom et al. (2019) resulted in a decrease in implementor time spent training the student, reducing the overall startup time costs by 20 min. This lack of differentiation suggests teachers with limited available time may consider the use of online modules in accordance with unique student needs. This finding is especially valuable to special educators who report limited planning time availability and competing demands that can hinder efforts to implement evidence-based practices (Barry et al., 2020; Vannest & Hagan-Burke, 2010).
Variations in the SM intervention (i.e., use of supplement reinforcement and length of monitoring interval) greatly influenced the AIT spent by the teacher. The use of reinforcement slightly increased intervention preparation time for teachers, the student training, and post-intervention AIT for teachers and students and substantially increased the during intervention AIT for teachers and students. The inclusion of supplemental reinforcement was found to substantially increase teacher during the implementation of AIT, as the use of reinforcement required the teacher to co-monitor (i.e., monitor along with the student at each monitoring interval) to ensure student accuracy and determine achievement of the reinforcement criterion had been achieved. Though co-monitoring is a recommended practice when including supplement reinforcement and was reportedly used in all I-Connect+SR implementation exemplars, it should be noted that the use of co-monitoring is not essential to the use of SM. This provides important feasibility information for teachers considering including supplemental reinforcement. For example, teachers considering the use of I-Connect+SR should determine if the increased time investment required to co-monitor is feasible in addition to their current workload, the student’s unique needs would benefit from this more intensive form of SM or if other accuracy monitoring methods (i.e., momentary time sampling to estimate on-task behavior) could be used to confirm reinforcement criterion achievement.
The length of the monitoring interval was found to have further implications for student engagement in academic tasks. Monitoring schedules ranged from 15 s to 5 min intervals across implementation exemplars; however, all but two exemplars reported 60 s intervals or less (mode = 30 s). Depending upon the duration of the monitoring interval, the time spent by the student to engage in SM could limit engagement in on-going tasks (e.g., academic tasks) given the 10 s estimation for the act of SM. For example, a student monitoring every 30 s (reported in 57% of implementation exemplars, n = 8) would have an estimated 20 s between monitoring opportunities to engage in an academic task. This finding has implications for students with ASD and ADHD who are likely to demonstrate deficits in attention and executive function abilities impacting academic achievement (McDougal et al., 2020; Visser et al., 2020). This finding may provide practical context to recent findings which conclude that longer monitoring intervals are associated with weaker gains in academic engagement than shorter monitoring intervals (Bruhn et al., 2022). Students engaging in SM to improve academic engagement at very short intervals (i.e., less than 30 s) may have limited time to engage in behaviors other than monitoring, naturally deflating the frequency of disengagement behaviors. Though practical guidance directs teachers to select a monitoring interval individualized to baseline observations of student behavior (Alberto & Troutman, 2017), limited empirical guidance exists to support teachers in selecting a monitoring interval length that is likely to be effective (Bruhn et al., 2022). Further investigation into the selection and differential effects of monitoring interval durations is warranted.
Cost-Effectiveness Analysis
A comparison of intervention effects to the total time teachers spent implementing the intervention provides a practical indicator of feasibility for teachers. All exemplars fell in the northeast (i.e., high cost and high effect) CEP quadrant for both student and teacher time, though, a note of caution is required when considering the “high” cost of this intervention. High cost is considered when any time expense is expected; as such, a comparison of time cost to likely time available estimates provides some context to these high costs. For student time, all exemplars indicated the time cost of using I-Connect was below the time a student is likely to be receiving non-academic instruction. However, when comparing teacher total time with intervention effects, the time spent implementing the intervention was clearly differentiated by intervention form suggesting implementation of I-Connect+SR would likely require more time than is estimated to be available. In addition, intervention effect estimates were not differentiated by available time suggesting caution is required before broad interpretation of these results. Students were not randomly assigned to primary studies, limiting the comparison of the effects produced by I-Connect SR+ to I-Connect.
The CEAC provides a secondary layer of feasibility information by providing a probability threshold for intervention agents to compare the probability of effect with time availability. The CEAC indicated that teachers with less than 40 min of time to spend planning, preparing, and training the student and students with less than 25 min to learn how to implement the intervention are risking the chance that the intervention may not result in a desired effect. This time estimate provides a valuable feasibility metric for teachers who are likely to have severely limited available planning time for intervention implementation that reduces implementation quality (Bettini et al., 2016; Vannest & Hagan-Burke, 2010; Zhang et al., 2021). Furthermore, detailed information of the time required to plan and implement a single intensive intervention for a single student provides additional consideration for teachers across their caseload. I-Connect requires an initial investment of 40 min from the teacher; however, the intervention is highly versatile, demonstrates effectiveness across student populations (Scheibel et al., 2023), and is likely effective across other outcomes categories (Briesch et al., 2019). The intervention’s versatility suggests the teacher’s initial investment of 40 min to plan and implement the intervention with a single student could yield further benefits by implementing the intervention with multiple students. This would result in a smaller startup cost for the teacher as the startup cost with subsequent students would not be included in the initial 30-min planning time, greatly reducing the initial startup cost. Intervention versatility across students is an important consideration for teachers looking to maximize the use of available time.
In addition, the cost-effectiveness analysis found that despite the additional time investment required to implement I-Connect+SR resulting effects remained consistent with the effects produced by I-Connect. This finding is consistent with findings from the previous meta-analysis of I-Connect (Scheibel et al., 2023) and the broader self-management literature which indicates SM with supplemental reinforcement is among the most popular forms of investigated self-management interventions yet produces varying effects for students receiving special education (Briesch et al., 2019). However, further interpretation of these findings requires caution as effect sizes in this study were derived from a synthesis of a relatively small evidence base with a self-selected sample of students. As such, only limited conclusions can be drawn about the differential effects of supplemental reinforcement when directly compared with the use of I-Connect in isolation. Teachers considering the use of supplemental reinforcement may find trialing use of I-Connect in isolation at first would be a better time investment, then intensifying the intervention to include reinforcement if a modest response to the intervention is observed. Additional research is needed to investigate the role of reinforcement in SM interventions and enhance practical guidance for whom supplemental reinforcement is most beneficial.
Limitations and Implications for Future Research
This study presented a novel adaptation of traditional economic evaluation methods to examine the implementation costs and comparative effects of intensive interventions. However, several limitations require acknowledgment and examination in future research. First, I-Connect is supported by implementation supports which provided detailed implementation steps, intervention components and estimated time durations, and when information was not reported in the evidence base, the intervention developer provided estimates and further detail. This information allowed the authors to apply the Ingredients Method with minor adaptations to the cost perspective and cost metric. However, it was not possible to independently verify these time estimates; as such the results should be interpreted with caution as estimates provided by intervention developers introduce the possibility of bias toward a shorter estimate. Although this is a limitation, the conclusions we draw will not be sensitive to small variations in estimates in AIT as these adjustments will not impact the estimated contribution of other ingredients to the total cost.
A second limitation is that we only considered the perspective of an intervention agent (teacher) and immediate benefiter (student) in a novel application of a cost analysis using a time cost metric, that is, a time cost analysis. Although the societal perspective would have allowed us to consider every cost regardless of who pays (e.g., district administrators, school administrators, and community agencies), this more general perspective would lessen the potential usefulness of findings for teachers who would then need to unpack the costs to select which costs apply to them. Consequently, to keep the results of this study accessible to teachers, cost was considered from their perspective and translated on the metric of most relevance to them—in this case, amount of minutes which are immediately generalizable to their local implementation context. The novelty of this methodology presented an additional limitation, as to our knowledge, this is the first economic evaluation study of an intensive intervention. As a result, there is a lack of available comparison studies to properly contextualize time expense findings beyond the amount time a teacher may have available. Further examination of intensive intervention cost is needed to better understand the time and resources consumed by these interventions. As these methods become more established in SCD research, it is anticipated the methods could easily be included in research syntheses to evaluate intervention effect and feasibility.
A third limitation and direction for future research can be identified in the reliance of these methods on translational reporting in primary efficacy studies and research syntheses. The absence of primary study reporting requires economic evaluators to make estimates from available information and introduces an element of subjectivity. The paucity of translational research practices (e.g., use of endogenous implementors) and inclusion of implementation details (e.g., intervention dosages, intervention package details, and preparation activities) has been recognized in intensive intervention primary efficacy studies and research syntheses (Johnson et al., 2018; Ledford et al., 2023). The potential utility of economic evaluation methods for intensive interventions underscores previous calls for detailed reporting of translational implementation details to better facilitate translation of research into practice (Johnson et al., 2018; Ledford et al., 2021). To better facilitate economic evaluations of intensive interventions, the following details should be reported: implementation steps and intervention components, materials (differentiated from research-specific materials), implementation steps time estimates, and location and accessibility of implementation supports.
Conclusion
This study provides an example of an intensive intervention economic evaluation to support intervention selection decisions made by teachers using a novel adaptation of economic evaluation methods to intensive interventions investigated in the context of SCD research methodology. The intervention, I-Connect, was found to require an initial startup time investment of 40 min from the teacher and 25 min from the student before a substantial effect is likely to occur. The findings suggest that economic evaluation, adapted for use with intensive interventions and SCD research methods, may provide a useful framework and measure of feasibility to better inform teacher decision-making. The economic evaluation adaptations and novel methods presented appear to demonstrate initial utility to examine implementation costs of intensive interventions investigated in the context of SCD research methodology; however, more research is needed to understand the boundaries of their use.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
