The Assessment of Consistency in Single-Case Experiments: Beyond A-B-A-B Designs

Consistency in SCEDs: Definition, History, and Relevance

Following the widely accepted What Works Clearinghouse guidelines for the conduct and analysis of SCEDs

‘Consistency of data in similar phases’ involves looking at data from all phases within the same condition (e.g., all ‘baseline’ phases; all ‘peer-tutoring’ phases) and examining the extent to which there is consistency in the data patterns from phases with the same conditions. The greater the consistency, the more likely the data represent a causal relation (Kratochwill et al., 2010, p. 18).

Contrary to the other five data aspects, consistency between similar phases is thus assessed comparing data patterns from phases implementing the same experimental conditions. While the definition by Kratochwill et al. can probably be traced back to the guidelines for visual analysis of SCEDs offered by Horner et al. (2005), the origins of consistency as a separate data aspect date back to Baer (1977) and Parsonson and Baer (1978) who stress the importance of assessing the congruity of experimentally similar phases. It was around this time—in the 1970s and 1980s—that researchers started compiling comprehensive overviews of SCE designs and analysis (Onghena, Tanious, De, & Michiels, 2019). Consistency is thus one of the oldest data aspects recommended for analysis of SCEDs. Furthermore, as the definition by Kratochwill et al. highlights, greater consistency between experimentally similar phases is an important moderator of a causal relationship between the manipulation of the independent variable and the scores on the dependent variable.

A few years after Parsonson and Baer (1978), Kazdin (1982) was among the first researchers to discuss a second type of consistency in SCED data: the consistency of the effects. Contrary to the consistency between experimentally similar phases, consistency of the effects is assessed for each potential demonstration of an effect:

Consistency refers to [. . .] the extent to which changes (in level, trend, or variability) are the same for each potential demonstration of effect. In SCD research, the critical factor in determining a functional relation is the consistency of behaviour change between conditions; consistent but small changes in level between conditions are superior to inconsistent changes of larger magnitude. (Ledford, Lane, & Severini, 2018, pp. 6-7)

This second type of consistency can increase our confidence in the existence of a causal relationship just as much as the consistency across experimentally similar conditions. As Ledford et al. (2018) highlight, consistent but small changes are more indicative of a successful intervention than large changes that cannot be replicated.

Distinguishing Two Types of Consistency for Different SCE Designs

It is important to clearly distinguish between the two types of consistency. The consistency of data patterns is not necessarily an aim of an experimental intervention. It is rather an indication that the introduction of the intervention (and possibly the withdrawal depending on the design) led to a similar pattern of responding. It should be highlighted that a similar pattern of responding is conceptually different from low variability. It can, for example, happen that a subject has highly variable scores on a dependent variable in the first intervention phase (e.g., 3,15,8,21). If these scores are replicated in the second intervention phase, the consistency of data patterns across phases representing the same condition is very high (indicative of a similar pattern of responding) even if the variability in each individual phase is high as well. Variability can be assessed within phases and changes in variability can be compared across phases, whereas the consistency of data patterns is, per definition, assessed between phases. Contrary to the consistency of data patterns, the consistency of the effects is a direct result of the intervention. For example, the intervention could lead to a consistent change in mean scores between adjacent phases each time the intervention is introduced. Moreover, note that a difference in variability (e.g., a reduction) across phases belonging to different conditions could be an aim for the intervention, whereas the consistency of data patterns from the same condition is not.

Furthermore, it is noteworthy that both types of consistency are conceptually different for different SCE designs. While in phase designs and changing criterion designs intervention effectiveness is demonstrated within participants, in multiple baseline designs intervention effectiveness is demonstrated both within and between participants, behaviors, or settings, which Hayes (1981) refers to as the combined-series strategy. For multiple baseline designs, the consistency of data patterns is compared between baseline phases and between intervention phases for different participants, behaviors, or settings. If in a multiple baseline across participants design the participants differ in their initial baseline measures, then the consistency of experimentally similar phases will be low as well. In a situation like that, the lack of consistency does not necessarily entail a lack of experimental control. Given that all participants started with a different baseline level, it might therefore be expected that the consistency of the data patterns remains low for the experimental phases. Similarly, in multiple baseline designs it is less likely that the intervention produces a consistent effect for all participants, behaviors, or settings. However, if a consistent effect is produced for participants starting with different initial baseline levels, this would entail different intervention phase levels (i.e., lack of consistency of data patterns within the intervention condition). In phase designs and changing criterion designs however, where intervention effectiveness is demonstrated within participants, a lack of consistency of data patterns can be more indicative of a lack of experimental control. The exception is phase designs in which a return to initial baseline levels is not possible during subsequent baseline phases (e.g., because the change in the dependent variable is irreversible). A final remark on the distinction between the two types of consistency concerns only the changing criterion design. It might be argued that, by definition, the two types of consistency are conceptually the same for the changing criterion design. The consistency of data patterns in the changing criterion design can be understood in terms of the extent to which the measurements align consistently with the given criterion in each phase. For the changing criterion design, this is conceptually the same as the consistency of the effect. The present paper explicitly deals with methods for assessing the consistency of data patterns only. In the next section, we present an existing proposal for assessing the consistency of data patterns in A-B-A-B designs.

Analyzing the Consistency of Data Patterns in SCEDs

Historically, visual analysis has been the dominant mode of analysis for SCEDs. Probably the earliest systematic review of SCED analytical techniques employed by applied researchers was conducted by Kratochwill and Brody (1978). Kratochwill and Brody reviewed four leading behavior modification journals for SCED studies published between 1963 and 1974 and found that the proportion of studies employing statistical inference ranged from 4% to 9%. Given these findings, Kratochwill and Brody concluded that “there is a need for more attention to the technology of graphing. In a science that depends so heavily on visual analysis [. . .] and its related statistical properties, it is necessary to derive some major guidelines in this area” (p. 302). At the same time, Kratochwill and Brody issued a warning that sole reliance on visual analysis may lead to erroneous conclusions as evidence about the flaws of visual analysis and need for statistical inference criteria started accumulating in the 1970s (DeProspero & Cohen, 1979; Gentile, Roden, & Klein, 1972; Jones, Vaught, & Weinrott, 1977; Jones, Weinrott, & Vaught, 1978). In this context, the earliest systematic guidelines for visual analysis of SCEDs were proposed by Parsonson and Baer (1978), which included, as previously mentioned, the first type of the data aspect consistency. It is this type of consistecy, the consistency of data patterns in experimentally similar phases, that we focus on in the present article. For existing proposals on the consistency of the effects, the interested reader is referred to Manolov (2018) for multiple baseline designs and Tanious, De, Michiels, Van den Noortgate, and Onghena (2019a) for A-B-A-B phase designs.

Regarding the consistency of data patterns in experimentally similar phases, only one quantification has been proposed so far. Tanious et al. (2019a) developed the CONsistency of DAta Patterns (CONDAP) measure for quantifying the degree of consistency between data patterns in A-B-A-B phase designs based on the Manhattan distance. The basic premise of CONDAP is that if two data patterns are highly consistent, then the standardized average Manhattan distance should be accordingly low. Therefore, lower CONDAP values indicate higher consistency and higher CONDAP values indicate lower consistency. For A-B-A-B phase designs, CONDAP can be calculated as shown in Equation 1.

C O N D A P = 1 k n s ∑ j = 1 k ∑ i = 1 n s | s i j − l i j | ( n s − 1 ) * S D s 2 + ( n l − 1 ) * S D l 2 n s + n l − 2

(1)

The numerator in Equation 1 is the mean Manhattan distance across all comparisons of k sequences of equal length n for the two phases being compared. In an A-B-A-B design, for each condition, there are two experimentally similar phases. If both phases have the same number of data points, the number of compared sequences is always equal to one and only one comparison of the two phases is necessary. If the two phases differ in lengths, the longer phase is shortened to a sequence that is of equal lengths to the shorter phase. The shorter phase is then compared to each sequence of the longer phase that is equal to the lengths of the shorter phase. S_ij is the ith data point for the jth sequence from the shorter phase and l_ij is the ith data point for the jth sequence from the longer phase. Figure 1 visualizes the calculation of CONDAP for an A-B-A-B design where the B1 phase has five measurements (8,6,7,7,6) and the B2 phase has three measurements (7,5,9). The CONDAP for Figure 1 can then be calculated as follows. First, the sum of the vertical dashed lines (the Manhattan distances) for each panel is calculated and divided by three (which is the length of the shorter phase) to obtain the average Manhattan distance per comparison of equal length of the two data patterns. Between each panel, the shorter phase is shifted by one measurement occasion to the right. Second, the three average Manhattan distances are then summed up and divided by the number of comparisons, in this case three. Finally, this overall average Manhattan distance is then divided by the pooled standard deviation of the two phases as proposed by Van den Noortgate and Onghena (2008) to obtain the scale invariant CONDAP (whose value would be 1.16 for the current example). If all experimentally similar phases have a standard deviation of zero, then the denominator of CONDAP would be zero as well, and it is recommended to calculate the overall average Manhattan distance without standardization (Tanious et al., 2019a)

OPEN IN VIEWER

Figure 1.

CONDAP calculation for an A-B-A-B phase design with unequal phase lengths.

Based on a systematic review of a sample of 119 applied A-B-A-B studies, Tanious, De, Michiels, van den Noortgate, and Onghena (2019b) proposed the following guidelines for interpreting CONDAP: very high, 0 ≤ CONDAP ≤ 0.5; high, 0.5 < CONDAP ≤ 1; medium, 1 < CONDAP < 1.5; low, 1.5 < CONDAP ≤ 2; very low, CONDAP > 2. The two data patterns in Figure 1 are thus medium consistent. It is important to note that these guidelines were developed specifically for designs using successive A-B comparisons. We will use these guidelines throughout the article for other designs for illustrative purposes and comparability, but they should be interpreted with caution.

Consistency of Data Patterns: When Should it Be Assessed?

It is noteworthy that the What Works Clearinghouse panel uses an A-B-A-B phase design as an example for demonstrating their developed guidelines. The immediate question following from that is whether consistency is only desirable in A-B-A-B designs. The definition by Kratochwill et al. refers to “consistency of data in similar phases”. From this statement, it can be deduced that each manipulation of the independent variable must occur at least twice in order to assess consistency. By definition, this excludes phase designs such as A-B, A-B-C, A-B-C-D, and A-B-BC-C in which each unique manipulation of the independent variable is only introduced once. In addition, there are phase designs in which only one of the manipulations of the independent variable is introduced twice. Examples of such designs are A-B-A, B-A-B, A-B-C-A, and A-B-C-B. In such designs, the consistency of data patterns can only be selectively assessed for the experimental condition that was introduced twice. Furthermore, even in phase designs where each manipulation of the independent variable occurs at least twice (e.g., A-B-A-B), consistency cannot always be assessed, in relation to the number of measurements available. In line with quality standards for SCEDs, each phase should contain a minimum of three data points to meet minimum evidence standards (e.g., Beeson & Robey, 2006; Kratochwill et al., 2010) and five data points per phase are required to meet evidence standards without reservation (Ganz & Ayres, 2018; Horner et al., 2005; Kratochwill et al., 2010, 2013; Tate et al., 2016; U.S. Department of Education, Institute of Education Sciences, 2016). Additionally, consistency is assessed by comparing data patterns that emerge over time making it implausible to quantify consistency for very short time-series. For these reasons, we recommend against quantifying consistency of data patterns with less than three data points per phase. By the same logic, it is not plausible to assess the consistency of data patterns for one particular kind of SCEDs: the alternating treatments design. This design utilizes rapid and repeated manipulations of two or more conditions (Wolery, Gast, & Ledford, 2018) and “the crucial factor of the design is the unique intervention phase in which two separate interventions are administered concurrently” (Kazdin, 2011, p. 198). Due to the rapid alternation of treatments, there are no distinguishable phases in this design so that no consistent data patterns across experimentally similar phases can emerge. In the following sections, we extend the existing CONDAP measure for designs in which consistency of data patterns is desirable: multiple baseline designs, phase designs with more than two consecutive A-B comparisons, and changing criterion designs.

Extensions of CONDAP for more than four Phases and Multiple Baseline Designs

The basic A-B-A-B phase design can be extended by as many phases as the researcher believes are necessary and feasible. One such extension is the A-B-A-B-A-B design in which one more baseline and one more experimental phase are added to the A-B-A-B structure. The advantages of adding one more A-B pattern include the following: repeated possibilities for demonstrating experimental control over the dependent variable, extended study until full clinical treatment has been achieved, and added flexibility (Barlow, Nock, & Hersen, 2009). Figure 2 shows an example of an applied study from the autism literature using this design. Angell, Nicholson, Watts, and Blum (2011) examined the effectiveness of a multicomponent adapted Power Card strategy to decrease latency during interactivity transitions for three children with developmental disabilities. The dependent variable was interactivity transitions that occurred within the classroom. This was measured as the latency of students with developmental disabilities in response to teacher cues to initiate classroom interactivity transitions. The baseline phases included typical classroom conditions. During intervention phases, several components were added to the students’ routines which included presentation of the Power Card, verbal cues by the teacher, and verbal praise by the teacher. Figure 2 shows the data of Quincy, an 11-year-old male diagnosed with autism.

OPEN IN VIEWER

Figure 2.

Example of an A-B-A-B-A-B design. Data from Angell et al. (2011).

As the definition by Kratochwill et al. (2010) suggests, consistency of similar data patterns is assessed separately for each manipulation of the independent variable. In the example in Figure 2, we can thus assess the consistency of data patterns for the three baseline phases and the three intervention phases separately. This can be achieved by adding one additional step to the CONDAP formula for A-B-A-B designs. Whereas in an A-B-A-B design, only one comparison is needed between the A₁ and A₂ phase on the one hand and the B₁ and B₂ phase on the other, an A-B-A-B-A-B requires additional comparisons to incorporate the A₃ and B₃ phases. To achieve this, we can first calculate separate CONDAP values for each comparison between A₁ and A₂, between A₁ and A₃, and between A₂ and A₃. The consistency matrix below shows the results of this first step.

The diagonal values of the matrix equal zero because each phase is perfectly consistent with itself (i.e., identical data patterns) as indicated by a CONDAP value of zero. Subsequently, an average across all three comparisons can be calculated to obtain the overall consistency of baseline data patterns: (1.06 + 2.37 + 1.65)/3 = 1.69. The baseline data patterns thus show low consistency using the interpretative guidelines developed for A-B-A-B designs. Following the same steps for the B-phases leads to obtaining a CONDAP value of 1.45 indicative of medium consistency. Beyond the application of the interpretative benchmarks, the consistency is lower for the baseline data patterns and this agrees well with the visual analysis of the data. Visual inspection of Figure 2 reveals that the A-phases are all highly variable and show differences in level whereas the B-phases are less variable and differences in level are smaller than between A-phases. The calculated CONDAP values express these differences in terms of higher consistency between experimental phases than baseline phases.

OPEN IN VIEWER

	A 1	A 2	A 3
A 1	0	1.06	2.37
A 2		0	1.65
A 3			0

For assessing the consistency in multiple baseline designs, the same steps can be followed. Multiple baseline designs across subjects can be defined as follows.

In the multiple baseline design across subjects, each individual targeted for treatment is exposed to the same environment. Treatment is delayed for each successive subject in time-lagged fashion because of the increased length of baselines required for each. The functional relationship between treatment and target behavior can be determined only when such treatment is applied to each subject in succession. (Barlow et al., 2009, p. 234)

If the goal is to change several distinct target behaviors in a single participant, a multiple baseline across behaviors design can be used (Rvachew, 1988). In such a design, the intervention is introduced in a time-lagged fashion to different target behaviors to demonstrate experimental control of the intervention over each target behavior. Figure 3 shows an example of a multiple baseline across subjects design. Scheeler, Morano, and Lee (2018) used this design to investigate the effectiveness of a training program for four paraeducators working with students diagnosed with Autism spectrum disorder. During baseline sessions, the paraeducators received only delayed feedback from a special education classroom teacher. During intervention sessions, the paraeducators received immediate feedback from the special education classroom teacher via bug-in-ear technology. The dependent variable displayed in Figure 3 is percentage of contingent specific praise statements delivered by the paraeducator to the students reported as the percentage of total praise statements.

OPEN IN VIEWER

Figure 3.

Example of a multiple baseline across participants design. Data from Scheeler et al. (2018).

To calculate the consistency across data patterns of experimentally similar phases, we can follow the same logic as in the A-B-A-B-A-B example. The only difference is that in a multiple baseline across subjects design, we compare experimentally similar phases across subjects instead of within a subject. Given that Scheeler et al. (2018) worked with four participants, we need to add one more row and column to the consistency matrix for A-B-A-B-A-B designs to incorporate the additional comparisons. This also holds true for designs consisting of adjacent A-B comparisons with more than six phases. The consistency matrix below shows the results of all consistency comparisons for the intervention phases of the Scheeler et al. data.

In a second step, which is identical to the consistency assessment for A-B-A-B-A-B designs, we can calculate the average across all comparisons: (1.33 + 1.02 + 1.28 + 1.15 + 1.67 + 1.20)/6 = 1.27. The consistency is higher than, for example, in the data patterns of the Angell et al. data set. Following the same steps for the baseline phases, leads to obtaining a CONDAP value of 1.17. In general, the number of consistency comparisons c per experimentally similar phase—the upper right triangle in the consistency matrix—for A-B-A-B-A-B and multiple baseline designs equals:

c = ( n p 2 − n p ) 2

(2)

In Equation 2, n p represents the number of experimentally similar phases. For example, the Scheeler et al. dataset contained four phases for each intervention. The number of consistency comparisons c for each experimentally similar phase thus equals ( 4 2 − 4 ) 2 = 6 .

OPEN IN VIEWER

	B 1	B 2	B 3	B 4
B 1	0	1.33	1.02	1.28
B 2		0	1.15	1.67
B 3			0	1.20
B 4				0

Extensions of CONDAP for Changing Criterion Designs

The changing criterion design has been introduced as a distinct design in a seminal paper by Hartmann and Hall (1976). As the name suggests, in this design, an individual is subjected to changing criteria (goals) for the rate of the target behavior. Barker, McCarthy, Jones, and Moran (2011) describe the procedures for this design as follows:

A criterion is set that represents a target (goal) for the participant to meet. This criterion (or goal) will change throughout the course of the study. It is anticipated that the variable improves in increments to match the criterion that is specified as part of the intervention (Kazdin, 1982). Normally, rewards or incentives are provided to facilitate the attainment of a designated criterion [. . .] In the changing criterion design, the required level of a target variable is altered repeatedly (e.g., increasing the amount of daily exercise time) to improve performance of this variable over time. (p. 109)

The changing criterion design is especially useful when immediate large changes in target behavior are either impossible or undesirable because in this design the researcher can apply gradual shifts toward a desired goal (Klein, Houlihan, Vincent, & Panahon, 2017). When the dependent variable in a changing criterion design changes according to the predetermined criterion levels, an intervention effect is demonstrated (Kinugasa, Cerin, & Hopper, 2004). Figure 4 shows an example of an applied changing criterion design. Voulgarakis and Forte (2015) used this design to examine the effectiveness of an escape extinction and negative reinforcement-based approach to treating food refusal in a child with cerebral palsy. The dependent variable displayed in Figure 4 is the number of instances of the child depositing a bite of food into his mouth during a 30-min period. During the first phase, baseline measures were collected with no intervention taking place. The criterions for the subsequent intervention phases were 5, 7, 10, 7, and 12 bites. During intervention phases, the child received positive verbal reinforcement and was allowed to exit the meal as well as meal area upon reaching the criterion. The upper left panel of Figure 4 shows the raw data.

OPEN IN VIEWER

Figure 4.

Example of a changing criterion design. Red horizontal lines indicate the criterion for each phase. Upper left panel: raw data; upper right panel: difference between each score and the corresponding (vertical dashed lines); lower left panel: converted difference scores from upper right panel. Data from Voulgarakis and Forte (2015).

In such a design, the assessment of consistency is highly desirable. If the rates of target behavior displayed by the participant are consistent with the criteria set by the researcher, then our confidence in the existence of a causal relationship increases.

However, since the criterion changes in each consecutive phase, the assessment of consistency for this design differs from the assessment of consistency in the previously discussed phase designs and multiple baseline designs for two reasons. First, there is only one baseline phase in the beginning of the study in which no intervention is present. After the initial baseline phase, no withdrawal of the intervention takes place (Barlow et al., 2009). The assessment of consistency of baseline phases is thus neither possible nor desirable for the changing criterion design. Second, in a changing criterion design the criterion rate for the target behavior set by the researcher changes for each consecutive phase. Therefore, the data patterns themselves are expected to be inconsistent if the intervention is successful and the rate of target behavior changes with each criterion change. Accordingly, we recommend inspecting the consistency of data patterns in relation to the criterion in each phase rather than the consistency of the raw data. Moreover, unlike A-B-A-B, A-B-A-B-A-B, and multiple-baseline designs, the assessment of consistency in a changing criterion design would not require distinguishing between consistency of data in similar phases (which is the focus of the current text) and consistency of effects. Actually, for a changing criterion design, there would be only one kind of consistency: the degree to which the measurements obtained match the criterion levels set by the researcher, in the different subphases of the intervention phase. In that sense, the assessment of consistency in a changing criterion design, as described here, is even more important for inferring a causal relation between the reinforcer and the target behavior, because it does not need to be complemented with a second type of consistency.

One measure of intervention effectiveness for changing criterion designs that is sensitive to the differences between the scores and the criterion is the Mean Absolute Difference which “is calculated by taking the difference between the scores and the criterion at each measurement occasion, dropping the sign, summing all these absolute differences, and dividing by the total number of scores” (Onghena et al., 2019, p. 4). Following the logic of the Mean Absolute Deviation, we propose to use the difference between the scores and the criterion in each phase for assessing the consistency of data patterns with a slight modification. Instead of taking the absolute difference, we take the difference between each score and the criterion as is in order to preserve the original data patterns. The added vertical dashed lines in the upper right panel of Figure 4 show these differences. For example, for the first experimental subphase the difference between the scores and the criterion of five are 0, 1, and 0. Converting each score to a difference score in this way gives the data shown in the lower left panel of Figure 4 whereby the baseline phase is dropped because no intervention or criterion was present in that phase and no assessment of consistency is possible. Subsequently, the difference scores can be used to assess the consistency between all pairs of subphases in the same way as for A-B-A-B-A-B and multiple baseline designs. For the scores depicted in Figure 4 we obtain the following consistency matrix where the numbers indicate the order in which the subphases occurred during the experiment.

Taking the average across all comparisons gives a CONDAP of 1.13 indicative of medium consistency when applying the benchmarks developed for A-B-A-B designs.

OPEN IN VIEWER

	1	2	3	4	5
1	0	1.15	0.58	0	1.63
2		0	1.73	1.15	1.63
3			0	0.58	1.22
4				0	1.63
5					0

In general, the number of consistency comparisons c for changing criterion designs equals:

c = ( n p − 1 ) 2 − n p 2

(3)

For changing criterions designs, the baseline phases needs to be subtracted from the overall number of phases before squaring the number of phases.