Linear Trend in Single-Case Visual and Quantitative Analyses

Abstract

The frequently used visual analysis of single-case data focuses on data aspects such as level, trend, variability, overlap, immediacy of effect, and consistency of data patterns; most of these aspects are also commonly quantified besides inspecting them visually. The present text focuses on trend, because even linear trend can be operatively defined in several different ways, while there are also different approaches for controlling for baseline trend. We recommend using a quantitative criterion for choosing a trend line fitting technique and comparing baseline and intervention slopes, instead of detrending. We implement our proposal in a free web-based application created specifically for following the What Works Clearinghouse Standards recommendations for visual analysis. This application is especially destined to applied researchers and provides graphical representation of the data, visual aids, and quantifications of the difference between phases in terms of level, trend, and overlap, as well as two quantifications of the immediate effect. An evaluation of the consistency of effects across replications of the AB sequence is also provided. For methodologists and statisticians, we include formulas and examples of the different straight line fitting and detrending techniques to improve the reproducibility of results and simulations.

Keywords

single-case designs trend visual analysis software

Single-case designs (SCD) data are still commonly analyzed visually (Campbell & Herzinger, 2010; Kratochwill, Levin, Horner, & Swoboda, 2014; Parker, Cryer, & Byrns, 2006; Smith, 2012), either on exclusive basis or jointly with an objective quantification. When describing SCD visual analysis, the What Works Clearinghouse (WWC) technical report (Kratochwill et al., 2010) mentions six aspects of the data: level, trend, overlap, variability, immediacy of effect, and consistency of data patterns across similar conditions. These data aspects can be assessed purely visually, but they are also frequently quantified. Specifically, level is usually represented by the mean and the median (Fisher, Kelley, & Lomas, 2003; Lane & Gast, 2014); trend is defined as the “best fitting straight line” (Kratochwill et al., 2010); variability can be represented using standard deviation bands (Fisher et al., 2003; Pfadt & Wheeler, 1995), range and the stability envelope (Lane & Gast, 2014); and overlap is commonly assessed in visual analysis via the percentage of nonoverlapping data (PND; Lane & Gast, 2014). PND is also frequently used as main quantification of effect (Schlosser, Lee, & Wendt, 2008; Scruggs & Mastropieri, 2013), despite its limitations (Allison & Gorman, 1994; Campbell, 2013), and despite the number of nonoverlap alternatives that have already been discussed extensively elsewhere (Parker, Vannest, & Davis, 2011; Wolery, Busick, Reichow, & Barton, 2010). The immediacy of the effect can be assessed focusing on the level of last three baseline measurements versus the first three intervention phase measurements (Kratochwill et al., 2010). Regarding the final data aspect, the WWC Standards define the consistency of similar phases as the “extent to which the data patterns in phases with the same (or similar) procedures are similar” (Kratochwill et al., 2010, p. 20). We refer to this topic in a later section titled “Consistency of Effects and Type of SCD.”

The Importance of Trend

The focus of the current text is on trend, due the evidence of the importance of trend lines for improving the performance of visual analysts (Bailey, 1984; Skiba, Deno, Marston, & Casey, 1989) and given that neither the WWC Standards nor Horner, Swaminathan, Sugai, and Smolkowski (2012) mention a specific technique for fitting trend. Trend has been the focus of attention in relation to both visual analysis (Mercer & Sterling, 2012) and statistical analytical techniques (e.g., Parker et al., 2006; Solomon, 2014), but even “linear trend” is an ambiguous term, unless clearly defined. Actually, to the best of our knowledge, there has been no broad review and no systematic comparison performed on the different ways in which a straight line can be fitted to the single-case data. To underline the importance of trend, we can relate it to the other data aspects mentioned in the WWC Standards. First, a trend line is a less restrictive visual representation of the data than a median line, which imposes lack of change in level with time, or stationarity. Second, the trend line can be used as a basis of the visual representation and the quantification of variability (i.e., if variability is defined as the amount of variation around the trend line). Third, as will be reviewed in depth in Online Appendix A, some nonoverlap indices take trend into account to provide a more meaningful comparison between conditions. Finally, the immediate effect of the intervention can also be conceptualized and quantified on the basis of the projection from the baseline (Horner et al., 2012), for instance, as the difference between the predictions for the first intervention phase value from the baseline and the intervention phase trend lines, as in piecewise regression (Center, Skiba, & Casey, 1985-1986). Thus, we consider that trend is crucial data aspect that requires thorough discussion.

Consistency of Effects and Type of SCD

The consistency of effects can be assessed in relation to the type of SCD used. Specifically, although comparing level, trend, and variability and assessing overlap can be done separately for each AB-pair of phases, an AB design is not sufficiently rigorous from a methodological perspective. Actually, the WWC Standards (Kratochwill et al., 2010) require a minimum of three replications of the effect that can be observed in an AB-pair. However, not all replications are equivalent. For instance, focusing on two recently compared designs (Novotny et al., 2014), a nonconcurrent multiple-baseline design would entail three AB-comparisons, whereas an ABAB design entails comparing A₁-B₁, B₁-A₂, and A₂-B₂. Therefore, what is different is that in an ABAB design, (a) there is a single participant and potentially more consistency can be expected; (b) one of the comparisons is in the BA order, which might affect the visual impression of the results; (c) the information from two of the phases is used twice, which could lead to apparently greater consistency; and (d) in some cases, A₂ may not be considered comparable with A₁, because the effect of the intervention may not wash out completely, which could lead to lower consistency. It is even more relevant to distinguish between a concurrent and a nonconcurrent multiple-baseline design across participants (Carr, 2005), given that for the former the staggered introduction of the interventions is crucial for assessing experimental control via within-series and between-series comparisons (Ferron, Moeyaert, Van Den Noortgate, & Beretvas, 2014; Horner et al., 2005). Therefore, even if there is similar consistency in AB-comparisons arising from these two designs, the evidence from a concurrent multiple-baseline design is stronger. In summary, it has to be underlined that in absence of replication and consistency of the effect, a difference observed in an AB-comparison (in terms of level, trend, variability or overlap) is insufficient. Moreover, the assessment of consistency across AB-comparisons has to take place considering the type of SCD actually used.

Aims of the Article, Justification, Intended Audience, and Organization

The article aims to provide an answer to several questions. As presented in Table 1, we consider that some of these questions—How to choose a technique for fitting a straight line? Is it necessary to always fit a (straight) line? Is it necessary to always control for baseline trend? How to follow the WWC Standards recommendations for visual analysis?—might be of greater interest to applied researchers. The ultimate aim is that applied researchers have a tool enabling them to perform visual analysis, focusing on each of the six data aspects mentioned in the WWC Standards in a systematic way. For that reason, we have included our comments and recommendations on these questions in the main text.

Table 1.

Questions Aimed to Be Answered, Intended Audience, and Organization of the Text.

Question	Intended audience	Content availability	Approach to answering the question	Summary for applied researchers
What trend line fitting techniques exist and how do they differ?	Methodologists/statisticians	Main text (verbal explanations) and Online Appendix A (formulas and examples)	Review of trend line fitting techniques incorporated in single-case data analysis procedures	There are several options that do not necessarily lead to the same result
How to choose a technique for fitting a straight line?	Applied researchers; Methodologists/statisticians	Main text (verbal explanations) and Online Appendix B (formulas and examples)	Search in the literature for a quantification of fit of a trend line to the data	Use the mean absolute scaled error criterion
Is it necessary to always fit a (straight) line?	Applied researchers	Main text	Review of alternatives and a recommendation	No, but it is easier, more parsimonious, more common
What ways of controlling for baseline trend have been incorporated in single-case data analytical procedures?	Methodologists/statisticians	Main text (verbal explanations) and Online Appendix C (formulas and examples)	Review of existing approaches to controlling for baseline trend	There are several options that do not necessarily lead to the same result
Is it necessary to always control for baseline trend?	Applied researchers	Main text	Discussion and a recommendation	Baseline trend control is not the same as detrending; separate trend lines can be fit
How to follow the WWC Standards recommendations for visual analysis?	Applied researchers	Main text	Creation of a web application	Use the web application created

Note. WWC = What Works Clearinghouse.

We also aim to provide answers to questions we might be of greater interest to methodologists and statisticians (What trend line fitting techniques exist and how do they differ? What ways of controlling for baseline trend have been incorporated in single-case data analytical procedures?). Thus, we provide formulas and examples on the existing options for fitting trend lines and for detrending in Online Appendices (https://osf.io/jb9dm/). We consider that a single document containing this information is relevant and necessary, because otherwise this information is either missing (for Tau-U and Baseline corrected Tau) or scattered in several documents, including difficult to access sources. For applied researchers not interested in the details, the brief verbal comments in the main text and the answers provided in the last column of Table 1 would be sufficient.

What Trend Line Fitting Techniques Exist and How Do They Differ?

The current overview is focused on techniques that have been incorporated in analytical procedures proposed for single-case data analysis. Thus, several trend estimation techniques are not included, such as the minimum m-estimation (Anderson & Schumacker, 2003; Yohai, 1987) tested by Brossart, Parker, and Castillo (2011) but not included in a data analytical procedure, or the least median of squares (see Wilcox, 2012, Chapter 10). The current review is also focused on linear trend here rather than on nonlinear trend (commented more extensively in the section titled “Is It Necessary to Always Fit a (Straight) Line?”), because it is simpler, potentially more easily understood (Chatfield, 2000), and also more frequently present in visual aids (Fisher et al., 2003; Lane & Gast, 2014). Thus, we prioritize the accessibility of the techniques (Parker, Vannest, & Davis, 2014). Moreover, in the WWC Standards, linear trends are the focus of attention, according to the illustrations provided in Kratochwill et al. (2010).

Apart from stating that there are many more trend line fitting techniques than the linear trend line fitting techniques incorporated in single-case data analytical procedures,¹ the answer to first part of the question is provided in Table 2. As seen in Table 2, least squares estimation is the technique most frequently incorporated in single-case data analytical procedures. Moreover, some analytical procedures model only baseline trend (for detrending or for comparing with the intervention phase data), whereas the majority of procedures using least squares estimation fit separate trend lines to the baseline and the intervention phases.

Table 2.

Cross Tabulation of Straight Line Fitting Techniques (Columns) and the Portion of the Data to Which the Trend Line Is Fitted (Rows).

Where trend is modeled	Least squares estimation	Bi-split (split-middle method)	Tri-split	Theil–Sen estimator	Differencing
Baseline only	Allison and Gorman (1993) ordinary least squares regression model	Conservative Dual Criterion (Fisher, Kelley, & Lomas, 2003); Percentage of data points exceeding median trend (Wolery, Busick, Reichow, & Barton, 2010)	Graph rotation procedure (Parker, Vannest, & Davis, 2014)	Baseline corrected Tau (Tarlow, 2017)	Mean Phase Difference (Manolov & Rochat, 2015; Manolov & Solanas, 2013)
Baseline and intervention phase separately	Last Treatment Day procedure (White, Rusch, Kazdin, & Hartmann, 1989); Piecewise regression (Center, Skiba, & Casey, 1985-1986); Mean and Slope Adjustment (MASAJ) mean plus trend shift (Parker, Cryer, & Byrns, 2006); Generalized least squares regression proposal by Swaminathan, Rogers, Horner, Sugai, and Smolkowski (2014); Standardized mean difference based on multilevel modeling of level and trend (Pustejovsky, Hedges, & Shadish, 2014)	Visual analysis (Lane & Gast, 2014; Miller, 1985)	None	None	Slope and Level Change (Solanas, Manolov, & Onghena, 2010)
Baseline and intervention phase jointly	Gorsuch’s (1983) trend analysis	None	None	None	None

Regarding the second part of the question (“How do the trend line fitting techniques differ?”), from an applied perspective, it is sufficient to say that greater differences between the trend lines fitted are expected when the data conform worse to a straight line (as is the case in the A phase in Figure 1). Complementarily, smaller differences are expected when the data are more easily approximated by a straight line (as is the case in the B phase in Figure 1).

Figure 1.

Data (obtained by Eilers & Hayes, 2015) for Jacob (Tier 3), a participant with autism spectrum disorder, treated with cognitive defusion exercise and exposure therapy.

In brief, ordinary least squares estimation fits the trend line in such a way that the squared difference between itself and the actual values is the smallest possible. In comparison with the ordinary least squares estimation, tri-split and the Theil–Sen estimator are intended to be more resistant to (i.e., less affected by) outliers. In contrast to the previously mentioned techniques, the main advantage of the split-middle method is its simplicity and possibility to use it manually (Miller, 1985). Finally, differencing can be used for estimating trend as the average of the differences between successive data points, and not only for detrending, as it will be mentioned later. For further detail, a technical and formulaic representation of the five straight line fitting techniques is available in Online Appendix A.

How to Choose a Technique for Fitting a Straight Line?

We consider that it is important to reach the same conclusion about intervention effect when focusing on the same data feature (here, trend). However, reaching the same conclusion is not ensured in light of the variety of straight line fitting techniques summarized in Table 2. Thus, it is not desirable that an apparently clear criterion such as “linear trend” would lead to very different graphical representations (superimposed lines) and numerical results. Accordingly, in visual analysis the need for formal decision rules has been underscored (Ottenbacher, 1990). Moreover, Parsonson and Baer (1992) explicitly who mention the importance of both comparing trend line fitting techniques and having a rule for such a comparison.

A logical option for choosing between techniques is on the basis of the reliability (Parker, Vannest, Davis, & Sauber, 2011) or the amount of fit of the trend line to the data. It can logically be expected that, for the same amount of residual variability, more reliable estimates will be obtained for longer data series. The amount of fit of the straight line to the data can be quantified in several ways, the most well-known of which is R², quantifying the proportion of data variability explained by a linear. Actually, Hyndman and Koehler (2006) compare several options for comparing the actual values with the fitted values (e.g., the trend line) and they propose a measure called the Mean Absolute Scaled Error (MASE). Values of MASE greater than one indicate that the trend line provides a worse fit than predicting each value from the previous one. In general, the greater the MASE, the worse the fit of the trend line to the data. In Online Appendix B, the formulas for several measures of fit, including the MASE recommended here, are provided.

Is It Necessary to Always Fit a (Straight) Line?

A straight line not always easily or meaningfully represents data. In general, it may not be always visually clear whether (a) a straight line represents the data well, (b) a straight or a curved line is a better representation, and (c) which curved line to choose. For simple monotonic relations in which there is only one bend in the curve representing the relation between the measurement occasions and the measurements, it is possible to use Mosteller and Tukey’s bulging rule (see, for example, Fox, 2016) for transforming the data and using afterward any of the techniques for finding a best-fitting straight line. However, single-case researchers may not be willing to transform the data, as it goes against the importance of an in-depth understanding of the values obtained (Fahmie & Hanley, 2008). Without the need to transform, three options have been mentioned in the literature to model nonlinear trends: (a) polynomial regression (Swaminathan, Rogers, Horner, Sugai, & Smolkowski, 2014), considering that Horner et al. (2012) explicitly mention quadratic lines; (b) local regression (Solmi, Onghena, Salmaso, & Bulté, 2014); and (c) generalized additive models: applying cubic regression splines with many knots (e.g., 10 knots for a data series of 15 measurements; Sullivan, Shadish, & Steiner, 2015). These two latter options are also useful when a nonlinear relation presents more than one bend, unlike quadratic models.

We consider that there are three criteria for selecting the appropriate model. First, in case there is substantive knowledge on the expected evolution of the target behavior (e.g., about the type of natural recovery during baseline, about the presence of an upper or a lower asymptote once the behavior has started improving after the intervention), such knowledge should be used. Second, scientific parsimony calls for using the simplest possible model that implies only a minor loss of information (or loss of goodness of fit) with respect to more complex models. In that sense, a linear model would be warranted in absence of a justification for a specific nonlinear model. Third, quantifications such as the R² and MASE may be used for selecting a model of the data. Finally, it is also possible to use single-case analytical procedures that do not summarize the data via trend lines or mean lines, such as the Nonoverlap of all pairs (NAP; Parker & Vannest, 2009).

What Ways of Controlling for Baseline Trend Have Been Incorporated in Single-Case Data Analytical Procedures?

Just as “linear trend” does not necessarily refer to a single possible straight line representing the relation between the measurement occasions and the measurements, “detrending” (i.e., removing baseline trend from the data) could also be an ambiguous term. Table 3 provides a brief summary of the different options for detrending, whereas in Online Appendix C, we have included the formulas for each detrending technique and an example. Note that apart from using baseline trend for detrending, some analytical procedures extrapolate baseline trend for comparison purposes (e.g., conservative dual criterion; percentage of data points exceeding median trend; mean phase difference), whereas others compare the slopes of the trend lines fitted separately with the baseline and intervention phase (e.g., piecewise and generalized least squares regression, multilevel models).

Table 3.

Approaches to Detrending in Several Single-Case Data Analytical Procedures.

Detrending approach	Single-case data analytical technique	Measurement units of the detrended data
Initial baseline regression with time as predictor	Allison and Gorman (1993)	Transformed (residuals)
Initial regression with time and the interaction between time and phase as predictors	Before the between-cases standardized mean difference (Shadish, Hedges, & Pustejovsky, 2014)	Transformed (residuals)
Single regression step modeling general trend and change in level	Trend analysis (Gorsuch, 1983)	Transformed (residuals)
Removing baseline trend from the whole series	Baseline corrected Tau Slope and level change	Original
Removing baseline trend from the intervention phase data	Mean and slope adjustment (Parker, Cryer, & Byrns, 2006)	Original
Subtracting the number of improving baseline data points from the nonoverlap comparison	Tau-U (Parker, Vannest, Davis, & Sauber, 2011)	Original
Differencing	Differencing analysis (Gorsuch, 1983) and ARIMA models (Harrop & Velicer, 1985)	Transformed

Note. ARIMA = autoregressive integrated moving average.

Is It Necessary to Always Control for Baseline Trend?

Parker et al. (2006) underscore the importance of taking baseline trend into account before quantifying intervention effects. However, it needs to be highlighted that taking baseline trend into account does not necessarily imply detrending (transforming the data), because baseline trend can be compared with the intervention phase trend. Actually, we here advocate for avoiding detrending by modeling separately the trend in each phase. In the next paragraph, we present the justification for our recommendation.

A first issue with detrending is that there are several different ways in which it can be achieved, as reviewed in the section titled “What Ways of Controlling for Baseline Trend Have Been Incorporated in Single-Case Data Analytical Procedures?” Thus, it would be necessary to have a criterion for choosing one of these options. A second issue is that it may not always be easy to decide for which data sets to detrend: always, only when baseline trend that is statistically significant, or only when the baseline trend represents well the data. A similar concern has been expressed by Parker, Vannest, Davis, and Sauber (2011), who underline that the length of the baseline, and the possibility of the data correction being excessively strong have to be considered. Moreover, detrending is not flawless. The evidence suggests that in some cases baseline trend control may be insufficient, leading to excessively large estimates of effect. For instance, Tarlow (2017) reports such evidence for Tau-U and also for Baseline corrected Tau, if the statistical significance of the baseline slope is assessed in baselines with fewer than 10 measurements (Tarlow, 2017). Complementarily, in other cases baseline trend control may be excessively strong control, leading to conservative estimates of effect (e.g., see Parker & Brossart, 2003; Gorsuch, 1983, for trend analysis, and Manolov, Arnau, Solanas, & Bono, 2010, for differencing analysis). Similarly, in relation to the initial detrending before applying the between-cases standardized mean difference (BC-SMD), Shadish, Hedges, and Pustejovsky (2014) underline that detrending should be used with caution, because it may remove part of the intervention effect when it is expressed as change in slope. More favorable results have been reported for the Allison and Gorman (1993) model (Manolov & Solanas, 2008; Tarlow, 2017), for the Mean Phase Difference (Tarlow, 2017), and for the Slope and Level Change procedure (Solanas, Manolov, & Onghena, 2010). However, it is difficult to disentangle the performance of the detrending technique from the performance of the data analytical procedures in which it is incorporated.

Regarding our recommendation, the separate modeling of baseline and intervention phase trend is best aligned with the way in which visual analysis proceeds. We here suggest choosing the straight line fitting technique on the basis of the MASE. In terms of further quantifications, it is possible to compare trend lines by means of a randomization test (Michiels, Heyvaert, Meulders, & Onghena, 2017). In addition, multilevel models also proceed estimating trend lines separately. For instance, an option to quantify the BC-SMD for data potentially presenting trend is to use multilevel models and maximum likelihood estimation (Pustejovsky, Hedges, & Shadish, 2014) instead of detrending first.

How to Follow the WWC Standards Recommendations for Visual Analysis?

In the present section, we describe a web-based application available at https://manolov.shinyapps.io/Overlap/ and making easier following the WWC Standards recommendations for visual analysis and also for favoring the integration of visual and statistical analysis (Horner et al., 2012). The website also reflects the fact that most quality standards include items on both visual and statistical analysis (Heyvaert, Wendt, Van Den Noortgate, & Onghena, 2015). In addition, we have included the conservative dual criterion (Fisher, Kelley, & Lomas, 2003), which is a visual aid for which there has already been evidence that it improves visual analysis (Stewart, Carr, Brandt, & McHenry, 2007; Wolfe & Slocum, 2015; Young & Daly, 2016) and that it performs well in terms of avoiding false positives (Lanovaz, Huxley, & Dufour, 2017). The input of the application is a data file organized as illustrated in the webpage. The output is provided in Figures 2 and 3.

Figure 2.

A snapshot of the web-based application for following the What Works Clearinghouse Standards recommendations for visual analysis.

Figure 3.

A snapshot of the web-based application for following the What Works Clearinghouse Standards recommendations for visual analysis and assessing consistency.

Regarding level, two lines marked with different colors are superimposed for each phase (one based on the within-phase mean and one based on the within-phase median) and the difference in means and medians is computed. Regarding trend, the MASE criterion is used to identify which of five techniques for finding a best-fitting straight line (least squares, Theil–Sen, bi-split, tri-split, or differencing) leads to better fit. The slopes for the trend lines for the two phases are provided. Moreover, the values of MASE and R² for each phase can be used to assess the goodness of fit. These values can also be understood as informing about trend stability: smaller MASE and greater R² correspond to data that show less variability around the trend line. Note that following the MASE criterion makes unnecessary the subjective decisions regarding the straight line fitting technique to use. Moreover, fitting trend lines separately to each phase makes unnecessary the subjective decision regarding how to detrend the intervention phase data.

Regarding data variability, the idea of a trend stability envelope (Lane & Gast, 2014) is followed: The limits of the envelope are defined by adding and subtracting 25% of the within-phase median to the previously identified best-fitting straight line. Regarding the immediacy of effect, one graphical representation and quantification focuses on the mean and median level of the last three baseline data points as compared, respectively, with the mean and median level of the first three intervention phase data points. Another graphical representation and quantification focuses on the prediction of the first intervention phase data point according to the baseline versus intervention phase trend lines, as previously identified according to MASE. Regarding overlap, the graphical representations shows, for each Phase B value, the number of Phase A values that it improves, apart from providing the values of the PND and the NAP.

All the aspects mentioned thus far are available in a single page (see Figure 2) with six panels, so that a visual analyst may have all this information visible simultaneously to respond to the “holistic and integrative nature” of visual analysis (Parker et al., 2006, p. 419). Overall, for the example data, it can be stated that Phase A data are not stable and not well represented by linear trend. Nevertheless, a general improvement is observed, especially at the end of the baseline. Phase B data are more stable and better represented by a slightly deteriorating trend line. In terms of the difference between phases, there is a clear and apparently large immediate decrease in the undesirable behavior and no overlap between the measurements of the different conditions.

On Figure 3, we represent the way in which the assessment of consistency of data patterns is implemented in the website. As stated in the beginning of the text, inferring a causal relation between intervention and target behavior requires at least three replications and consistency in the effects. In this case, all three tiers from the nonconcurrent multiple-baseline design used by Eilers and Hayes (2015) are represented in an attempt to explore whether there is a similarity between (a) the data from the A phases, (b) the data from B phases, and (c) the type of change between phases. Specifically, a multilevel model is implemented (see Baek, Petit-Bois, Van Den Noortgate, Beretvas, & Ferron, 2016, for a visually based example), including general trend, the effect of the intervention on the trend, and effect of the intervention on level. The average estimates obtained via the multilevel model² for the three tiers are presented with a thick line. These average estimates can be compared with the least squares regression lines fitted separately to each phase within each tier. The degree of similarity between the average levels and trends and the individual levels and trends can be used to assess the consistency of data patterns. In this particular example, there is greater similarity of the data patterns for the tiers represented in the upper part of Figure 3, whereas the tier in the lower part shows a different pattern.

Is It Necessary to Follow the WWC Standards?

It has been clearly stated by the WWC panel of experts (Kratochwill et al., 2013) that the Standards described in the 2010 document are pilot and not definitive. Accordingly, there have been efforts to make them more systematic by developing protocols (Maggin, Briesch, & Chafouleas, 2013) and also to emphasize the availability of additional tools related to the assessment of external validity (Hitchcock, Kratochwill, & Chezan, 2015). It is also important to state that there is a variety of rubrics proposing criteria for methodological quality (Smith, 2012) and that these rubrics do not always agree (Maggin, Briesch, Chafouleas, Ferguson, & Clark, 2014). This challenge can also be seen in the wider context of concerns expressed regarding the accuracy of WWC reports and the procedures and standards that guide them (Wood, 2017).

Despite these challenges, if the focus is put specifically on evidence standards and not on design standards, the picture looks somewhat different. The main tool for judging the availability of sufficient evidence of a functional relation is visual analysis and six data features are highlighted. These data features are well-aligned with (a) the ones identified in empirical research (e.g., Knapp, 1983); (b) the ones discussed by authorities in the field, who were not members of the WWC panel of experts (e.g., Lane & Gast, 2014; Parsonson & Baer, 1992); (c) the data features on which the empirically-supported conservative dual criterion (Fisher et al., 2003) is based; and (d) the data features included in the set of rules used by Krueger, Rapp, Ott, Lood, and Novotny (2013) for defining what a clear change and a potential for clear change is. In that sense, it seems logical to focus on these data features, although the specific way in which they are assessed (e.g., how to estimate the best-fitting trend line, how to quantify nonoverlap) and the importance of each of them when determining whether a functional relation has been established are much more open to debate. We make recommendations regarding how each of the data features can be assessed, but their weight in the overall assessment of intervention effectiveness is yet to be discussed thoroughly.

Finally, assessing the cumulative amount of evidence can be done following the 5-3-20 rule (Kratochwill et al., 2013), specifying, respectively, the minimum number of studies, independent research teams, and participants required for considering a practice as “evidence-based.” However, this is not the only option. For instance, Lanovaz and Rapp (2016) suggest that any such recommendation should have an objective basis. In that sense, they provide criteria based on the binomial distribution for the minimum proportion of experiments with positive results to achieve a success rate of at least 50% and a confidence interval range of 40% or less.

Discussion

To favor the transparency of reporting (Tate et al., 2016) and the replicability of results, we recommend that researchers clearly state exactly how a trend line is fitted and how baseline trend is controlled for, if such a control is performed. We hope that the details provided here and the information depicted in the output of the web application are helpful for transparent reporting. In addition, similar explicit statements regarding the quantification of the difference in level, overlap, immediacy of effect, or variability are also required.

Implications

The main implication of the current work for applied researchers and practitioners is the possibility to carry out, online and for free, systematic visual analysis, using visual aids and quantifications referring to the six data features highlighted in the WWC Standards. Regarding the analytical challenges identified in the article, the question of exactly how to detrend (i.e., how to remove baseline trend) is avoided by fitting trend lines separately to each phase, whereas the question about the trend line fitting technique is answered objectively by using the MASE. The quantification, implemented in the website, of the remaining data aspects has also been explained, to improve transparency.

The main contribution for methodologists and statisticians is the systematization of best-fitting straight line techniques and detrending procedures. The formulaic representations included in the online appendices are useful for simulation studies and further software implementations to improve reproducibility.

Limitations

Regarding the overview of techniques for finding a best-fitting straight line, it was already underlined that it should not be considered comprehensive, as the focus is on techniques included in single-case data analytical procedures. Another limitation already mentioned is the focus on linear trend, which should be understood as a parsimonious simplification of reality and not as an assumption that all trends are linear. Moreover, we did not perform a simulation study to compare the different trend line fitting techniques; we rather implemented, in the web-based application, an objective criterion based on MASE for selecting the trend line fitting technique to use for each separate actually obtained data set.

Regarding the application developed, we do not claim that it includes all possible ways of implementing the WWC Standards recommendations for visual analysis. On one hand, some researchers may prefer other ways of representing variability. For instance, in the single-case data analysis (SCDA) plug-in (Bulté & Onghena, 2012) for R-Commander (which is part of R; https://www.R-project.org/), variability can be represented using range lines, range bars, and a trended range. On the other hand, the SCDA, unlike the application presented in the current text, allows representing nonlinear trends via running medians (Tukey, 1977), without assuming any particular model for the data. Thus, we encourage applied researchers to also use the SCDA plug-in for visual analysis.

Footnotes

Acknowledgements

The author would like to thank Dr. Patrick Onghena for his insightful feedback on previous versions of this text.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Supplemental Material

Supplemental material is available for this article online.

Author Biography

Rumen Manolov, PhD, is researcher and lecturer at the Faculty of Psychology at the University of Barcelona in Spain. His investigation is focused on single-case designs, specifically, testing and proposing data analytical techniques, as well as developing and promoting R code and user-friendly Shiny applications for single-case data analysis.

References

Allison

D. B.

Gorman

B. S.

(1993). Calculating effect sizes for meta-analysis: The case of the single case. Behaviour Research and Therapy, 31, 621-631.

Allison

D. B.

Gorman

B. S.

(1994). “Make things as simple as possible, but no simpler.” A rejoinder to Scruggs and Mastropieri. Behaviour Research and Therapy, 32, 885-890.

Anderson

Schumacker

R. E.

(2003). A comparison of five robust regression methods with ordinary least squares regression: Relative efficiency, bias, and test of the null hypothesis. Understanding Statistics, 2, 70-103.

Baek

E. K.

Petit-Bois

Van Den Noortgate

Beretvas

S. N.

Ferron

J. M.

(2016). Using visual analysis to evaluate and refine multilevel models of single-case studies. The Journal of Special Education, 50, 18-26.

Bailey

D. B.

(1984). Effects of lines of progress and semilogarithmic charts on ratings of charted data. Journal of Applied Behavior Analysis, 17, 359-365.

Brossart

D. F.

Parker

R. I.

Castillo

L. G.

(2011). Robust regression for single-case data analysis: How can it help? Behavior Research Methods, 43, 710-719.

Bulté

Onghena

(2012). When the truth hits you between the eyes: A software tool for the visual analysis of single-case experimental data. Methodology, 8, 104-114.

Campbell

J. M.

(2013). Commentary on PND at 25. Remedial and Special Education, 34, 20-25.

Campbell

J. M.

Herzinger

C. V.

(2010). Statistics and single subject research methodology. In Gast

D. L.

(Ed.), Single subject research methodology in behavioral sciences (pp. 417-453). London, England: Routledge.

10.

Carr

J. E.

(2005). Recommendations for reporting multiple-baseline designs across participants. Behavioral Interventions, 20, 219-224.

11.

Center

B. A.

Skiba

R. J.

Casey

(1985-1986). A methodology for the quantitative synthesis of intra-subject design research. The Journal of Special Education, 19, 387-400.

12.

Chatfield

(2000). Time-series forecasting. London, England: Chapman & Hall.

13.

Eilers

H. J.

Hayes

S. C.

(2015). Exposure and response prevention therapy with cognitive defusion exercises to reduce repetitive and restrictive behaviors displayed by children with autism spectrum disorder. Research in Autism Spectrum Disorders, 19, 18-31.

14.

Fahmie

T. A.

Hanley

G. P.

(2008). Progressing toward data intimacy: A review of within-session data analysis. Journal of Applied Behavior Analysis, 41, 319-331.

15.

Ferron

J. M.

Moeyaert

Van Den Noortgate

Beretvas

S. N.

(2014). Estimating causal effects from multiple-baseline studies: Implications for design and analysis. Psychological Methods, 19, 493-510.

16.

Fisher

W. W.

Kelley

M. E.

Lomas

J. E.

(2003). Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs. Journal of Applied Behavior Analysis, 36, 387-406.

17.

Fox

(2016). Applied regression analysis and generalized linear models (3rd ed.). London, England: Sage.

18.

Gage

N. A.

Lewis

T. J.

(2013). Analysis of effect for single-case design research. Journal of Applied Sport Psychology, 25, 46-60.

19.

Gorsuch

R. L.

(1983). Three methods for analyzing limited time-series (N of 1) data. Behavioral Assessment, 5, 141-154.

20.

Harrop

J. W.

Velicer

W. F.

(1985). A comparison of alternative approaches to the analysis of interrupted time-series. Multivariate Behavioral Research, 20, 27-44.

21.

Heyvaert

Wendt

Van Den Noortgate

Onghena

(2015). Randomization and data-analysis items in quality standards for single-case experimental studies. The Journal of Special Education, 49, 146-156.

22.

Hitchcock

J. H.

Kratochwill

T. R.

Chezan

L. C.

(2015). What Works Clearinghouse standards and generalization of single-case design evidence. Journal of Behavioral Education, 24, 459-469.

23.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005). The use of single subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179.

24.

Horner

R. H.

Swaminathan

Sugai

Smolkowski

(2012). Considerations for the systematic analysis and use of single-case research. Education and Treatment of Children, 35, 269-290.

25.

Hyndman

R. J.

Koehler

A. B.

(2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22, 679-688.

26.

Knapp

T. J.

(1983). Behavioral analysts’ visual appraisal of behavior change in graphic display. Behavioral Assessment, 5, 155-164.

27.

Kratochwill

T. R.

Hitchcock

J. H.

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2010). Single case designs technical documentation. In What Works Clearinghouse: Procedures and standards handbook (Version 1.0). Retrieved from https://ies.ed.gov/ncee/wwc/Document/229

28.

Kratochwill

T. R.

Hitchcock

J. H.

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2013). Single-case intervention research design standards. Remedial and Special Education, 34, 26-38.

29.

Kratochwill

T. R.

Levin

J. R.

Horner

R. H.

Swoboda

C. M.

(2014). Visual analysis of single-case intervention research: Conceptual and methodological issues. In Kratochwill

T. R.

Levin

J. R.

(Eds.), Single-case intervention research: Methodological and statistical advances (pp. 91-125). Washington, DC: American Psychological Association.

30.

Krueger

T. K.

Rapp

J. T.

Ott

L. M.

Lood

E. A.

Novotny

M. A.

(2013). False positives in A-B designs: Potential implications for practitioners. Behavior Modification, 37, 615-630.

31.

Lane

J. D.

Gast

D. L.

(2014). Visual analysis in single case experimental design studies: Brief review and guidelines. Neuropsychological Rehabilitation, 24, 445-463.

32.

Lanovaz

M. J.

Huxley

S. C.

Dufour

M. M.

(2017). Using the dual-criteria methods to supplement visual inspection: An analysis of non-simulated data. Journal of Applied Behavior Analysis, 50, 662-667. doi:10.1002/jaba.394

33.

Lanovaz

M. J.

Rapp

J. T.

(2016). Using single-case experiments to support evidence-based decisions: How much is enough? Behavior Modification, 40, 377-395.

34.

Maggin

D. M.

Briesch

A. M.

Chafouleas

S. M.

(2013). An application of the What Works Clearinghouse standards for evaluating single-subject research: Synthesis of the self-management literature base. Remedial and Special Education, 34, 44-58.

35.

Maggin

D. M.

Briesch

A. M.

Chafouleas

S. M.

Ferguson

T. D.

Clark

(2014). A comparison of rubrics for identifying empirically supported practices with single-case research. Journal of Behavioral Education, 23, 287-311.

36.

Manolov

Arnau

Solanas

Bono

(2010). Regression-based techniques for statistical decision making in single-case designs. Psicothema, 22, 1026-1032.

37.

Manolov

Moeyaert

(2017). How can single-case data be analyzed? Software resources, tutorial, and reflections on analysis. Behavior Modification, 41, 179-228.

38.

Manolov

Rochat

(2015). Further developments in summarising and meta-analysing single-case data: An illustration with neurobehavioural interventions in acquired brain injury. Neuropsychological Rehabilitation, 25, 637-662.

39.

Manolov

Solanas

(2008). Comparing N = 1 effect size indices in presence of autocorrelation. Behavior Modification, 32, 860-875.

40.

Manolov

Solanas

(2013). A comparison of mean phase difference and generalized least squares for analyzing single-case data. Journal of School Psychology, 51, 201-215.

41.

Mercer

S. H.

Sterling

H. E.

(2012). The impact of baseline trend control on visual analysis of single-case data. Journal of School Psychology, 50, 403-419.

42.

Michiels

Heyvaert

Meulders

Onghena

(2017). Confidence intervals for single-case effect size measures based on randomization test inversion. Behavior Research Methods, 49, 363-381.

43.

Miller

M. J.

(1985). Analyzing client change graphically. Journal of Counseling and Development, 63, 491-494.

44.

Novotny

M. A.

Sharp

K. J.

Rapp

J. T.

Jelinski

J. D.

Lood

E. A.

Steffes

A. K.

(2014). False positives with visual analysis for nonconcurrent multiple baseline designs and ABAB designs: Preliminary findings. Research in Autism Spectrum Disorders, 8, 933-943.

45.

Ottenbacher

K. J.

(1990). When is a picture worth a thousand p values? A comparison of visual and quantitative methods to analyze single subject data. The Journal of Special Education, 23, 436-449.

46.

Parker

R. I.

Brossart

D. F.

(2003). Evaluating single-case research data: A comparison of seven statistical methods. Behavior Therapy, 34, 189-211.

47.

Parker

R. I.

Cryer

Byrns

(2006). Controlling baseline trend in single-case research. School Psychology Quarterly, 21, 418-443.

48.

Parker

R. I.

Vannest

K. J.

(2009). An improved effect size for single-case research: Nonoverlap of all pairs. Behavior Therapy, 40, 357-367.

49.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

(2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322.

50.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

(2014). A simple method to control positive baseline trend within data nonoverlap. The Journal of Special Education, 48, 79-91.

51.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

Sauber

S. B.

(2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42, 284-299.

52.

Parsonson

B. S.

Baer

D. M.

(1992). The visual analysis of data, and current research into the stimuli controlling it. In Kratochwill

T. R.

Levin

J. R.

(Eds.), Single-case research designs and analysis: New directions for psychology and education (pp. 15-40). Hillsdale, NJ: Lawrence Erlbaum.

53.

Pfadt

Wheeler

D. J.

(1995). Using statistical process control to make data-based clinical decisions. Journal of Applied Behavior Analysis, 28, 349-370.

54.

Pustejovsky

J. E.

Hedges

L. V.

Shadish

W. R.

(2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39, 368-393.

55.

Schlosser

R. W.

Lee

D. L.

Wendt

(2008). Application of the percentage of non-overlapping data (PND) in systematic reviews and meta-analyses: A systematic review of reporting characteristics. Evidence-Based Communication Assessment and Intervention, 2, 163-187.

56.

Scruggs

T. E.

Mastropieri

M. A.

(2013). PND at 25: Past, present, and future trends in summarizing single-subject research. Remedial and Special Education, 34, 9-19.

57.

Shadish

W. R.

Hedges

L. V.

Pustejovsky

J. E.

(2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52, 123-147.

58.

Skiba

Deno

Marston

Casey

(1989). Influence of trend estimation and subject familiarity on practitioners judgements of intervention effectiveness. The Journal of Special Education, 22, 433-446.

59.

Smith

J. D.

(2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17, 510-550.

60.

Solanas

Manolov

Onghena

(2010). Estimating slope and level change in N=1 designs. Behavior Modification, 34, 195-218.

61.

Solmi

Onghena

Salmaso

Bulté

(2014). A permutation solution to test for treatment effects in alternation design single-case experiments. Communications in Statistics—Simulation and Computation, 43, 1094-1111.

62.

Solomon

B. G.

(2014). Violations of assumptions in school-based single-case data: Implications for the selection and interpretation of effect sizes. Behavior Modification, 38, 477-496.

63.

Stewart

K. K.

Carr

J. E.

Brandt

C. W.

McHenry

M. M.

(2007). An evaluation of the conservative dual-criterion method for teaching university students to visually inspect AB-design graphs. Journal of Applied Behavior Analysis, 40, 713-718.

64.

Sullivan

K. J.

Shadish

W. R.

Steiner

P. M.

(2015). An introduction to modeling longitudinal data with generalized additive models: Applications to single-case designs. Psychological Methods, 20, 26-42.

65.

Swaminathan

Rogers

H. J.

Horner

Sugai

Smolkowski

(2014). Regression models for the analysis of single case designs. Neuropsychological Rehabilitation, 24, 554-571.

66.

Tarlow

(2017). An improved rank correlation effect size statistic for single-case designs: Baseline corrected Tau. Behavior Modification, 41, 427-467.

67.

Tate

R. L.

Perdices

Rosenkoetter

McDonald

Togher

Vohra

(2016). The Single-Case Reporting Guideline In BEhavioural Interventions (SCRIBE) 2016: Explanation and elaboration. Archives of Scientific Psychology, 4, 10-31.

68.

Tukey

J. W.

(1977). Exploratory data analysis. London, England: Addison-Wesley.

69.

White

D. M.

Rusch

F. R.

Kazdin

A. E.

Hartmann

D. P.

(1989). Applications of meta-analysis in individual subject research. Behavioral Assessment, 11, 281-296.

70.

Wilcox

R. R.

(2012). Introduction to robust estimation and hypothesis testing (3rd ed.). London, England: Academic Press.

71.

Wolery

Busick

Reichow

Barton

E. E.

(2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. The Journal of Special Education, 44, 18-29.

72.

Wolfe

Slocum

T. A.

(2015). A comparison of two approaches to training visual analysis of AB graphs. Journal of Applied Behavior Analysis, 48, 472-477.

73.

Wood

T. W.

(2017). Does the What Works Clearinghouse really work? Investigations into issues of policy, practice, and transparency (A National Institute for Direct Instruction White Paper). Office of Research and Evaluation. Retrieved from https://www.nifdi.org/docman/research/white-papers/1431-does-the-what-works-clearinghouse-really-work-investigations-into-issues-of-policy-practice-and-transparency/file

74.

Yohai

V. J.

(1987). High breakdown-point and high efficiency robust estimates for regression. The Annals of Statistics, 15, 642-656.

75.

Young

N. D.

Daly

E. J.

III . (2016). An evaluation of prompting and reinforcement for training visual analysis skills. Journal of Behavioral Education, 25, 95-119.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.33 MB

0.27 MB